Leading a Complex Enterprise Data Platform Through an Architecture Pivot

The Mandate

The initiative was intended to consolidate data from three core enterprise systems into a centralized reporting and analytics foundation. The target environment would automate source ingestion, retain structured historical data and provide a more reliable foundation for Power BI reporting.

The initial implementation plan followed a conventional path from source connectivity through transformation and warehousing to reporting and handover. Based on the information available during solution design, there was little indication that the architecture would later become one of the defining issues of the engagement.

The Starting Point

Azure Data Factory formed the primary integration and transformation layer in the original design, feeding an enterprise SQL Data Warehouse and the downstream Power BI environment. A Self-hosted Integration Runtime provided connectivity into the on-premises environment.

The design was reasonable for what was understood at the time. Expected volumes and integration patterns appeared compatible with the proposed architecture, and delivery began against a structured plan covering integration, warehouse configuration, validation, reporting enablement and handover.

When the Plan Met Reality

The picture changed once implementation reached the source data in greater depth. The technical team encountered nested and irregular payloads, custom fields, variable schemas and transformation requirements that were considerably more complex than anticipated. Permissions and authentication also introduced constraints, while some progress depended on configuration or action outside the immediate delivery team.

For a period, these appeared to be implementation issues that could be worked through within the existing architecture. As more evidence emerged, continuing to extend the original model became increasingly difficult to justify.

The decision was no longer about whether Azure Data Factory could technically be made to handle the workload. The more important question was whether continuing down that path remained the right choice for the project.

The Decision Point

The technical findings were brought together and assessed against the remaining delivery obligations. Kosmos coordinated this reassessment with additional data engineering expertise so that the decision could be made with a clearer understanding of the implementation effort and the consequences for the wider engagement.

The conclusion was to evolve the processing model. The resulting architecture pivot was not a correction of an obviously flawed original design; implementation had revealed information that was not available at the outset. From a delivery perspective, the change also needed to be understood by the service provider and the end client, reflected in the remaining plan and translated into a realistic view of effort, dependencies and authorization.

Architecture Evolution

Azure Databricks was introduced to handle the more complex ingestion, normalization and transformation workload. Azure Data Factory remained in the architecture where it continued to be useful for orchestration and connectivity.

The revised design introduced a Medallion Architecture across Bronze, Silver and Gold data layers and supported more reusable, configuration-driven processing patterns. The architecture now reflected the structure and complexity of the data that had emerged during implementation rather than the assumptions available during initial design.

The Turnaround — Re-establishing the Delivery Path

By late October 2025, the engagement that began in August had reached a genuine turning point. It was more than a technical adjustment: it changed the expertise required to complete the work and reshaped the sequence of the remaining delivery.

Kosmos led the re-baselining from the actual position of the project. Completed work was reviewed to determine what could remain in place, unresolved engineering was reassessed against the new architecture, and external dependencies were incorporated into the revised path to completion. The remaining authorization was aligned to that plan, and revised execution began in November 2025.

The project did not restart from zero. Azure Data Factory remained in use where it still served the revised design, including on-premises connectivity through the existing integration runtime, and the SQL warehouse remained part of the target architecture. Additional data engineering capability could therefore be introduced without discarding viable work or disrupting continuity across the client relationship, governance and overall delivery ownership.

Managing the Delivery Complexity

Once the architecture had been reset, execution still depended on factors that sat across several parties. Platform configuration and access had to line up with the work of the technical team, while authentication and third-party enablement continued to affect progress.

The engagement therefore involved ongoing coordination between the service provider, end-client stakeholders, technical specialists and external parties. Effective dependency management meant keeping a clear distinction between engineering issues, approvals and actions that sat outside the immediate project team.

Kosmos kept ownership and impact visible as dependencies moved through the project. That helped the teams understand where progress was genuinely constrained and where other work could continue, rather than allowing every unresolved item to become part of a single undifferentiated delay. It also gave the service provider and the end client a consistent view of the engagement as the delivery path moved away from the original plan.

Maintaining Commercial Control

The revised architecture changed the effort required to finish the project, creating a commercial challenge alongside the delivery challenge. Because the work was being delivered within a professional services model, additional effort could not simply be absorbed into the engagement as the solution evolved. Kosmos worked within the formal change control process so that the implications of the revised approach remained visible and appropriately authorized.

Later in the engagement, authorized delivery capacity was fully consumed while work still remained. Execution was paused rather than allowing the project to continue without a clear commercial basis. The remaining scope and dependencies were reconciled, and a defined completion path was established before delivery resumed.

This part of the project required careful stakeholder management. The technical team needed a clear boundary around authorized work, the client needed transparency around what remained, and the service provider had to protect the economics of the engagement without damaging the customer relationship. Kosmos worked across those interests to keep commercial decisions connected to the delivery reality.

The Final Push

With the remaining dependencies sufficiently resolved and the completion work authorized, the engagement moved into its final delivery phase. Outstanding integration work was completed, the revised processing framework was validated and the environment was prepared for operational transition.

Closure covered the wider operating handover as well. Final synchronization and infrastructure validation were completed alongside reporting review, technical documentation, code handover and knowledge transfer. In March 2026, approximately four months after the revised delivery path began, the customer accepted a platform ready for operation, bringing the professional services engagement to a clean close for both parties.

Capability Gains

The architecture pivot ultimately produced a more capable platform than the one originally envisioned. Databricks became the core processing layer, supported by Bronze, Silver and Gold data layers and a reusable configuration-driven framework that could accommodate more complex transformations and provide a stronger starting point for future integrations.

The complexity discovered during implementation therefore became part of the value of the final solution. Rather than forcing the data into the assumptions of the original architecture, the project adapted the platform to the reality of the environment.

The Outcome

The engagement concluded with customer acceptance of an enterprise data platform connecting three core business systems into a centralized reporting foundation.

• A hybrid Azure architecture using Azure Databricks and Azure Data Factory
• Bronze, Silver and Gold processing layers based on a Medallion Architecture
• Reusable configuration-driven ingestion and transformation patterns
• Secure data storage and credential management
• Integration with the enterprise SQL Data Warehouse
• Power BI reporting enablement
• Technical documentation and code handover
• Knowledge transfer and formal customer acceptance

The project had moved considerably from its starting point by the time it closed. The technical design had changed, new expertise had been introduced and the commercial path had required active management. Even so, the engagement reached completion with the customer satisfied with the delivered platform and the technology service provider satisfied with the way the work had been carried through to closure.

What This Engagement Demonstrated

Technology delivery rarely unfolds exactly as it was designed. In complex initiatives, implementation inevitably adds information that was unavailable when the original plan was created. The value of delivery leadership in this engagement was not in preserving that plan unchanged, but in recognizing when its assumptions no longer held and creating a controlled way forward without losing sight of the customer commitment.

As the project evolved, Kosmos maintained continuity while the technical direction, resourcing and commercial conditions changed around it. Decisions were translated back into delivery implications, ownership remained visible and the relationship between the service provider and its customer remained intact. The engagement demonstrates the same discipline Kosmos brings to Recovery and Turnaround work: recognizing when the current path is no longer viable, restoring control and establishing a credible route to completion.