Migrating a Fortune 500 SAP BW estate to a Databricks lakehouse

Nobody funds "move to Databricks"

The client runs its enterprise reporting on SAP BW. Two decades of modeled history sit in it, along with the business logic of a manufacturer that has grown by acquisition. In many cases that logic exists nowhere else in written form.

Nobody phrased the mandate as a platform migration. They phrased it as a problem. The analytics estate has to keep serving finance, supply chain and operations for the next decade, and the current foundation cannot get there. Not on cost, not on the skills market, and not on the analytical workloads the business now wants to run.

So the work came in three parts, in this order. Establish what the target state should be. Get it to something the business could commit budget to. Then deliver it. I own the architecture and the delivery. The funding case was a team effort that I built the technical roadmap for.

Scope set from an inventory, not a diagram

The most common way a migration of this size goes wrong is that scope gets set from an architecture diagram, then discovered from the source system eighteen months later. Before committing to anything we inventoried the estate object by object.

That inventory is the program. Every object got profiled for row counts, change rate, key stability and downstream consumers. Then it got a decision. Migrate it, re-derive it, or retire it. A good share of a twenty-year-old estate is dead, and finding that out early is worth more than any tooling choice made later.

What the profiling changes

• Scope becomes defensible. You can hold a schedule and a budget against "250 tables". You cannot hold either against "the master data".
• Sequencing gets obvious. Counting consumers per object tells you which models have to move together and which can wait.
• Dead weight gets left behind. Every retired object is scope that never needs testing, reconciling or cutting over.

Where the estate should land

Three credible target states, each with a real case for it. The decision had to survive a steering committee, so we made it on the criteria the business cares about. Total cost over the horizon, skills availability, workload fit, and how much of the existing reporting estate survives the move.

The second option deserves a note. I have run the BW-versus-Datasphere evaluation as a contested decision on another program, where the platform choice was wide open and the wrong answer would have cost a year. You de-risk that with a structured feasibility evaluation that produces evidence without exposing scope, timeline or budget while it runs.

What made a multi-year migration fundable

A multi-year platform migration is one of the hardest things to fund in an enterprise. The value is real but arrives late, while the cost is immediate and exact. The program was approved at approximately $2.8M through 2029. That was a team outcome and a partner-led conversation with the CFO. My part was the layer underneath it, the analytics roadmap and the technical case it rests on, built so the money was never being asked for in the abstract.

In practice that meant four things, and they are the things I hold a roadmap to.

1. Tie every workstream to a capability the business already wants. Not "migrate the finance models" but "make month-end close reportable on day two", because someone senior already owns that outcome.
2. Sequence so value lands before the end. A roadmap that pays nothing until 2029 will not survive its second budget cycle. Each wave has to deliver something the business can point at.
3. Quantify the cost of doing nothing. Run-cost trajectory, the skills market, the reporting backlog that never clears. The counterfactual belongs in the case, not in the appendix.
4. Show the risk position honestly. A case with no risks in it reads as naïve. A steering committee that finds the risks on its own stops trusting the numbers.

Medallion on Delta Lake, governed by Unity Catalog

The target is deliberately conventional. On a program with this many moving parts and this long a horizon, architectural novelty is a liability. The design has to be legible to engineers who join in 2027 and auditable by people who were not in the room.

The rules the design is held to

• Bronze is replayable. Silver and gold can be rebuilt from bronze without going back to SAP. That one property is what makes a five-year program survivable.
• Business logic lives in the model. Not in a report measure, and not in a notebook someone ran once.
• Grain is declared before build. Every fact table, in writing.
• No layer is skipped for speed. The shortcut from bronze straight to a report is the first thing that gets built under pressure. It is also the first thing that becomes unmaintainable.

Four hops, each one earning its place

Getting data out of SAP is the part of an SAP migration that plans consistently underestimate. The chain below is deliberately not the shortest possible path. Each hop is there because removing it costs more than keeping it.

Non-negotiables on the chain

• Idempotent loads. A re-run reproduces bronze exactly. Nothing about recovery should need judgement at 3am.
• Watermarked and audited. Every run writes a control row, so "did last night load?" is a query rather than a conversation.
• Extraction windows agreed with SAP basis. The source system's load profile is a design constraint. Negotiate it early instead of discovering it in production.
• Immutable landing. Files land once, partitioned by source and load date, and never get edited in place.

Preserving history instead of rebuilding it

There is always pressure to treat a migration as a clean-slate rebuild. It is a tempting idea, because the old models carry compromises and building fresh feels faster. This program committed to the opposite. Historical enterprise data gets carried across rather than rebuilt, and that one decision shapes everything downstream.

Why carrying it is the cheaper answer

• Reconciliation is the real cost of a rebuild. New numbers have to be proven equal to old numbers, report by report and year by year. Rebuilding multiplies that work. Migrating the history contains it.
• Regulated reporting will not accept a restatement just because a platform changed. The comparatives have to tie.
• The old logic encodes decisions nobody remembers making. Re-deriving it from scratch means re-litigating twenty years of business rules with people who have long since moved on.

Models move in waves rather than all at once, grouped by the consumers they share so a wave can be cut over as one unit. Each wave carries its own reconciliation pack of old versus new, at the grain the business reads.

Unity Catalog, designed in at the start

Governance introduced late becomes a bottleneck, and by the time it arrives everyone has built their way around it. Designed in from the first table it does the opposite. It removes questions that would otherwise get argued per project for five years.

• One catalog topology across dev, test and prod, so promotion is a path rather than a rewrite.
• A named owner per schema. A person rather than a team inbox, someone who answers for what is in it.
• Access through groups mapped to business roles, never to individuals. Grant to individuals and the model rots within two quarters.
• PII and confidential data classified at column level and enforced by the platform rather than by a policy document nobody opens.
• Lineage as the impact-analysis tool. When a source column changes in SAP, "what breaks?" has an answer in minutes.

I took the same posture on a multi-region agriscience program, where data governance and protection ran across the enterprise data warehouse. Compliance works as a delivery guardrail rather than a bottleneck. It is faster that way, not slower, and it is the only version that survives an audit.

The success condition is that nothing looks different

Around 200 Power BI reports sit on top of this estate. A migration looks like the natural moment to redesign them. It is also how a platform program picks up a change-management program it was never funded for.

The target is that reports re-point rather than get rebuilt. Gold keeps semantics stable enough that the model changes underneath a report and the person reading it on Monday morning notices nothing at all.

1. Reconcile before cutover, report by report. Old against new, at the grain the business reads, with any variance explained rather than waved at.
2. Show the reconciliation to the report owner. You do not ask for trust here. You evidence it.
3. Cut over by wave, with the old object switched off. Give parallel-run a defined end date or it becomes permanent.
4. Separate SQL warehouses by workload so BI queries and engineering jobs stop competing for compute. That competition is the most common cause of "the new platform feels slower".

Two concurrent programs, one delivery machine

I lead two concurrent programs for the same client, this migration and an S/4HANA analytics delivery, directing up to 12 data engineers and reporting to the client VP. Running them as one delivery machine rather than two is what stops them competing for the same people and the same source-system windows.

• Azure DevOps on scrum cadences. User stories and RICEFWs get tracked where the work happens, not in a status deck assembled the night before.
• An active RAID log, worked weekly. Risks come up early and get closed instead of being discovered at cutover.
• Scope shaped and committed deliberately. I shape proposals and SOWs with the firm’s pursuit teams, then own delivery against exactly what was committed. Having the same person on both sides of that line is the only way estimates stay honest.
• Executive and steering-committee reporting that ties every project back to the business case that funded it.

The five that can break it

A RAID log with forty entries is a filing system. These five get worked personally, because each one can take the program down rather than just slow it.

In delivery, funded through 2029