Microsoft Fabric vs Databricks Gold Layer for Payers

‍

Payer data teams face a critical platform decision when building the Gold layer. This is the final stage of the data pipeline where clean, transformed data becomes the source of truth for regulatory submissions, quality reporting, and financial close. The tables built here directly affect business outcomes. In this article, we'll go over which platform is the right Gold layer foundation for payer data teams.

‍

Why the Gold Layer Is a Revenue Layer, Not a Reporting Layer

‍

Framing the Gold layer as a "reporting layer" leads to the wrong platform choice. These outputs are operational and financial, not informational.

‍

What Payer Teams Are Actually Building at the Gold Layer

‍

What lives at the Gold layer in a Medicare Advantage operation? Risk adjustment tables feeding CMS submission files. STARS measure rollups determining whether a plan qualifies for quality bonus payments. MLR calculations a CFO signs off on at period close. MMR reconciliation logic aligning member-level revenue to what CMS pays monthly. Professional encounter data records submitted in X12 837 5010 format, where a professional EDR carries up to 12 diagnosis codes and an institutional EDR carries up to 25. These determine what gets filed with CMS, what gets booked in the general ledger, and what drives every downstream member-level decision.

‍

The CMS RADV program exists to confirm that diagnoses submitted for risk adjustment are supported in the medical record, with the 2023 final rule codifying extrapolation beginning with payment year 2018. That is why Gold layer lineage is a revenue-defense control, not a cosmetic governance feature.

‍

Mid-Modernization vs Net-New Build

‍

Most payer data teams in this evaluation fall into one of two situations. Mid-modernization means Bronze and Silver are already on Azure, likely with ADF pipelines ingesting 834 and 837 files via SFTP, and the question is whether the Gold layer extends that ecosystem with Fabric or breaks toward Databricks. Net-new means standing up a payer data warehouse from scratch before the first pipeline gets built. Both are legitimate. The right answer differs by scenario.

‍

Why Generic Platform Benchmarks Fail Payer Evaluation Criteria

‍

Most platform comparisons test SQL throughput, data volume, or BI latency. None of that captures what breaks in a payer environment. They do not test for audit traceability under RADV scrutiny, SCD Type 2 support for V28 HCC recapture history, MAO-004 submission pipeline integrity, or refresh cadence alignment to monthly MMR reconciliation cycles. Generic benchmarks were not built to fail gracefully in front of a CMS examiner. The Gold layer has to be.

‍

Where Microsoft Fabric Wins and What to Consider

‍

Fabric is compelling for certain payer architectures. But the conditions under which it wins are specific.

OneLake and the Power BI Advantage for Payers

‍

If the plan already runs Power BI for operational reporting, Fabric's Direct Lake mode is a real advantage. Microsoft states that Direct Lake refresh is a low-cost metadata-framing operation that can complete in seconds, and that Direct Lake or Import mode typically outperforms DirectQuery for report interactions.

For a quality team refreshing STARS denominator and numerator views after monthly eligibility and encounter updates and pushing results to executive scorecards, that means live refresh without waiting on a full semantic model reload. OneLake also simplifies access management when Gold outputs need to reach multiple downstream consumers without duplicating data across storage accounts.

‍

Fabric's Structural Edge for Regional Health Plans Already Running Azure Data Factory and SFTP-Based EDI Pipelines

‍

If the Bronze layer is built on ADF pulling 834 enrollment files and 835/837 claims via SFTP, Fabric extends that stack without a hard infrastructure break. Azure Active Directory integration means no separate identity layer for Gold layer governance. On the commercial side, Fabric East US list pricing runs from $2,102.40/month for F16 to $8,409.60/month for F64, with roughly 41 percent savings at one- or three-year reservation terms. At F64 or above, Power BI report consumers no longer need individual Pro licenses, and that tier includes 64 TB of mirrored replica storage. For regional plans with a predictable reporting footprint, that capacity-first model is easier to budget than Databricks' workload-first DBU structure.

‍

Architectural Considerations for Complex Multi-Source Payer Pipelines in Fabric

‍

Fabric's lakehouse capabilities are maturing rapidly, and Microsoft continues to expand its pipeline orchestration features. For payer teams building complex multi-source pipelines, it's worth understanding the current architecture boundaries. 

‍

Fabric Data Factory pipelines have an activity cap, and Dataflow Gen2 has query limits per publish cycle. Teams with highly concurrent SCD Type 2 requirements across many source systems should validate their pipeline design against these parameters early. In most regional health plan scenarios, these boundaries are manageable with good architectural planning.

‍

Building a Custom HEDIS Engine in Fabric: What to Plan For

‍

A full HEDIS engine manages measure specifications across dozens of clinical domains, denominator populations shifting monthly as 834 eligibility files change, and numerator logic pulling from pharmacy claims, lab results, and ADT feeds. The orchestration demands are significant. 

‍

Fabric now has a documented SCD Type 2 pattern in Dataflow Gen2 and recently added Extended SCD Type 2 support in Copy job for Fabric Data Warehouse as a preview capability. Fabric's SCD2 support is actively expanding. The documented pattern in Dataflow Gen2 and the Extended SCD Type 2 preview in Copy job for Fabric Data Warehouse give teams viable paths today. 

‍

Microsoft's roadmap continues to mature this capability. Teams planning a custom HEDIS engine should design their pipeline around these patterns early to take full advantage as the features reach general availability.

‍

Delta Live Tables and SCD Type 2

‍

Databricks has two capabilities that shift the conversation for payer Gold layer workloads specifically: Delta Live Tables and native SCD Type 2. Not for every workload. For the ones where submission integrity and audit lineage are non-negotiable.

‍

Delta Live Tables for CMS Submission Pipelines

‍

Delta Live Tables enables defining data quality expectations declaratively on the pipeline, not as a post-hoc monitoring layer. For a MAO-004 submission pipeline, a single row with a missing or malformed ICD-10 code can break the entire CMS file. With DLT, expectations are enforced at the table level, routing failed records to a quarantine table rather than letting them contaminate the submission output. Databricks recommends serverless pipelines for new builds, noting that serverless uses enhanced autoscaling, eliminates manual cluster configuration, and always uses Unity Catalog by default, meaning governance and lineage are built in from the start.

‍

SCD Type 2 for Attribution, HCC & Eligibility Tracking

‍

Member attribution changes. A PCP changes mid-year. Plan enrollment terminates and re-activates. HCC recapture status shifts as providers confirm or reject suspect diagnoses during chart reviews. All of that must be tracked historically, not just the current state. 

‍

Databricks' AUTO CDC APIs automate SCD Type 1 and Type 2, allowing teams to specify which columns trigger historical versioning, and automatically capture run metrics including rows upserted and deleted. That maps directly to PCP reassignment history, eligibility effective and termination date versioning, RAF version snapshots, and V28 recapture history where not every changed field should trigger a full historical rewrite.

‍

Unity Catalog Lineage for RADV Audits

‍

Unity Catalog captures lineage automatically at the column level and aggregates it across all workspaces attached to the metastore. External lineage is available in public preview, meaning upstream systems and downstream tools like Power BI appear in the same lineage graph. For a RADV audit, an examiner can trace exactly which 837 claim record, from which trading partner, on which submission date, produced a specific HCC submission. One important nuance: Databricks documents a rolling 1-year window for lineage system tables, while Catalog Explorer and the lineage API retain data captured after September 1, 2024 indefinitely. Teams that need multi-year queryable lineage for internal audit analytics should archive system-table outputs proactively.

‍

The Tradeoff for Azure-Centric Stacks

‍

Databricks is not free of tradeoffs. Lakeflow Jobs on Azure is $0.15 per DBU. Lakeflow Spark Declarative Pipelines range from $0.20 per DBU at Core to $0.36 at Advanced. Databricks SQL Pro is $0.55 per DBU and SQL Serverless is $0.70 per DBU all-in. A payer running 25,000 Jobs DBUs per month is looking at roughly $3,750 in DBU charges before Azure VM cost. That workload-proportional structure gives precise control but adds FinOps complexity that a Fabric capacity model avoids entirely.

‍

Payer Use Case Scorecard

‍

Risk Adjustment Pipelines

‍

This is one of the more technically demanding workloads for any Gold layer platform. The V28 HCC model changes which conditions count toward a RAF score and how they are scored in combination. Tracking recapture history across the MORL-to-MORM transition requires SCD Type 2 versioning on member-level HCC tables and the RAF impact at each point. Databricks' native Delta MERGE and time-travel offer one well-established approach to this pattern. Fabric teams can address this with the Dataflow Gen2 SCD Type 2 pattern and Extended SCD Type 2 preview in Fabric Data Warehouse — and teams should validate their specific submission volume against their chosen implementation before go-live.

‍

STARS Measure Calculation

‍

This workload can go either way. If the quality team runs STARS reporting in Power BI, Fabric's Direct Lake mode gives a real advantage on refresh latency and dashboard performance. Monthly 834 eligibility updates that change HEDIS denominator populations feed cleanly into OneLake, and Direct Lake queries reflect those updates without a full semantic model refresh. If STARS outputs also feed downstream into risk stratification models or Python-based care management platforms, not just dashboards, Databricks handles that fan-out better.

‍

MLR and Financial Close

‍

CMS requires MA organizations to report MLR annually and enforces an 85 percent statutory minimum, with the 2025 MLR Annual Reporting Form due July 31, 2026. The CFO cares whether the contract-level output is reproducible, explainable, and closes on time, not whether the platform is "unified." If close depends on many reconciled source streams and row-level drillback into why period-close numbers moved, Databricks has the lineage and CDC edge. If the Gold layer is producing contract-level outputs with controlled transformations, Fabric's consumption and pricing model works well. Forcing a single winner here does a disservice to the real decision.

‍

CMS Submission Readiness

‍

For CMS submission traceability, the evidence chain matters. A regional MA plan tracing why an HCC is missing from a beneficiary's MOR must walk from the MOR and MMR back to MAO-004 eligibility and then to the accepted or rejected encounter record. Databricks' declarative quality model, AUTO CDC support, and workspace-spanning Unity Catalog lineage structurally align with that requirement. Fabric addresses this well in Azure-native programs, and teams that design their lineage architecture deliberately from the start can build a fully traceable and auditable Gold layer on Fabric.

‍

Unity Catalog vs Microsoft Purview for Payers

‍

Governance at the Gold layer is a continuous operational requirement that shows up in RADV audits, internal compliance reviews, and state regulatory examinations — not a one-time setup task.

‍

Data Lineage From Bronze EDI Ingestion to Final CMS Submission File

‍

Unity Catalog tracks lineage automatically at the column level across all workspaces. In Fabric, Microsoft Purview integration governs from source to Power BI report, applies sensitivity labels, and logs user activity to Purview audit. However, Microsoft discloses that for non-Power BI Fabric items, Purview lineage does not yet support external upstream sources, cross-workspace lineage, or Notebook-to-Pipeline lineage. For a payer tracing Bronze EDI ingestion end-to-end to a CMS submission table across multiple workspaces, that is a documented limitation, not speculation.

‍

PHI Access Controls and Role-Based Governance for Regulated Gold Layer Tables

‍

Both platforms support role-based access control with PHI-appropriate granularity. Fabric's Azure Active Directory integration is a genuine advantage in Microsoft-centric organizations. Unity Catalog's ABAC policies, row filters, and column masks enable more granular control when different teams need different slices of the same Gold layer table actuarial accessing MLR data without member-level clinical detail, quality accessing STARS denominators without claims financials. Databricks recommends ABAC when policies must scale across many tables, which is precisely the payer Gold layer pattern in practice.

‍

Audit Log Standards That Satisfy RADV Examiners and Internal Compliance Teams

‍

RADV examiners want a specific HCC submission traced to a specific clinical encounter, a specific 837 transaction, and a specific submission date. Delta Lake's append-only transaction log combined with Unity Catalog's audit log at system.access.audit provides that evidence trail. Fabric's audit logging through Purview provides a solid foundation for RADV readiness, and teams should test their audit package assembly end to end on whichever platform they choose before an examiner arrives, not after.

‍

A Framework for Payer Data Teams

‍

Decision Matrix

‍

Three questions drive this decision. First, how deep is the Azure investment? ADF, Azure AD, Power BI, and Azure DevOps as the core stack makes a separate Databricks environment operationally costly. Second, how complex are the pipelines? Five-plus source systems with concurrent writes, SCD Type 2 requirements, and CMS submission outputs favor Databricks. Third, how are the EDI files arriving? SFTP-based 834/837 ingestion already in ADF integrates naturally into Fabric. HIE ADT feeds, MAO-004 pipelines, and multi-partner eligibility files are well suited for Databricks' pipeline orchestration model.

‍

When to Standardize on Fabric and When to Introduce Databricks as the Gold Layer Specialist

‍

Standardize on Fabric when the Gold layer's hardest problem is business consumption rather than historical-state management and when predictable capacity economics matter. A regional health plan modernizing STARS, finance, and contract reporting with moderate pipeline complexity is the clearest fit.

‍

Introduce Databricks as the Gold layer specialist when pipelines must absorb encounter corrections, eligibility deltas, and chart-review updates and prove it to a CMS examiner on demand. Databricks can run at Gold while keeping ADF and Azure infrastructure for Bronze and Silver ingestion. That seam is deliberate, not a workaround.

‍

A hybrid path also exists. Microsoft documents that Azure Databricks can read OneLake data through OneLake URIs and shortcuts, making a credible pattern: Databricks for Gold pipeline engineering and governed transformation, Fabric for Power BI-native semantic serving and business distribution. Vendor-built EDW options from Innovaccer, Arcadia, and HealthEdge remain a relevant alternative to the in-house build decision, but teams evaluating at this depth have likely already weighed that tradeoff.

‍

Final Thoughts

‍

Microsoft Fabric and Databricks are both serious platforms. Neither is a wrong answer in isolation. The wrong Gold layer foundation breaks different things: in Fabric, the weak point is orchestration maturity and documented lineage gaps for complex multi-system payer pipelines; in Databricks, it is cost model complexity and the operational overhead of another specialized platform in an otherwise Azure-centric stack.

‍

If the stack is Azure-native, consumption is Power BI-first, and pipeline complexity is moderate, Fabric is a strong and coherent choice. If the Gold layer is feeding CMS submission pipelines, supporting active RADV audit packages, and managing V28 recapture history the 2026 MORM transition demands, Databricks earns its place as the Gold layer foundation even in an Azure-first environment. The platform built on now is the one the team defends to CMS examiners in two years. Build accordingly.

‍

Frequently Asked Questions

‍

How can Invene help with the Gold layer platform decision?

‍

Invene is a healthcare technology firm specializing in data engineering, cloud infrastructure, and AI for payers. From evaluating platform fit to building production-ready Gold layer pipelines, Invene brings the engineering depth and payer domain expertise to get the decision right and execute on it.

‍

Can Microsoft Fabric and Databricks run together in the same payer data warehouse?

‍

Yes, and a hybrid path is often the right answer. A common architecture keeps Bronze and Silver on ADF and Azure infrastructure while running Databricks at the Gold layer for CMS submission pipelines and RADV-traceable outputs. Fabric then serves as the semantic and reporting layer on top of Gold outputs published to OneLake via the documented shortcut integration. That seam is architectural intentionality, not a limitation.

‍

How does the Fabric capacity pricing model compare to Databricks DBU pricing for payer workloads?

‍

Fabric uses a capacity-first model with predictable monthly costs and a meaningful BI-consumption threshold at F64 where per-user Pro licenses drop away. Databricks uses a workload-first DBU model where costs scale in proportion to the pipelines run, giving control but adding FinOps complexity. For a payer with a predictable executive reporting footprint, Fabric is easier to budget. For variable MAO-004 submission pipelines, RAF versioning, and HEDIS compute, Databricks' proportional cost structure may be more accurate even if harder to forecast.

‍

Does Fabric's Purview governance fully satisfy payer audit traceability requirements?

‍

For many payer workloads, yes. For cross-workspace, multi-hop lineage from Bronze EDI ingestion to a final CMS submission table, it's worth noting that Purview's lineage coverage for Fabric is actively expanding. Some scenarios, including external upstream sources, cross-workspace lineage, and Notebook-to-Pipeline lineage for non-Power BI Fabric items, are on the roadmap as Microsoft continues to develop this area. Teams preparing for RADV audits should validate their end-to-end audit package on whichever platform they choose well before an examiner arrives.

‍

How should a payer data team think about HEDIS engine options relative to this platform decision?

‍

When buying a commercial HEDIS engine, it typically delivers measure outputs the Gold layer ingests as a source, partially decoupling the platform choice from the HEDIS build. When building a custom HEDIS engine in-house, the orchestration maturity of the Gold layer platform is a direct input. For teams building a custom HEDIS engine in-house, the orchestration design matters. Fabric's Dataflow Gen2 SCD Type 2 patterns and expanding pipeline capabilities give teams a viable path, and aligning the pipeline architecture to Fabric's current feature set from the start is the key to a successful build.

‍

‍