From Materialized Views to Change-Aware Data Products: Why Change Data Feed Matters

The Problem Banks Already Know How to Solve - and the One They Don't

Banks are good at capturing change near the source. CDC pipelines, incremental loads, and Delta Lake have made it practical to track every card transaction, deposit, withdrawal, loan payment, and account update as it flows through the ingestion layer.

That part is largely solved.

The harder problem sits one layer up. After all those raw transactions are transformed into a business-level data product - say, a customer summary table - how does a downstream system know that something meaningful changed?

Consider a simplified customer metrics table maintained as a Materialized View:

New transactions arrive. The transformation runs. The view is refreshed:

Your source CDC pipeline can tell you which transactions arrived. It can tell you that five new spend records were written for customer C1001.

But a fraud detection system, a risk engine, or a CRM workflow may not care about those five transactions at all. What it cares about is this:

Risk Score changed from 42 to 67.

That is a different kind of signal. And the question of how to reliably surface it - without forcing every downstream consumer to re-read or re-compare the entire transformed dataset - is the architectural gap that Databricks Change Data Feed (CDF) on Materialized Views is designed to close.

---

Why Traditional Approaches Fall Short

Before CDF on Materialized Views was available, teams handled this problem in a few ways. None of them are particularly clean.

Full dataset comparison. Read the entire current state of the transformed table, compare it row-by-row against a previous snapshot, and emit the differences. This works at small scale. At enterprise scale, it is expensive, slow, and operationally fragile. Daily snapshots. Store a copy of the transformed table every night and diff it the next morning. This introduces latency by design and creates significant storage overhead. It also means your downstream consumers are always working with yesterday's changes, not today's. Custom MERGE logic. Build bespoke MERGE statements that track which rows changed and write those changes to a staging table. This works, but it requires every pipeline to implement its own change-tracking logic. It does not compose well across teams or data products. Intermediate staging tables. Insert changed rows into a separate "delta" table that downstream consumers poll. This is essentially manual CDC at the transformation layer - you are rebuilding infrastructure that should not need to be rebuilt. Downstream recomputation. Let each consumer re-run the transformation themselves to detect what changed. This is the most common approach in practice, and it is also the most wasteful. Every consumer pays the full cost of the transformation, even when only a small fraction of the data changed.

All of these approaches share a common flaw: they treat change detection as the downstream consumer's problem. The data product itself has no native way to say "here is what changed since you last looked."

---

What CDF on Materialized Views Changes

Databricks Change Data Feed is not a new concept at the raw table level - it has been available on Delta tables for some time. What changes with CDF support on Materialized Views is where in the pipeline change detection happens.

Previously, CDF operated at the storage layer. You could track which rows were inserted, updated, or deleted in a Delta table. That is useful for source and bronze-layer tracking, but it does not tell you anything about business-level results.

With CDF on Materialized Views, the change feed is generated at the output of the transformation - at the data product layer. The Materialized View itself can emit a record of what changed after each refresh.

The architecture looks like this:

Banking Transactions
        ↓
Bronze / Delta
        ↓
Business Transformations
        ↓
Materialized View
(Customer / Account Metrics)
        ↓
Change Data Feed
        ↓
Changed Business Results
        ↓
Risk | Fraud | CRM | APIs | Analytics

The distinction matters. Traditional CDC asks: which source records changed? CDF at the data product layer asks: which business result changed?

Those are fundamentally different questions. And for most downstream consumers in a banking context, the second question is the one they actually need answered.

---

A Customer 360 Example

Think about what a Customer 360 data product looks like in practice. A single customer might have hundreds of underlying transactions in a given month - card swipes, ATM withdrawals, direct deposits, loan payments, fee reversals. Each of those transactions is a source-level event.

But a downstream application - a CRM, a risk engine, a fraud alert system - does not need to process all 200 transactions. It needs to know that the customer's profile changed in a meaningful way:

• •Risk Score: 42 → 67
• •Monthly Spend: $8,500 → $11,200
• •Loan Exposure: $120K → $145K
• •Customer Segment: Preferred → High Value

With CDF on the Materialized View, the downstream system receives exactly this: a record of what changed at the business-result level, not a replay of every source transaction.

This enables a more incremental and event-oriented downstream architecture. Instead of polling the full Customer 360 table every hour, a fraud detection pipeline can subscribe to the change feed and process only the customers whose risk profiles actually shifted. A CRM workflow can trigger a relationship manager alert only when a customer's segment changes - not on every transaction.

The data product becomes a source of meaningful signals, not just a queryable snapshot.

---

Reconciliation: Focusing Work Where It Matters

One of the less-discussed benefits of change-aware data products is what they enable for reconciliation processes.

Consider a loan balance reconciliation. Yesterday, customer C1001 had a loan balance of $120,000. Today, after processing, the balance is $125,000. The change is +$5,000.

In a traditional architecture, a reconciliation job might re-process the entire loan portfolio to identify discrepancies - comparing current balances against expected values for every customer, every period. This is computationally expensive and operationally slow, especially at scale.

With a change-aware Materialized View, the reconciliation process can focus specifically on the customers and accounts whose balances actually changed in the current processing window. Instead of scanning millions of rows, the job processes the subset of records flagged by the change feed.

This does not eliminate the need for periodic full reconciliations - regulatory and audit requirements often mandate them. But it allows the high-frequency, operational reconciliation layer to be significantly more targeted and efficient.

---

CDF Is Not the Same as an Audit History

This is worth stating clearly, because it is easy to conflate the two.

CDF on a Materialized View tells you what changed during a specific refresh cycle. It is operationally useful for propagating changes to downstream consumers. It is not, by itself, a durable audit history.

Banking audit and regulatory requirements typically demand:

• •Durable historical storage - changes must be retained for defined periods, often years
• •Defined retention policies - with legal hold and deletion capabilities
• •Lineage - the ability to trace a result back through every transformation that produced it
• •Business rule and version tracking - knowing which version of a transformation logic produced a given result
• •Reproducibility - the ability to re-derive a historical result from source data
• •Access controls - fine-grained permissions on who can read historical change records

CDF does not automatically provide these things. What it provides is a mechanism to detect and distribute changes as they occur. For audit and regulatory purposes, those changes may still need to be persisted into a governed historical layer - a separate, purpose-built store with the retention, lineage, and access controls that compliance requires.

A practical architecture separates these concerns:

Materialized View
       ↓
      CDF
     /   \
Operational   Persistent
Consumers     Audit History

Operational consumers - fraud detection, CRM triggers, risk alerts - consume the change feed directly and act on it in near-real-time. The audit history layer persists the same changes into a governed store with the metadata and retention policies that compliance demands.

These are complementary, not competing. CDF is the detection and distribution mechanism. The audit layer is the governance and retention mechanism. Both are necessary in a regulated environment.

---

The Broader Architectural Shift

Data platforms have traditionally been designed around a simple pattern:

Store → Transform → Query

You ingest data, transform it into useful shapes, and expose it for querying. Change detection, when it happens at all, is bolted on as an afterthought - usually by the downstream consumer, usually in a way that does not generalize.

CDF on Materialized Views is part of a broader shift toward a different pattern:

Store → Transform → Detect Meaningful Change → Propagate

CDC addressed the change-detection problem near the ingestion layer. It made it practical to track what changed in source systems and propagate those changes incrementally through the bronze and silver layers. That was a significant architectural improvement.

CDF on transformed data products extends the same principle further into the pipeline - to the layer where business-level results live. It means that a Materialized View is not just a queryable snapshot. It is a source of change signals that downstream consumers can subscribe to and act on.

This matters because the cost of change detection should not be paid by every consumer independently. It should be paid once, at the data product layer, and the results should be available to anyone who needs them.

---

Closing Thought

A useful data product should not only answer "What is the current state?" It should also be capable of telling its consumers "What changed since you last looked?"

That second capability is not a nice-to-have. In a banking context - where risk scores shift, customer segments change, and loan exposures move - it is the difference between a data product that enables real-time decision-making and one that requires every downstream team to rebuild their own change-detection logic from scratch.

Change detection has historically been treated as an infrastructure problem that each consumer solves independently. CDF on Materialized Views makes it possible to treat change detection as part of the contract of the data product itself - something the platform provides, not something every team reinvents.

That is the architectural idea worth building toward.