Databricks Industry Data Models: From AI Coding to AI Architecture

For the last couple of years, most conversations about GenAI in data engineering have focused on one question:

How much code can AI generate?

SQL. PySpark. dbt models. Documentation. Unit tests. Pipeline configurations.

All useful.

But recently I came across Databricks' Lakehouse Industry Data Models repository, and it made me think the more interesting question may be:

How much of the architecture around the code can AI help us build?

At first, the numbers caught my attention.

The repository currently covers 40 industries, 80 models, 23,092 tables/data products, 885,842 attributes, 156,641 foreign-key relationships, and 11,661 metric views.

But after looking deeper, the 23,000 tables were no longer the most interesting part.

It was the process behind them.

Databricks describes an AI-driven approach called the Vibe Data Modeling Agent, where a business description can progressively become domains, tables, attributes, relationships, governance metadata and metrics.

That is a very different use of GenAI from simply asking an LLM to generate a CREATE TABLE statement.

And I think it points toward an important direction for enterprise data engineering:

We are moving from AI-assisted coding toward AI-assisted architecture.

---

The Traditional Data-Modeling Journey

Think about what happens when a new data platform or domain starts.

Someone writes business requirements.

Those requirements move through several layers:

• •Business Requirements
• •Business Domains
• •Conceptual Data Model
• •Logical Data Model
• •Physical Data Model
• •Tables and Columns
• •Relationships
• •DDL
• •Governance
• •Semantic Layer / Metrics

In a large organization, several different people may participate in this process.

Business analysts understand the requirements. Data architects define the domains. Data modelers create entities and relationships. Engineers implement the physical structures. Governance teams classify the data. Analytics teams define metrics.

None of this is inherently bad.

The challenge is the amount of translation happening between these stages.

And every translation creates opportunities for something to get lost.

A business concept becomes a requirement. The requirement becomes an entity. The entity becomes a table. The table becomes a pipeline. The pipeline eventually becomes a metric.

By the time somebody asks: "Where did this number actually come from?" we sometimes need lineage across half the platform to answer it.

---

What Databricks Is Experimenting With

The Databricks repository takes a different approach.

Instead of manually creating every layer independently, the modeling process starts with business intent and progressively converts it into structured metadata.

The repository describes an eight-stage modeling pipeline:

• ›Tier classification
• ›Domain generation
• ›Sub-domain expansion
• ›Product/table generation
• ›Attribute/column generation
• ›Foreign-key linking
• ›Semantic deduplication and naming
• ›Metric-view and ontology synthesis

The important word here is progressively.

This isn't: Prompt, then generate 500 tables.

It is closer to: Intent, then Domain, then Model, then Relationships, then Validation, then Semantics, then Deployable Metadata.

That architectural difference matters.

---

Generation Is Easy. Trust Is Harder.

This is where I think many enterprise GenAI implementations get into trouble.

LLMs are very good at generating things.

Ask for 100 SQL queries. You will get them. Ask for 200 test cases. You will probably get those too. Ask for an entire data model. You can get hundreds of tables within minutes.

But volume of generation isn't the same as quality of engineering.

A generated model can contain:

• •Invalid relationships
• •Duplicated concepts
• •Inconsistent naming
• •Dangling foreign keys
• •Circular dependencies
• •Questionable grain
• •Semantically similar entities represented differently

And the larger the generated output becomes, the harder manual validation becomes.

This leads to a principle I increasingly believe is important for AI-assisted data engineering:

Use AI where ambiguity exists. Use deterministic engineering where rules exist.

---

AI Should Not Validate Everything AI Generates

This is one of the things I particularly liked about the architecture described in this project.

Structural validators are placed between generation stages.

The repository describes checks around areas such as cycles, bidirectional relationships, dangling foreign keys and model fidelity.

Think about a simple example.

An LLM might infer that a CUSTOMER can have POLICY records, connected through customer_id. Determining that a customer can have policies requires understanding the business domain. AI can help with that.

But once the relationship has been defined, checking whether customer_id actually exists on both sides does not require another LLM. That's deterministic.

Similarly:

• •AI can suggest an entity
• •AI can propose attributes
• •AI can infer relationships

But software can validate:

• •Does the referenced table exist?
• •Does the referenced column exist?
• •Is there a circular dependency?
• •Is the naming convention valid?
• •Is the datatype permitted?
• •Is the relationship structurally valid?

Those questions have deterministic answers.

So instead of a simple LLM-to-output pipeline, we start getting:

Business Intent
      |
 AI Generation
      |
   Validation
      |
 AI Generation
      |
   Validation
      |
Governed Metadata

For me, this is a much more interesting enterprise AI pattern.

---

The ECM vs MVM Idea Is Equally Interesting

Databricks provides two versions of each industry model.

Expanded Coverage Model (ECM) - The ECM provides broad coverage of an industry's domains and concepts. Minimum Viable Model (MVM) - The MVM reduces that broader model into a more focused starting point while preserving important relationships.

Across the repository, Databricks reports that the MVMs contain roughly 39% of the ECM table count while retaining around 59% of the foreign-key relationships.

That tells us something important.

Enterprise data modeling isn't necessarily a "How many tables can we generate?" problem.

It's much closer to: What is the minimum architecture required to solve this business problem without losing the important relationships?

Suppose an industry reference model contains 500 entities. An organization building a claims analytics platform may only require 100 of them. An underwriting application might need a different 120. Executive reporting might need 60.

Generating the comprehensive model is one problem.

Selecting the right model for a particular business problem is another - and potentially more valuable - problem.

---

The Model Doesn't Stop at an ER Diagram

Another important part is where the generated metadata eventually goes.

The repository includes tooling for deploying these models into Unity Catalog, with artifacts including schemas, tables, relationships, governance tags and metric views.

So conceptually we can start thinking about a pipeline like this:

Business Intent
      |
Industry / Domain Model
      |
Tables + Attributes
      |
Relationships
      |
Structural Validation
      |
Governance Metadata
      |
Unity Catalog
      |
Metric Views
      |
Analytics + AI

Now the data model is no longer just documentation.

It becomes executable metadata.

And I think this is where things start getting really interesting.

---

Metadata Becomes More Important in the GenAI Era - Not Less

There is an assumption I sometimes hear that increasingly capable LLMs will reduce the need for metadata.

I think the opposite may happen.

The more AI we put into enterprise data platforms, the more context those systems require.

Consider the questions an enterprise data agent needs to answer:

• •What does this column mean?
• •What is the grain?
• •Which source created it?
• •What transformation rule produced it?
• •Which business domain owns it?
• •Is it sensitive?
• •Which metrics depend on it?
• •Which other datasets reference it?
• •What is the approved business definition?
• •Can AI modify it automatically?
• •Does changing it require human approval?

Those aren't primarily coding questions.

They are metadata questions.

Without that context, the LLM is guessing. With structured metadata, the model can operate inside boundaries.

---

Reference Models Are Not Enterprise Models

There is also an important distinction to make.

An industry reference model - even a very comprehensive one - is not the same thing as an organization's enterprise data model.

Every organization has its own source systems, business terminology, historical architecture, business keys, transformation rules, regulatory requirements, reporting definitions, data-quality rules and exceptions.

Databricks itself positions these models as accelerators and starting points rather than finished organization-specific implementations.

So I wouldn't look at this repository and conclude: "AI can now generate my enterprise data model."

I would ask a different question.

---

What Happens When Industry Metadata Meets Enterprise Metadata?

This is where I think the next step becomes particularly interesting.

Imagine taking:

• •Industry Reference Model
• •Existing Enterprise DDL
• •STTM / Mapping Specifications
• •Business Glossary
• •Data Quality Rules
• •Existing Target Models
• •Naming Standards

and providing those to an architecture agent.

Now the task isn't: "Create an insurance data model."

It becomes: "Compare our current model with an industry reference architecture. Map existing entities. Identify missing business concepts. Detect duplicated structures. Recommend extensions. Generate proposed mappings and show everything requiring human approval."

Conceptually:

Industry Model       Enterprise Metadata
      \                    /
       \                  /
        |                |
        Metadata Intelligence
                 |
           Gap Analysis
                 |
       Proposed Architecture
                 |
           Human Approval
                 |
        Governed Deployment

That is a very different proposition.

We're not replacing the data architect.

We're giving the architect a much stronger starting point.

---

The Role of the Architect Changes

I don't think the future data architect spends less time thinking.

I think the architect spends less time manually translating.

Instead of spending hours creating repetitive attributes and relationships, more time can go toward questions like:

• •Is this actually the correct business grain?
• •Should these concepts belong to the same domain?
• •Does this relationship represent how the business really operates?
• •Which model should become authoritative?
• •What should AI be allowed to generate automatically?
• •Where must humans approve the change?

That's higher-value architecture work.

---

From AI-Assisted Coding to AI-Assisted Architecture

The first wave of GenAI for data engineering was largely about productivity:

Generate SQL. Generate PySpark. Generate documentation. Generate tests.

We're now starting to see another layer emerge:

Generate domains. Propose data products. Infer relationships. Build semantic models. Generate governance metadata. Define metrics. Compare architectures.

And eventually: Reason about the architecture itself.

That doesn't mean handing architecture decisions completely to an LLM.

Quite the opposite.

The architecture I'm most interested in looks like:

AI generation + structured metadata + deterministic validation + human approval

Each component solves a different problem.

AI handles ambiguity and interpretation. Metadata provides context. Deterministic systems enforce known rules. Humans own the important architectural decisions.

---

My Takeaway

I initially opened the Databricks repository because 23,092 tables sounded interesting.

I came away thinking the tables are probably the least interesting part.

The more important pattern is:

Intent
  |
Metadata
  |
AI Generation
  |
Deterministic Validation
  |
Governance
  |
Human Approval
  |
Deployment

That same pattern could eventually apply far beyond industry data models:

STTM generation. Schema evolution. Data-quality rules. Pipeline generation. Semantic models. Metric definitions. Documentation. Impact analysis. Source-to-target mapping.

The future of AI in data engineering may not be about asking: "How much code can the LLM write?"

The better question might be:

How much enterprise knowledge can we convert into structured metadata - and how safely can AI reason over it?

That's where I think things get really interesting.

---

The Databricks Lakehouse Industry Data Models project is available on GitHub.