Practical articles on AI, Data Engineering, Metadata-Driven Development, Snowflake, Enterprise Architecture, and Modern Data Platforms.
Real-world engineering lessons, technical deep dives, and implementation guides by Amit Singh · 18 articles published
Showing 7 articles tagged #Databricks
Databricks Lakeflow Spark Declarative Pipelines now support generally available pipeline parameters, letting data engineers reuse the same transformation logic across development, QA, and production by supplying different configuration values at runtime.
A practical walkthrough for ingesting P&C claim documents into a Bronze-Silver-Gold lakehouse using Databricks ai_parse_document and ai_extract 2.1, with deterministic validation and a human-review step before data reaches Gold.
Explore how the Databricks Lakehouse Industry Data Models approach could be extended to Property & Casualty insurance, covering policy, coverage, risk, premium, claims, grain and AI-assisted data engineering.
Databricks' 23,000+ table Industry Data Models show how GenAI, metadata, deterministic validation, Unity Catalog and human governance could reshape enterprise data architecture.
Understanding agentic AI concepts is important - but the most valuable skill is engineering judgment: knowing when a prompt, Python script, or SQL query is the better answer than an agent.
How parallel query generation and multi-pivot reranking can improve enterprise knowledge retrieval beyond traditional RAG.
Banks already know how to capture changes at the transaction level. But what happens when a downstream system needs to know that a customer's risk score moved from 42 to 67 - not which five transactions caused it? That is the architectural gap CDF on Materialized Views is designed to close.