Practical articles on AI, Data Engineering, Metadata-Driven Development, Snowflake, Enterprise Architecture, and Modern Data Platforms.
Real-world engineering lessons, technical deep dives, and implementation guides by Amit Singh · 16 articles published
Showing 12 articles tagged #Data Engineering
A practical walkthrough for ingesting P&C claim documents into a Bronze-Silver-Gold lakehouse using Databricks ai_parse_document and ai_extract 2.1, with deterministic validation and a human-review step before data reaches Gold.
Explore how the Databricks Lakehouse Industry Data Models approach could be extended to Property & Casualty insurance, covering policy, coverage, risk, premium, claims, grain and AI-assisted data engineering.
Databricks' 23,000+ table Industry Data Models show how GenAI, metadata, deterministic validation, Unity Catalog and human governance could reshape enterprise data architecture.
Understanding agentic AI concepts is important - but the most valuable skill is engineering judgment: knowing when a prompt, Python script, or SQL query is the better answer than an agent.
How parallel query generation and multi-pivot reranking can improve enterprise knowledge retrieval beyond traditional RAG.
A practical pattern for moving from source-to-target mappings and business definitions to a semantic Customer 360 model that business users can query safely.
Legacy ETL modernization is more than a code conversion exercise. Translating Informatica PowerCenter mappings to Snowflake SQL requires preserving transformation intent, surfacing hidden assumptions, and ensuring human approval before any artifact is released.
Organizations spend millions modernizing data platforms - yet many programs still struggle. The reason is surprisingly simple: technology changes, metadata remains. Here's why metadata-driven engineering changes the conversation.
Every migration starts from scratch. Teams rebuild business logic, mappings, transformations, documentation, and test cases - again and again. The technology changes. The business logic does not. What if the real asset is the metadata?