Practical articles on AI, Data Engineering, Metadata-Driven Development, Snowflake, Enterprise Architecture, and Modern Data Platforms.
Real-world engineering lessons, technical deep dives, and implementation guides by Amit Singh · 18 articles published
Showing 14 articles tagged #Data Engineering
Databricks Lakeflow Spark Declarative Pipelines now support generally available pipeline parameters, letting data engineers reuse the same transformation logic across development, QA, and production by supplying different configuration values at runtime.
A practical guide to the Snowflake Cortex Agent SDK, covering Python setup, Cortex Analyst semantic modeling, Cortex Search, streaming, background execution, security, and the governance considerations that make AI agents reliable on enterprise data.
A practical walkthrough for ingesting P&C claim documents into a Bronze-Silver-Gold lakehouse using Databricks ai_parse_document and ai_extract 2.1, with deterministic validation and a human-review step before data reaches Gold.
Explore how the Databricks Lakehouse Industry Data Models approach could be extended to Property & Casualty insurance, covering policy, coverage, risk, premium, claims, grain and AI-assisted data engineering.
Databricks' 23,000+ table Industry Data Models show how GenAI, metadata, deterministic validation, Unity Catalog and human governance could reshape enterprise data architecture.
Understanding agentic AI concepts is important - but the most valuable skill is engineering judgment: knowing when a prompt, Python script, or SQL query is the better answer than an agent.
How parallel query generation and multi-pivot reranking can improve enterprise knowledge retrieval beyond traditional RAG.
A practical pattern for moving from source-to-target mappings and business definitions to a semantic Customer 360 model that business users can query safely.
Legacy ETL modernization is more than a code conversion exercise. Translating Informatica PowerCenter mappings to Snowflake SQL requires preserving transformation intent, surfacing hidden assumptions, and ensuring human approval before any artifact is released.