Engineering notes.
What we learn building lakehouses, agents and pipelines, written for the engineers who have to run them.
A Practical Unity Catalog Model for the Enterprise
A catalog layout that survives real org charts: environment-first catalogs, domain schemas, access through groups only, and governance applied by CI rather than by hand.
DLT vs Workflows: Who Owns the Table?
Delta Live Tables and Databricks Workflows overlap less than people think. The deciding question is ownership: does the pipeline own the table, or does a job just run some tasks?
Databricks Asset Bundles: CI/CD Done Right
One databricks.yml, three targets, no clicking in production. The promotion pattern we recommend for every Databricks project, with a working GitHub Actions setup.
A Metadata-Driven Ingestion Framework in Azure Data Factory
One pipeline, hundreds of tables: a control table drives the source, load type, watermark and target path, so adding a source is a row, not a release.
Direct Lake or Databricks SQL? How to Benchmark Your Own Workload
Fabric Direct Lake and Databricks SQL both serve Power BI from Delta tables. Which one wins depends on your queries, so here is a fair way to measure it on your own data.
Migrating from Synapse Dedicated Pools to Fabric Warehouse
Inventory, wave plan, parallel run, cutover, and the T-SQL differences that actually cause trouble. A practical plan for moving off Synapse dedicated SQL pools.
Building Your First Agent on Enterprise Data with Agent Bricks
From raw documents to a governed, evaluated and deployed agent on Databricks: ingestion, chunking, vector search, tools, evaluation and monitoring, step by step.
Cost Engineering on Databricks: Where the Money Goes and the Levers That Move It
Before you optimise anything, find out what you are paying for. A practical guide to Databricks cost: system tables, compute choices, Photon, liquid clustering and guardrails.
No posts match that search. Try a different term, or clear the filter.