Skip to content
Books for Data Engineers
Free sample
Menu
The BookHow These Books Are MadeBlogAbout

Series · Books for Data Engineers

Databricks for Data Engineers

A Hands-On Case Study in Production Lakehouse Engineering

For the senior data engineer who already writes production SQL and Python and wants the lakehouse mental model, Delta internals, and the architecture judgment calls that survive a design review — not a notebooks tour.

11 chapters · ~88,000 words · Lakeflow-era Databricks · DBR 17.3 LTS · hands-on case study

Join the waitlist
Status: in productionChapters 1–2 available freeWaitlist open

What you'll build

A single hands-on lakehouse (e-commerce domain), built progressively across all 11 chapters in one Unity Catalog catalog: five OLTP entities plus a click-event stream, structured bronze → silver → gold on Delta and runnable end to end on the free tier.

Bronze

Raw, as-delivered tables from a governed landing volume: customers, products, orders, order items, payments, events.

Silver

Conformed, deduplicated, tested — declarative pipelines with expectations on every layer.

Gold

Serving marts: daily_revenue, customer_orders, hourly_events — read through a SQL warehouse.

This is for you if

  • ✓You've shipped pipelines on some stack and need the "when do I use which" framework the Databricks docs never give.
  • ✓You want Delta's transaction log, streaming semantics, and serverless cost mechanics explained straight.
  • ✓You need Unity Catalog governance patterns that survive a real org chart.
  • ✓You're preparing for interviews — every chapter ends with interview-style self-check questions, backed by a model-answer appendix.

Not for you if

  • —You're looking for a first introduction to Spark, SQL, or Python.
  • —You want workspace UI click-through instructions.
  • —You need ML, MLflow, or Mosaic AI coverage — this book stays on the data engineering path.

Sample

Be first to read it

Free sample chapters go up here the day writing starts.

Email signup opens soon. Until then, write to vincent@yzcworks.com.

Contents & progress

Production ledger

Every chapter's real status — this book is written in the open.

ChTitleStatusCovers
01The Lakehouse & the Databricks PlatformIn draftThe lakehouse claim, compute and its two version channels, Unity Catalog, the Lakeflow family, and the composition framework: which tool for which workload.
02Delta Lake EssentialsOutlinedThe transaction log and ACID mechanics, time travel, schema enforcement vs. evolution, MERGE as the bronze-refresh idiom.
03The First Vertical Slice: Medallion in PracticeOutlinedBronze → silver → gold end to end: silver conformance, gold marts, queried from a SQL warehouse.
04Ingestion at Scale: Auto Loader & Lakeflow ConnectOutlinedThe ingestion decision tree, Auto Loader mechanics, schema inference and evolution, the event stream lands.
05Streaming & Incremental ProcessingOutlinedStructured Streaming's mental model, AvailableNow triggers, watermarks, foreachBatch, Change Data Feed.
06Lakeflow Spark Declarative PipelinesOutlinedStreaming tables, materialized views, expectations — and the SDP vs. Jobs-plus-notebooks vs. dbt decision framework.
07Governance & Data Quality with Unity CatalogOutlinedThe UC object model at depth, least-privilege schema design, lineage, constraints, governed quality rules.
08Orchestration with Lakeflow JobsOutlinedThe job/task/trigger model, file-arrival triggers, repair runs, and when an external orchestrator owns the DAG.
09Performance & Cost EngineeringOutlinedDBU budgeting, liquid clustering and predictive optimization, Spark UI diagnosis, query profiles.
10CI/CD & Team WorkflowOutlinedGit folders, Declarative Automation Bundles, environment strategy on one workspace, CI wiring.
11Production Operations & the Road AheadOutlinedSystem tables, incident triage and backfill runbooks, ops SLOs, and the platform's road ahead.

Pricing

Pricing

Not yet available for purchase

Sales open when the book is finished. Waitlist subscribers get 30% off during launch week.

Standard Edition

$19.90 USD, one-time

The complete book and everything that ships with it.

  • —PDF + ePub + web (HTML) edition
  • —Templates & checklists pack
  • —Prompt pack for AI-assisted Databricks work
  • —Free updates within this major version

Purchase not yet open

Join the waitlist

How delivery works

  1. 01Checkout is handled by Lemon Squeezy, which acts as the merchant of record and handles sales tax and VAT.
  2. 02Right after payment you get a receipt email with download links for every file. The web edition is a zipped HTML folder you open in any browser. The same files stay available from your Lemon Squeezy order page.
  3. 03The case-study code is a separate public repository under the MIT license, free for everyone. It opens at launch.
  4. 04Updates within this major version are uploaded to the same order, so you download them from the same place.

Buying for a team? Email vincent@yzcworks.com for a team licence. See the refund policy and terms.

Proof

Every example runs — check for yourself

The full case-study project will be a public, runnable repository under the MIT license. It opens at launch.

https://github.com/vincentyin-data

Vincent Yin

Senior Data Engineer

Vincent Yin is a senior data engineer with 18 years building data platforms across insurance and finance — enterprise data lakes, lakehouse migrations, dbt-modeled warehouses, and most recently production AI applications on the data stack. This series distills the judgment calls — not the syntax — that separate working pipelines from production-grade ones.

How these books are made →

Refund policy & FAQ

30-day, no questions asked. Read the full policy.

Join the waitlist

Email signup opens soon. Until then, write to vincent@yzcworks.com.