Volume 4 — Data Engineering¶
Moving data reliably
Where data comes from and how it gets somewhere useful: SQL and databases, file formats, batch and streaming pipelines, orchestration, modelling and data quality.
Partly available
Some lessons in this volume are written; the remainder are outlined below and not yet authored.
What you will be able to do¶
- Write correct, performant SQL against a real database
- Choose the right file format for a workload and justify it
- Build a reproducible pipeline with tests
- Diagnose a data quality incident methodically
Chapters¶
- SQL fundamentals for engineers
- Databases and SQL with Python
- HTTP and consuming REST APIs
- File formats: CSV, JSON, Parquet, Arrow
- DuckDB for local analytics
- ETL vs ELT and why the order changed
- Orchestration and scheduling
- Data modelling for analytics
- Data quality, contracts and testing
- Streaming basics
Lessons available now¶
These lessons come from the Version 1 course and are complete.
HTTP, REST APIs & Databases