Data Engineering
Lab Curriculum.

BUG FOUND: engineers who can write a pipeline once, but cannot keep one reliable.

$ step-01 brushup // weeks 1–2 — Topics taught:

1. Python for data: pandas basics, file and API ingestion.
2. SQL for pipelines: staging tables, incremental loads, and upserts.
3. Pipeline structure: extract, transform, load, and idempotency.
4. Scheduling and orchestration basics; retries and failure handling.
5. Schema basics: types, nullability, and schema evolution.
6. Logging, monitoring, and reading pipeline logs.

$ step-02 debug // weeks 3–4 — Production Debugging. Tools: Python, SQL, Airflow-style scheduler / cron logs, a sample warehouse.

• BUG #1: Nightly job fails — read the log, find the bad record / connection error, fix and rerun idempotently.
• BUG #2: Duplicate records after a rerun — find the non-idempotent load, fix the upsert key, deduplicate history.
• BUG #3: Missing partition for one day — find the date / timezone bug, backfill correctly, verify row counts.
• BUG #4: Schema change breaks downstream — adapt the pipeline, version the schema, notify downstream with a change note.

$ step-03 navigate // weeks 5–7 — Codebase / System Navigation:

• Map lineage for an existing pipeline set: sources, transforms, destinations, dependencies.
• Make a schema change safely: update code, tests, and downstream expectations, with a rollback plan.
• Add monitoring and alerts (row-count, freshness, null-rate checks) to an unmonitored pipeline.
• Optimise a slow job: find the bottleneck (scan, join, or serial step) and show before/after runtime.

$ step-04 ship // weeks 8–9 — Shipping Features: capstone — a reliable, monitored pipeline with data-quality checks, a runbook, and a failure drill (break it, recover it, document it).

$ ship → PROOF — exact output: pipeline code, monitoring dashboard / alert config, data-quality report, and a runbook including the failure-drill result.

$ run --lab data-engineering → ✓ SHIPPED, job-ready