# Data Engineering Zoomcamp — Admin & Architecture Guide

This guide is the technical companion to the
[Data Engineering Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
repository. Where the User Guide explains how to *take* the course, this guide
explains how the course's **reference implementations** are built: the
architecture of each module's stack, the exact configuration shipped in the
repo, the security posture of the bundled environments, and the operational
commands to run and tear them down.

Audience: instructors maintaining the materials, engineers studying the
reference stacks, and anyone adapting these pipelines toward production.

> [!IMPORTANT]
> Every container stack, credential, and Terraform plan in this repository is a
> **teaching artifact tuned for local development**. The defaults prioritize
> ease of learning over security. This guide flags exactly where those defaults
> must change before any non-local use. See [Section 6 — Security](#6-security-considerations).

---

## 1. Executive Summary

The repository is a curriculum, not a deployable product, but it contains a
complete, coherent **reference data platform** assembled from open-source and
cloud-managed components. Across seven modules it stands up, in order:

| Layer | Technology | Module |
|-------|-----------|--------|
| Containerization | Docker, Docker Compose | 1 |
| Operational database | PostgreSQL 18 | 1, 2 |
| Infrastructure as Code | Terraform (GCP; AWS variant) | 1 |
| Orchestration | Kestra v1.1 | 2 |
| Data lake | Google Cloud Storage | 2, 3, 6 |
| Data warehouse | Google BigQuery | 2, 3, 4 |
| Transformation | dbt (Core + DuckDB / Cloud + BigQuery) | 4 |
| Data platform | Bruin | 5 |
| Batch compute | Apache Spark / PySpark, Dataproc | 6 |
| Streaming | Redpanda (Kafka), Apache Flink | 7 |

The unifying workload is the **NYC TLC taxi trip dataset** (yellow/green, CSV),
which flows through every module so learners see the same data ingested,
warehoused, transformed, batch-processed, and streamed.

**Repository facts** (commit `c59a5ea`):

- **406 files**, ~134 KB of analyzed source across **44 high-signal folders**.
- Polyglot: Python (Click/argparse scripts, PySpark, PyFlink), Java (Kafka
  examples), SQL (BigQuery, dbt), HCL (Terraform), YAML (Docker Compose, Kestra
  flows, dbt config).
- Primary languages by content: Jupyter Notebook + Python, with Java and YAML
  configuration.

---

## 2. Repository Architecture

### 2.1 Top-level layout

```text
data-engineering-zoomcamp/
├── 01-docker-terraform/      # Docker, Postgres, ingestion pipeline, Terraform
│   ├── docker-sql/           #   11-lesson workshop + pipeline/
│   └── terraform/            #   GCP + AWS Terraform variants
├── 02-workflow-orchestration/# Kestra compose stack + 11 example flows
│   └── flows/
├── 03-data-warehouse/        # BigQuery SQL + GCS upload helpers (extras/)
├── 04-analytics-engineering/ # dbt project (taxi_rides_ny/) + class notes
├── 05-data-platforms/        # Bruin notes + tutorial pointers
├── 06-batch/                 # PySpark code + per-OS Spark setup
├── 07-streaming/             # PyFlink workshop, Kafka theory (Java), extras
├── cohorts/                  # Archived 2022–2026 materials + current homework
└── projects/                 # Capstone instructions
```

The **current** curriculum lives in the numbered top-level folders. The
[`cohorts/`](cohorts/) tree is an append-only archive: each year captures that
cohort's homework, deadlines, and tooling (e.g., Airflow/Mage in 2022–2024,
since superseded by Kestra). When reading the repo, treat `cohorts/` as
historical unless you are looking up a specific year's homework.

> [!NOTE]
> The single declared "entry point" (`07-streaming/workshop/live/main.py`) is a
> placeholder stub (`print("Hello from streaming-workshop!")`). There is **no
> application entry point** for the repository as a whole — each module is run
> independently via its own README, Compose file, or script. This is expected
> for a curriculum repo.

### 2.2 The recurring data flow

Every module is a variation on the same pipeline shape, which is the course's
central architectural lesson:

```text
        ┌─────────────┐     ┌──────────────┐     ┌───────────────┐     ┌────────────┐
Source  │  Extract    │ --> │  Load/Stage  │ --> │  Transform    │ --> │  Serve     │
(CSV)   │  (HTTP/wget)│     │ (Postgres/   │     │ (SQL/dbt/     │     │ (BigQuery/ │
        │             │     │  GCS/Kafka)  │     │  Spark/Flink) │     │  reports)  │
        └─────────────┘     └──────────────┘     └───────────────┘     └────────────┘
```

- **Module 1** runs this locally: CSV → Postgres via a Python script.
- **Module 2** orchestrates it: Kestra flows do ETL (local Postgres) then ELT
  (GCS → BigQuery).
- **Modules 3–4** focus on the warehouse + transform stages (BigQuery, dbt).
- **Module 6** swaps the engine for distributed batch (Spark).
- **Module 7** makes it real-time (Kafka → Flink → Postgres).

A defining teaching point is the **ETL vs. ELT** distinction (Module 2.4): load
*before* transform when the warehouse can do the heavy lifting at scale.

---

## 3. Module Reference Stacks

This section documents each module's deployable stack as configured in the repo.

### 3.1 Module 1 — Docker + Postgres + Terraform

**Local data stack** ([`docker-sql/pipeline/docker-compose.yaml`](01-docker-terraform/docker-sql/pipeline/docker-compose.yaml)):

| Service | Image | Ports | Credentials | Volume |
|---------|-------|-------|-------------|--------|
| `pgdatabase` | `postgres:18` | `5432:5432` | `root` / `root`, DB `ny_taxi` | `ny_taxi_postgres_data:/var/lib/postgresql` |
| `pgadmin` | `dpage/pgadmin4` | `8085:80` | `[email removed]` / `root` | `pgadmin_data:/var/lib/pgadmin` |

**Ingestion pipeline** ([`pipeline/ingest_data.py`](01-docker-terraform/docker-sql/pipeline/ingest_data.py)):
a Click CLI that streams a month of taxi CSV from
`github.com/DataTalksClub/nyc-tlc-data` and chunk-loads it via SQLAlchemy
(`postgresql+psycopg://…`) with a tqdm progress bar. It declares an explicit
`dtype` map (all 16 taxi columns) and parses `tpep_pickup_datetime` /
`tpep_dropoff_datetime` as dates. Default chunk size is 100,000 rows; the first
chunk writes the header with `if_exists='replace'`, subsequent chunks
`append`.

**Container image** ([`pipeline/Dockerfile`](01-docker-terraform/docker-sql/pipeline/Dockerfile)):

```dockerfile
FROM python:3.13.11-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/
WORKDIR /code
ENV PATH="/code/.venv/bin:$PATH"
COPY pyproject.toml .python-version uv.lock ./
RUN uv sync --locked
COPY ingest_data.py .
ENTRYPOINT ["python", "ingest_data.py"]
```

> [!NOTE]
> Dependency management uses **`uv`** with a committed `uv.lock` and
> `uv sync --locked` for reproducible builds. The slim Python 3.13 base plus
> copy-lock-first layering keeps the image small and cache-friendly. (The
> module's `08` lesson shows an `ENTRYPOINT` variant using `uv run python`; the
> committed Dockerfile invokes `python` directly since `uv sync` populates the
> venv on `PATH`.)

**Terraform** ([`terraform/terraform/terraform_with_variables/`](01-docker-terraform/terraform/terraform/terraform_with_variables/)):

- Provider: `hashicorp/google` (pinned `5.6.0`; the basic example pins
  `4.51.0`). Authenticates via a service-account JSON file (`var.credentials`)
  or `GOOGLE_APPLICATION_CREDENTIALS`.
- Resources: `google_storage_bucket` (force-destroy, 1-day abort-incomplete-
  upload lifecycle) and `google_bigquery_dataset`.
- Variables: `project`, `region` (default `us-central1`), `location` (default
  `US`), `bq_dataset_name` (`demo_dataset`), `gcs_bucket_name`,
  `gcs_storage_class` (`STANDARD`).

An **AWS equivalent**
([`terraform_with_variable_AWS/`](01-docker-terraform/terraform/terraform/terraform_with_variable_AWS/))
mirrors the GCP design for learners on AWS:

| GCP | AWS equivalent |
|-----|----------------|
| GCS bucket | S3 bucket (`hashicorp/aws ~> 5.0`) |
| Uniform bucket-level access | S3 Public Access Block (all four blocks on) |
| Object lifecycle (age 30) | S3 lifecycle expiration (30 days) |
| BigQuery dataset | AWS Glue Catalog database |

It also enables S3 versioning and defaults to region `eu-north-1`.

### 3.2 Module 2 — Kestra orchestration

**Stack** ([`02-workflow-orchestration/docker-compose.yml`](02-workflow-orchestration/docker-compose.yml)):

| Service | Image | Ports | Notes |
|---------|-------|-------|-------|
| `kestra` | `kestra/kestra:v1.1` | `8080:8080`, `8081:8081` | `server standalone`, runs as `root` to reach the Docker socket |
| `kestra_postgres` | `postgres:18` | — | Kestra metadata DB (`kestra` / `k3str4`), health-checked |
| `pgdatabase` | `postgres:18` | `5432:5432` | Reused `ny_taxi` DB (`root`/`root`) |
| `pgadmin` | `dpage/pgadmin4` | `8085:80` | `[email removed]` / `root` |

Kestra's `KESTRA_CONFIGURATION` (embedded YAML) wires a Postgres-backed
repository and queue, local storage at `/app/storage`, basic auth
(`[email removed]` / `Admin1234!`), and a tmp working dir under
`/tmp/kestra-wd`. It bind-mounts `/var/run/docker.sock` so task runners can
spawn containers.

> [!WARNING]
> The Kestra service runs as **`root`** and mounts the host Docker socket. The
> compose comment notes this is for development only; production should use a
> rootless setup (e.g.
> [Podman Compose](https://kestra.io/docs/installation/podman-compose)). A
> mounted Docker socket is effectively host-root access.

**Flow catalog** ([`flows/`](02-workflow-orchestration/flows/)) — eleven YAML
flows demonstrating the full feature surface:

| Flow | Concepts |
|------|----------|
| `01_hello_world` | tasks, inputs, variables, outputs, schedule trigger (daily 10:00), plugin defaults (ERROR log level), concurrency limit 2 |
| `02_python` | running Python; returning task outputs (DockerHub pull count) |
| `03_getting_started_data_pipeline` | HTTP extract → Python transform → DuckDB query |
| `04_postgres_taxi` | partitioned CSV → monthly tables → merge to final (local Postgres) |
| `05_postgres_taxi_scheduled` | cron schedule (09:00 UTC) + backfill (green 2019) |
| `06_gcp_kv` | KV Store values: `GCP_PROJECT_ID`, `GCP_LOCATION`, `GCP_BUCKET_NAME`, `GCP_DATASET` |
| `07_gcp_setup` | create GCS bucket + BigQuery dataset |
| `08_gcp_taxi` | ELT: extract → GCS upload → external/monthly/merge BigQuery tables → purge |
| `09_gcp_taxi_scheduled` | scheduled full-dataset backfill (green 09:00, yellow 10:00 UTC) |
| `10_chat_without_rag` | LLM prompt without context (baseline) |
| `11_chat_with_rag` | ingest docs → embeddings (KV Store) → RAG-grounded answer |

Module 2.5 adds an **AI layer**: Kestra's AI Copilot configured for Google
Gemini (`gemini-2.5-flash`, `api-key: ${GEMINI_API_KEY}`) plus a RAG demo. This
is the architectural lesson that **context engineering** (Copilot/RAG) beats an
un-grounded LLM for generating correct, current flow code.

### 3.3 Module 3 — BigQuery warehouse

No long-running services; the module is SQL + GCS loaders. Helper scripts in
[`extras/`](03-data-warehouse/extras/) (`web_to_gcs.py`,
`web_to_gcs_with_progress_bar.py`) download taxi CSVs and upload to GCS using
the Google Cloud Storage client. The destination bucket is read from the
`GCP_GCS_BUCKET` environment variable (placeholder default
`dtc-data-lake-bucketname`). SQL files (`big_query.sql`, `big_query_ml.sql`)
hold the partitioning/clustering and BigQuery ML demonstrations.

Architecturally the module teaches: external tables over GCS, **partitioning**
(prune by date) vs **clustering** (co-locate by column), cost control, and
in-warehouse ML via `CREATE MODEL`.

### 3.4 Module 4 — dbt transformation

**Project** ([`taxi_rides_ny/dbt_project.yml`](04-analytics-engineering/taxi_rides_ny/dbt_project.yml)):

- `name: taxi_rides_ny`, `profile: taxi_rides_ny`,
  `require-dbt-version: [">=1.7.0", "<3.0.0"]`.
- Layered materializations: `staging` → **view**, `intermediate` → **table**,
  `marts` → **table**.
- Dev sampling vars: `dev_start_date = 2019-01-01`, `dev_end_date = 2019-02-01`.
- `flags: require_generic_test_arguments_property: true` (modern dbt test
  syntax).

Two execution targets share one project: **local** (DuckDB + dbt Core, zero
cost) and **cloud** (BigQuery + dbt Cloud). The model tree under `models/`
(`staging/`, `intermediate/`, `marts/`, `marts/reporting/`) plus `seeds/`,
`macros/`, `snapshots/`, and `tests/` is a canonical dbt project structure.

> [!NOTE]
> Profiles are environment-specific and **not committed** in a usable form —
> learners create `~/.dbt/profiles.yml` (or use dbt Cloud's connection). The
> setup guides (`setup/local_setup.md`, `setup/cloud_setup.md`) provide the
> exact profile blocks.

### 3.5 Module 5 — Bruin data platform

Bruin is introduced via its hosted template (`bruin init zoomcamp
my-taxi-pipeline`) rather than committed code; the repo carries reference
**notes** ([`notes/`](05-data-platforms/notes/)) for completed implementations.
Architecture concepts: a **project** (`.bruin.yml` with environments and
connections) contains **pipelines** (one schedule each) made of **assets** (SQL,
Python, or YAML files that each produce a table/view). A three-layer pipeline
(ingestion → staging → reports) over DuckDB mirrors the dbt project's shape but
within a single integrated tool that also covers ingestion, quality, lineage,
and orchestration. The Bruin **MCP** server exposes the platform to AI agents.

### 3.6 Module 6 — Spark batch

No bundled cluster; learners install Spark locally (per-OS guides in
[`setup/`](06-batch/setup/)) or use Dataproc. The reference job
[`code/06_spark_sql.py`](06-batch/code/06_spark_sql.py) is an argparse CLI
(`--input_green`, `--input_yellow`, `--output`) that reads Parquet, renames the
green/yellow timestamp columns to a common schema, registers temp views, and
runs a revenue aggregation. `06_spark_sql_big_query.py` is the same job writing
to BigQuery via the Spark-BigQuery connector. [`code/download_data.sh`](06-batch/code/download_data.sh)
bulk-downloads a year of a taxi type into `data/raw/<type>/<year>/<month>/`.

Architecture coverage: cluster anatomy (driver/executors), GroupBy and Join
internals (shuffles), RDDs, a local standalone cluster, Dataproc submission, and
GCS/BigQuery connectivity.

### 3.7 Module 7 — Streaming (Redpanda + Flink)

**Workshop stack** ([`workshop/docker-compose.yml`](07-streaming/workshop/docker-compose.yml)):
a `redpanda` broker (`redpandadata/redpanda:v25.3.9`) plus Flink (JobManager/
TaskManager via `Dockerfile.flink`, with an ARM64 variant) and Postgres as the
sink. Target topology:

```text
Producer (Python, kafka-python) → Redpanda (Kafka) → Flink (PyFlink) → PostgreSQL
```

Redpanda exposes dual listeners — internal `redpanda:29092` (other containers,
e.g. Flink) and external `localhost:9092` (host producer/consumer) — because
Kafka clients bootstrap then reconnect to advertised addresses. Resource flags
(`--smp 1`, `--reserve-memory 0M`, `--overprovisioned`) tune it for a laptop.

Two reference tracks accompany the workshop:

- [`theory/java/kafka_examples/`](07-streaming/theory/java/kafka_examples/) — a
  Gradle Java project (`kafka-clients 3.3.1`, `kafka-streams`, Confluent Avro
  serializers `7.3.1`, `avro 1.11.0`, JUnit 5) demonstrating producers,
  consumers, Streams topologies, custom SerDes, and Avro schema management with
  Schema Registry.
- [`extras/`](07-streaming/extras/) — prior-year Python/PyFlink examples
  (`kafka-python`, `confluent-kafka`, `faust`, `pyspark`) and a PyFlink job
  (`apache-flink==1.16.0`).

---

## 4. System Requirements

| Requirement | Detail |
|-------------|--------|
| **Docker + Docker Compose** | Required for Modules 1, 2, 7 (Postgres, pgAdmin, Kestra, Redpanda, Flink). |
| **Python 3.x + `uv`** | Ingestion scripts, PySpark, PyFlink, helper scripts. The pipeline image pins `python:3.13.11-slim`. |
| **Terraform** | Module 1 cloud provisioning. |
| **GCP account** | Modules 1–4, 6 — GCS, BigQuery, Dataproc. Free tier sufficient. AWS variant available for Module 1. |
| **Java/Gradle** | Only for the optional Module 7 Kafka theory examples. |
| **Apache Spark** | Module 6 — local install or Dataproc. Colab fallback documented. |
| **Memory** | DuckDB `dbt build` (Module 4) and local Spark (Module 6) are the most memory-hungry steps; date-range sampling mitigates. |

---

## 5. Configuration Mastery

### 5.1 Credentials and connection settings (as shipped)

| Setting | Value | Source file |
|---------|-------|-------------|
| Taxi Postgres user / pass / db | `root` / `root` / `ny_taxi` | `docker-sql/pipeline/docker-compose.yaml` |
| Postgres port | `5432` | same |
| pgAdmin email / pass | `[email removed]` / `root` | same |
| pgAdmin port | `8085:80` | same |
| Kestra basic auth | `[email removed]` / `Admin1234!` | `02-…/docker-compose.yml` |
| Kestra metadata DB | `kestra` / `k3str4` | same |
| Kestra UI / mgmt ports | `8080`, `8081` | same |
| Redpanda Kafka ports | `9092` (ext), `29092` (int), `8082`/`28082` (proxy) | `07-…/workshop/docker-compose.yml` |
| Gemini model / key | `gemini-2.5-flash` / `${GEMINI_API_KEY}` | Module 2.5 README |
| GCS bucket env var | `GCP_GCS_BUCKET` | `03-…/extras/web_to_gcs.py` |

### 5.2 Terraform variables

| Variable | Default | Notes |
|----------|---------|-------|
| `project` | `<Your Project ID>` | Required; pass via `-var` |
| `region` | `us-central1` | |
| `location` | `US` | Bucket + dataset location |
| `bq_dataset_name` | `demo_dataset` | |
| `gcs_bucket_name` | `terraform-demo-terra-bucket` | Must be globally unique |
| `gcs_storage_class` | `STANDARD` | |
| `credentials` | `<Path to SA json>` | Or use `GOOGLE_APPLICATION_CREDENTIALS` |
| (AWS) `aws_region` | `eu-north-1` | |
| (AWS) `bucket_name` | `data-engineering-zoomcamp-…` | Globally unique S3 name |

### 5.3 Version pins worth noting

| Component | Pinned version | Where |
|-----------|---------------|-------|
| Postgres | `18` | All compose files (current modules) |
| Kestra | `v1.1` | Module 2 compose + README ("pin for reproducibility; do NOT use `:develop`") |
| Redpanda | `v25.3.9` | Module 7 workshop |
| Python (image) | `3.13.11-slim` | Module 1 Dockerfile |
| google provider | `5.6.0` / `4.51.0` | Terraform variants |
| dbt | `>=1.7.0,<3.0.0` | dbt_project.yml |
| Kafka clients (Java) | `3.3.1` | build.gradle |
| Confluent serializers | `7.3.1` | build.gradle |
| PyFlink | `1.16.0` | extras requirements |

> [!TIP]
> The README explicitly warns against using `kestra/kestra:develop` — it is a
> bleeding-edge build that may break flows. Pinning is a deliberate
> reproducibility practice the course models throughout.

---

## 6. Security Considerations

This repository's stacks are intentionally insecure for learning convenience.
The following are the concrete issues an admin must address before any exposure
beyond `localhost`.

### 6.1 Hard-coded development credentials

Every bundled service ships with trivial, public credentials (`root`/`root`,
`[email removed]`/`root`, `[email removed]`/`Admin1234!`, `kestra`/`k3str4`).
These are fine on an isolated laptop and **must never** be reused on any
network-reachable host.

> [!CAUTION]
> Do not deploy any of these `docker-compose` files to a public VM as-is.
> Bound services (Postgres 5432, pgAdmin 8085, Kestra 8080, Redpanda 9092) would
> be exploitable immediately with the published passwords.

### 6.2 Docker socket exposure (Kestra)

The Kestra service runs as `root` and mounts `/var/run/docker.sock`. This grants
the container full control of the host's Docker daemon — equivalent to host
root. The compose file documents this and points to a rootless production setup.

### 6.3 Service-account key handling

Multiple modules use a GCP service-account JSON key. The materials repeatedly
warn:

- **Never commit** the key to Git (use a proper `.gitignore`; Module 1
  references the GitHub Terraform `.gitignore`).
- In Kestra, store credentials in the **KV Store** or as **Secrets**, not inline
  in flow YAML.
- For Terraform, a documented **fallback** uses short-lived **impersonation
  tokens** (`google_service_account_access_token`, 3600s lifetime) instead of a
  downloaded key — the more secure pattern when org policy forbids key creation.

### 6.4 Secrets in AI workflows

Module 2.5 uses a Gemini API key via `${GEMINI_API_KEY}` (environment variable),
and the README warns against committing API keys — use env vars or the KV Store.

### 6.5 Cloud cost as a security/ops concern

`force_destroy = true` on buckets and `terraform destroy` discipline are
emphasized to prevent orphaned, billable cloud resources. Backfilling the full
taxi dataset is explicitly deferred to the cloud (Module 2.4) precisely because
it would exhaust local resources.

---

## 7. Operations

### 7.1 Running each stack

```bash
# Module 1 — local Postgres + pgAdmin
cd 01-docker-terraform/docker-sql/pipeline && docker compose up -d

# Module 2 — Kestra + Postgres
cd 02-workflow-orchestration && docker compose up -d   # UI: http://localhost:8080

# Module 7 — Redpanda broker (then Flink services)
cd 07-streaming/workshop && docker compose up redpanda -d
```

### 7.2 Importing Kestra flows programmatically

```bash
for f in flows/*.yaml; do
  curl -X POST -u '[email removed]:Admin1234!' \
    http://localhost:8080/api/v1/flows/import -F "fileUpload=@$f"
done
```

### 7.3 Cloud provisioning lifecycle

```bash
gcloud auth application-default login
terraform init
terraform plan  -var="project=$GCP_PROJECT_ID"
terraform apply -var="project=$GCP_PROJECT_ID"
# ... do the module work ...
terraform destroy
```

### 7.4 Teardown checklist

- [ ] `docker compose down` for every started stack (add `--volumes` to wipe
      data).
- [ ] `terraform destroy` for any provisioned cloud infra.
- [ ] Delete GCS buckets / BigQuery datasets created outside Terraform (e.g. via
      Kestra `07_gcp_setup`).
- [ ] Remove `pg-network` if created manually: `docker network rm pg-network`.
- [ ] Revoke or delete any temporary service-account keys.

### 7.5 Health and verification

- Kestra metadata Postgres has a `pg_isready` healthcheck with `depends_on`
  conditions sequencing startup.
- Verify Redpanda with `docker compose ps` (expect `STATUS: Up`).
- For pgAdmin↔Postgres, connectivity depends on both being on the same Docker
  network and using the **container name** as host.

---

## 8. Code Patterns & Quality

The repository models several good engineering practices that are themselves
teaching content:

- **Locked, reproducible builds** — `uv sync --locked` with committed
  `uv.lock`; copy-lockfiles-first Docker layering.
- **Explicit schemas** — `ingest_data.py` declares an exact `dtype` map and
  date columns rather than relying on inference, preventing silent type drift.
- **Chunked, idempotent loading** — header-first `replace` then `append`, so
  re-runs are safe.
- **Pinned dependencies everywhere** — images, providers, plugins, and the
  Kestra version are all pinned for reproducibility.
- **Layered transformation** — dbt staging(view)→intermediate(table)→
  marts(table) and Bruin ingestion→staging→reports both teach the medallion
  pattern.
- **Dev sampling** — dbt date-window vars and partial backfills keep local
  iteration cheap.
- **Dual-cloud parity** — the GCP and AWS Terraform variants document a
  service-by-service mapping, reinforcing cloud-agnostic concepts.

### Known data/code issues (from GitHub)

The project's issue tracker surfaces real, learner-reported items an admin or
contributor should be aware of:

| Issue | Summary |
|-------|---------|
| #846 | Reported wrong answer for Module 1 homework Q1 |
| #737 | Duplicate `delivery_report()` definition in `06-streaming/python/avro_example/producer.py` |
| #591 | YouTube thumbnails/links not always rendering in READMEs |
| #843, #845, #847 | Certificate / setup questions (support, not defects) |

> [!NOTE]
> Most open issues are support questions and homework clarifications rather than
> code defects — consistent with a curriculum repo whose "users" are learners.

---

## 9. Potential Improvements

For maintainers and contributors adapting these materials:

- **Consolidate duplicated compose definitions.** Postgres/pgAdmin service
  blocks are repeated across Modules 1, 2, and cohort archives with slight
  drift (named volume vs. bind mount, `version:` key present/absent). A shared
  base or documented canonical block would reduce divergence.
- **Externalize credentials even for local dev.** Shipping a `.env.example`
  with the `root`/`root`/`Admin1234!` values referenced via `${VAR}` would let
  learners change them in one place and build the habit early.
- **Fix the streaming entry-point stub.** `07-streaming/workshop/live/main.py`
  is a placeholder; the live workshop builds the real producer/consumer
  elsewhere. Either wire it up or rename it to avoid confusion.
- **Address the duplicate `delivery_report()` (#737)** in the Avro example.
- **Normalize README video-embed markup (#591)** so thumbnails render reliably.
- **Default Terraform to the impersonation-token pattern** rather than a
  downloaded key, making the more secure path the default rather than the
  fallback.

---

## 10. Quick Reference

### Ports

| Port | Service | Module |
|------|---------|--------|
| 5432 | PostgreSQL (`ny_taxi`) | 1, 2 |
| 8080 | Kestra UI | 2 |
| 8081 | Kestra management | 2 |
| 8085 | pgAdmin | 1, 2 |
| 9092 / 29092 | Redpanda Kafka (ext / int) | 7 |
| 8082 / 28082 | Redpanda HTTP proxy | 7 |

### Key files

| File | Purpose |
|------|---------|
| `01-docker-terraform/docker-sql/pipeline/ingest_data.py` | Click CLI taxi ingester |
| `01-docker-terraform/docker-sql/pipeline/docker-compose.yaml` | Local Postgres + pgAdmin |
| `01-docker-terraform/terraform/.../main.tf` + `variables.tf` | GCS + BigQuery IaC |
| `02-workflow-orchestration/docker-compose.yml` | Kestra + Postgres stack |
| `02-workflow-orchestration/flows/*.yaml` | 11 example orchestration flows |
| `04-analytics-engineering/taxi_rides_ny/dbt_project.yml` | dbt project config |
| `06-batch/code/06_spark_sql.py` | PySpark revenue job |
| `07-streaming/workshop/docker-compose.yml` | Redpanda + Flink + Postgres |

### Commands

```bash
docker compose up -d / down [--volumes]   # per-module stacks
terraform init / plan / apply / destroy   # cloud infra
dbt deps / seed / run / test / build      # transformations
uv sync --locked ; uv run python <s>.py   # Python envs
./06-batch/code/download_data.sh <type> <year>   # bulk data download
```

---

_Generated June 2026 from DataTalksClub/data-engineering-zoomcamp at c59a5ea. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._