Data Engineering Zoomcamp — User Guide

The Data Engineering Zoomcamp is a free, 9-week, hands-on course from DataTalks.Club that teaches you to build an end-to-end data pipeline from scratch using industry-standard tools. This repository (DataTalksClub/data-engineering-zoomcamp) holds every lesson, code sample, configuration file, and homework for the course.

This guide shows you how to use the repository: how to enroll or study at your own pace, how to set up your environment, and how to work through each module's hands-on materials. It is written for learners — whether you joined a live cohort or found the repo on your own and want to teach yourself.

Note

This is a learning repository, not an installable software package. "Using" it means cloning it, following the module instructions, and running the example pipelines on your own machine and cloud account. There is no pip install; the deliverable is your own skill and a portfolio project.

1. Quick Start

If you want to start learning in the next ten minutes, here is the shortest path.

# 1. Clone the course repository
git clone https://github.com/DataTalksClub/data-engineering-zoomcamp.git
cd data-engineering-zoomcamp

# 2. Open Module 1 and start the Docker + Postgres workshop
cd 01-docker-terraform/docker-sql

Then open 01-docker-terraform/docker-sql/README.md and follow the eleven numbered lessons in order. Each module folder is self-contained and pairs a README.md with short video lectures.

Tip

The course is video-first. Every module README embeds YouTube links next to each topic. Watch the video, then run the commands from the README. The full playlist is the Course Playlist on YouTube.

What you will build

By the end of the course you will have built a complete data platform that:

• Ingests the public NYC taxi trip dataset from raw CSV files
• Containers everything with Docker
• Provisions cloud infrastructure with Terraform
• Orchestrates workflows with Kestra
• Loads data into a BigQuery data warehouse
• Transforms it with dbt
• Processes it in batch with Apache Spark
• Streams it in real time with Kafka / Flink
• Wraps up with a capstone project of your own design

2. How to Enroll

You can take the course in two ways: as part of a scheduled cohort (with deadlines, a leaderboard, and a certificate) or self-paced at any time.

Join the next cohort

After you sign up, read after-sign-up.md for next steps.

Study self-paced

All materials are public and remain available after each cohort ends. To learn independently:

1. Watch the course videos in module order.
2. Join the DataTalks.Club Slack and the #course-data-engineering channel.
3. Use the FAQ document when you get stuck.

Note

Self-paced learners can do every exercise and homework, but certificates are only issued to learners who complete a live cohort within its deadlines. See Section 11.

Prerequisites

To get the most out of the course you should have:

• Basic coding experience
• Familiarity with SQL
• Experience with Python (helpful but not required)

No prior data engineering experience is necessary.

3. Course Map

The repository is organized as one folder per module, numbered in the order you should take them. Workshops and a final project are interleaved.

Note

The dataset used throughout is the NYC Taxi & Limousine Commission (TLC) trip data. The course uses the CSV copies hosted at github.com/DataTalksClub/nyc-tlc-data, not the Parquet files on the nyc.gov site. CSV is easier for newcomers to inspect with a spreadsheet or text editor.

Past cohorts

The cohorts/ directory archives previous years (2022–2026), including homework, deadlines, and tools that have since been replaced (for example, Airflow and Mage in older cohorts). Use the current top-level module folders for the latest version; browse cohorts/ only if you want historical material or a tool a past cohort used.

4. Setting Up Your Environment

You do not need to install everything up front. Each module tells you what it needs. Install these as you reach the modules that use them.

Tip

The course standardizes on uv for Python environment and dependency management. It is fast and replaces pip + venv. Where older notes mention pip install, you can usually substitute uv add / uv run.

Clone the repository

git clone https://github.com/DataTalksClub/data-engineering-zoomcamp.git
cd data-engineering-zoomcamp

data-engineering-zoomcamp/
├── 01-docker-terraform/
├── 02-workflow-orchestration/
├── 03-data-warehouse/
├── 04-analytics-engineering/
├── 05-data-platforms/
├── 06-batch/
├── 07-streaming/
├── cohorts/          # past-year archives + current homework
└── projects/         # capstone instructions

Important

Several modules require a Google Cloud account. GCP offers a free tier and trial credits that are sufficient for the course. Always run terraform destroy and delete cloud resources when you finish a module to avoid charges. See Section 5.3.

5. Module 1 — Docker, Postgres & Terraform

Module 1 is the foundation and the most hands-on. You will containerize a Postgres database, write a Python ingestion pipeline, and provision cloud infrastructure with Terraform. Work through the eleven lessons in 01-docker-terraform/docker-sql/ in order.

5.1 Run PostgreSQL in a container

From docker-sql/04-postgres-docker.md:

docker run -it --rm \
  -e POSTGRES_USER="root" \
  -e POSTGRES_PASSWORD="root" \
  -e POSTGRES_DB="ny_taxi" \
  -v ny_taxi_postgres_data:/var/lib/postgresql \
  -p 5432:5432 \
  postgres:18

Connect to it with pgcli:

uv add --dev pgcli
uv run pgcli -h localhost -p 5432 -u root -d ny_taxi

When prompted, the password is root.

-- List tables, create one, query it
\dt
CREATE TABLE test (id INTEGER, name VARCHAR(50));
INSERT INTO test VALUES (1, 'Hello Docker');
SELECT * FROM test;
\q

Note

The default credentials are root / root and the database is ny_taxi. These values are used consistently across Modules 1 and 2, so memorize them.

5.2 Ingest the taxi data

The repository ships a ready-made ingestion script, pipeline/ingest_data.py, built with Click. It downloads a month of yellow-taxi CSV data and loads it into Postgres in chunks.

cd 01-docker-terraform/docker-sql/pipeline
uv run python ingest_data.py \
  --pg-user=root \
  --pg-pass=root \
  --pg-host=localhost \
  --pg-port=5432 \
  --pg-db=ny_taxi \
  --year=2021 \
  --month=1 \
  --target-table=yellow_taxi_data

All ingestion options:

Tip

The script reads the CSV with a progress bar and writes the table header first, then appends data chunk by chunk. If you re-run it against the same --target-table, it replaces the existing table — safe to re-run.

5.3 Add pgAdmin (web database UI)

pgcli is great for quick queries; pgAdmin is a web UI for richer browsing. Both containers must share a Docker network so they can find each other.

# Create a shared network
docker network create pg-network

# Postgres on the network, named so others can reach it
docker run -it \
  -e POSTGRES_USER="root" -e POSTGRES_PASSWORD="root" -e POSTGRES_DB="ny_taxi" \
  -v ny_taxi_postgres_data:/var/lib/postgresql \
  -p 5432:5432 --network=pg-network --name pgdatabase \
  postgres:18

# pgAdmin on the same network (in another terminal)
docker run -it \
  -e PGADMIN_DEFAULT_EMAIL="[email removed]" \
  -e PGADMIN_DEFAULT_PASSWORD="root" \
  -v pgadmin_data:/var/lib/pgadmin \
  -p 8085:80 --network=pg-network --name pgadmin \
  dpage/pgadmin4

Open http://localhost:8085, log in with [email removed] / root, then Register → Server with host pgdatabase, port 5432, user root, password root.

5.4 Dockerize the pipeline

Build the ingestion script into an image and run it on the network so it can reach the pgdatabase container (08-dockerizing-ingestion.md):

cd pipeline
docker build -t taxi_ingest:v001 .

docker run -it \
  --network=pg-network \
  taxi_ingest:v001 \
    --pg-user=root --pg-pass=root \
    --pg-host=pgdatabase --pg-port=5432 --pg-db=ny_taxi \
    --target-table=yellow_taxi_trips

Important

Inside Docker, --pg-host must be the container name (pgdatabase), not localhost. localhost inside a container refers to the container itself.

5.5 One command with Docker Compose

Instead of starting Postgres and pgAdmin by hand, the pipeline/docker-compose.yaml starts both at once:

cd 01-docker-terraform/docker-sql/pipeline
docker compose up -d   # start in background
docker compose down    # stop

This brings up pgdatabase (Postgres 18, port 5432) and pgadmin (port 8085) with named volumes so your data survives restarts.

5.6 Provision cloud infrastructure with Terraform

Module 1 closes by creating a Google Cloud Storage bucket and a BigQuery dataset with Terraform. From terraform/README.md:

# Authenticate for this session
gcloud auth application-default login

terraform init                                   # initialize
terraform plan  -var="project=<your-gcp-project-id>"   # preview
terraform apply -var="project=<your-gcp-project-id>"   # create
terraform destroy                                # tear down when done

Caution

Always run terraform destroy when you finish a module. Leaving cloud resources running can incur charges. Also never commit your service-account JSON key to Git — treat it like a password.

Windows users should follow the extra steps in terraform/windows.md.

6. Module 2 — Workflow Orchestration with Kestra

Module 2 replaces the manual ingestion script with orchestrated flows in Kestra, an open-source, YAML-based orchestrator. The eleven example flows in flows/ build up from "Hello World" to a scheduled, cloud-backed ELT pipeline with an AI assistant.

Start Kestra

cd 02-workflow-orchestration
docker compose up -d

Open the Kestra UI at http://localhost:8080. Shut down with docker compose down.

Note

The bundled docker-compose.yml runs Kestra v1.1, a dedicated Kestra Postgres, plus the same ny_taxi Postgres and pgAdmin from Module 1. Default Kestra login is [email removed] / Admin1234!.

Add the example flows

Paste each YAML into the Kestra editor, or import them all via the API:

curl -X POST -u '[email removed]:Admin1234!' \
  http://localhost:8080/api/v1/flows/import \
  -F fileUpload=@flows/04_postgres_taxi.yaml

The flows progress through the core concepts:

Tip

Section 2.5 of the module shows how to use Kestra's AI Copilot (powered by Google Gemini) to generate flows from a natural-language prompt, and compares it to plain ChatGPT to illustrate why context matters. You will need a free Gemini API key.

Warning

Your GCP_CREDS service account contains sensitive data. Keep it in Kestra's KV Store or as a Secret — never hard-code it into a flow or commit it to Git.

7. Module 3 — Data Warehousing with BigQuery

Module 3 (03-data-warehouse/) moves from a local database to a cloud data warehouse. It is mostly video + SQL, with these topics:

• Introduction to BigQuery and external vs. native tables
• Partitioning and clustering (and when to use each)
• BigQuery cost and performance best practices
• BigQuery internals
• BigQuery ML — training and deploying models with SQL

Helper scripts in extras/ download the taxi data and upload it to GCS — for example web_to_gcs.py and a progress-bar variant.

# The upload target bucket is read from an environment variable
export GCP_GCS_BUCKET="your-bucket-name"
python 03-data-warehouse/extras/web_to_gcs.py

Note

web_to_gcs.py reads the bucket name from the GCP_GCS_BUCKET environment variable (default placeholder: dtc-data-lake-bucketname). Set it to your own bucket before running.

The accompanying SQL files (big_query.sql, big_query_ml.sql) hold every query demonstrated in the videos.

8. Module 4 — Analytics Engineering with dbt

Module 4 (04-analytics-engineering/) transforms the raw warehouse data into clean analytical models with dbt. The complete dbt project lives in taxi_rides_ny/.

Choose your setup path

Follow setup/local_setup.md or setup/cloud_setup.md.

Run the dbt project

cd 04-analytics-engineering/taxi_rides_ny
dbt deps      # install packages
dbt seed      # load lookup/seed data
dbt run       # build all models
dbt test      # run data tests
dbt build     # seed + run + test in dependency order

The project is configured in dbt_project.yml:

• Models are layered staging (views) → intermediate (tables) → marts (tables).
• A dev environment samples a date window (dev_start_date = 2019-01-01, dev_end_date = 2019-02-01) so you don't process the whole dataset while developing.

Note

This module focuses on yellow and green taxi data for 2019–2020. FHV data, even if you loaded it in Module 3, is not used here.

Tip

If you hit out-of-memory errors during dbt build with DuckDB, see setup/duckdb_troubleshooting.md. New to dbt? The class notes in class_notes/ walk through every concept, and the dbt Fundamentals course is a recommended primer.

9. Module 5 — Data Platforms with Bruin

Module 5 (05-data-platforms/) introduces Bruin, a single platform that combines ingestion, transformation, orchestration, data quality, and lineage. The hands-on tutorial is a fill-in-the-blanks template:

bruin init zoomcamp my-taxi-pipeline
cd my-taxi-pipeline

You then complete the TODO-marked configuration and asset files, guided by inline comments, to build a three-layer pipeline (ingestion → staging → reports) over the NYC taxi data with DuckDB.

Common Bruin commands:

The reference notes in notes/ contain completed implementations and cover projects, pipelines, assets (SQL / Python / YAML), variables, the Bruin MCP for AI agents, and deploying to Bruin Cloud.

Tip

Bruin has a VS Code / Cursor extension and an MCP server, so you can build and query the pipeline conversationally with an AI agent (Section 5.4).

10. Modules 6 & 7 — Batch and Streaming

Module 6 — Batch Processing with Spark

Module 6 (06-batch/) teaches Apache Spark / PySpark. Install Spark for your OS first:

• Windows
• Linux
• macOS

Download the data, then run the example Spark SQL job (code/06_spark_sql.py):

# Download a year of a taxi type into data/raw/...
./06-batch/code/download_data.sh yellow 2020

# Run the Spark SQL job
python 06-batch/code/06_spark_sql.py \
  --input_green=data/raw/green/2020/ \
  --input_yellow=data/raw/yellow/2020/ \
  --output=data/report/revenue/

The module covers Spark DataFrames, Spark SQL, the anatomy of a cluster, internals of GroupBy and Joins, RDDs, running a local standalone cluster, Google Dataproc, and connecting Spark to BigQuery. The 06_spark_sql_big_query.py variant writes results straight to BigQuery.

Note

If local Spark setup is difficult, the README offers a Google Colab fallback — but try the local setup first; it is closer to real work.

Module 7 — Stream Processing

Module 7 (07-streaming/) is built around the PyFlink workshop in workshop/, which builds a real-time pipeline:

Producer (Python) → Kafka (Redpanda) → Flink → PostgreSQL

Prerequisites: Docker + Docker Compose, uv, and a SQL client (pgcli, DBeaver, pgAdmin, or DataGrip).

cd 07-streaming/workshop
docker compose up redpanda -d   # start the Kafka-compatible broker
docker compose ps               # confirm it's running

Note

The workshop uses Redpanda — a Kafka-compatible broker that needs no JVM and no ZooKeeper, so it starts in seconds with one container. Every Kafka client library works against it unchanged. Producers/consumers connect on port 9092 (external) and Flink connects on 29092 (internal).

Two optional study tracks accompany the workshop:

• theory/ — Kafka concepts with Java code examples (kafka_examples/), covering producers, consumers, Kafka Streams, KSQL, and Avro schema management.
• extras/ — supplementary Python and PyFlink examples from previous years (kafka-python, confluent-kafka, faust, pyspark).

11. Certificates

Certificates are awarded to learners who complete a live cohort within its deadlines. From certificates.md, eligibility requires:

• Submitting the required homework by the deadlines, and
• Completing and peer-reviewing the capstone project.

After the cohort ends and results are finalized, a certificate link is published that you can add to your LinkedIn profile. Self-paced learners can do all the work but are not issued certificates.

12. The Capstone Project

The projects/ folder holds the final-project instructions. You apply everything from the course to a real-world dataset of your choice:

1. Pick a dataset and a problem.
2. Build an end-to-end pipeline (ingestion → warehouse → transformation → dashboard).
3. Submit your project.
4. Peer-review other learners' projects (required for your own to count).

See projects/datasets.md for dataset suggestions if you don't have one in mind.

Tip

Reuse the course's reference stack for your capstone — Docker + Terraform + Kestra + BigQuery + dbt is a complete, employer-recognizable pipeline.

13. Getting Help & Community

The course runs on community. When you get stuck:

• Slack — Join DataTalks.Club Slack and the #course-data-engineering channel.
• Ask good questions — Follow asking-questions.md so people can help you quickly (search first, share errors and what you tried).
• FAQ — The course FAQ answers most common setup and homework questions.
• Telegram — Announcements channel.

Tip

Learn in public. learning-in-public.md explains how sharing your progress (blog posts, notes, LinkedIn) earns bonus points in a cohort and builds your portfolio. The community has contributed dozens of note sets, linked at the bottom of each module README.

For a curated list of further reading, see awesome-data-engineering.md.

14. Troubleshooting

Common issues drawn from the module READMEs and the project's GitHub issues.

Error: Kestra UI won't load — port 8080 already in use

Cause: pgAdmin or another service is already bound to port 8080. Solution: Change Kestra's port mapping in docker-compose.yml from 8080:8080 to 18080:8080, then open http://localhost:18080. Prevention: Check docker ps for port conflicts before starting a new stack.

Error: BigQueryError ... CSV table references column position 17, but line contains only 14 columns

Cause: The CSV in GCS was not fully downloaded or uploaded (a network/ transfer glitch), so its column count is short — it is not a real schema mismatch. Solution: Re-run the entire execution, including re-downloading the CSV from GitHub and re-uploading it to GCS. Prevention: Verify file sizes after upload; let downloads finish before loading.

Error: pgAdmin can't connect to Postgres

Cause: The two containers are on different Docker networks, or you used localhost as the host. Solution: Put both containers on the same network (--network=pg-network) and use the container name (pgdatabase) as the host in pgAdmin. Prevention: Prefer Docker Compose, which puts all services on one network automatically.

Error: DuckDB out-of-memory during dbt build (Module 4)

Cause: Building all models over the full dataset exceeds available memory. Solution: Follow 04-analytics-engineering/setup/duckdb_troubleshooting.md; limit the date range with the dev_start_date / dev_end_date vars. Prevention: Develop against the sampled dev window, not the full history.

Error: pip install cheat.sh-style "package not found" for course tools

Cause: Course tooling is run via Docker or uv, not always installed from PyPI directly. Solution: Use the exact commands in each module README (uv add, docker run, docker compose up). Prevention: Don't substitute your own install method until you've completed a module once as written.

Homework answer seems wrong

Cause: Homework and datasets occasionally have reported discrepancies (see open GitHub issues such as the Module 1 Question 1 thread). Solution: Check the cohort's homework file under cohorts/2026/<module>/homework.md and ask in Slack — instructors and peers clarify quickly. Prevention: Always use the current cohort's homework file, not an older year's.

15. FAQ

Q: Is the course really free? Yes. All videos, code, and materials are free and remain public after each cohort.

Q: Do I have to wait for a cohort to start? No. You can study self-paced anytime. Cohorts add deadlines, a leaderboard, and a certificate.

Q: How much does the cloud cost? GCP's free tier and trial credits cover the course if you tear down resources (terraform destroy, delete buckets/datasets) when done. Module 4's local path (DuckDB) and several workshops are entirely free.

Q: Do I need to know data engineering already? No. You need basic coding and some SQL. Python helps but isn't required.

Q: Which Python tool does the course use — pip or uv? The current materials use uv for environments and dependencies. Older cohort notes may show pip/venv; the commands are equivalent.

Q: What dataset will I work with? The NYC TLC yellow/green taxi trip data, in CSV form, from DataTalksClub/nyc-tlc-data.

Q: Can I use AWS instead of GCP? The course is GCP-focused, but Module 1 includes an AWS Terraform equivalent (terraform_with_variable_AWS/) that mirrors the GCS + BigQuery setup with S3 + Glue for learners who prefer AWS.

Q: Where are the old Airflow / Mage / Prefect materials? In cohorts/<year>/. The current course uses Kestra for orchestration; past tools are archived for reference.

Q: How do I get a certificate? Complete a live cohort's homework and capstone within the deadlines. Self-paced study does not include a certificate.

Q: Can I contribute notes or fixes? Yes. Each module README has a "Community notes" section — open a pull request to add your notes above the marker line.

Q: Who runs the course? DataTalks.Club, led by Alexey Grigorev with a team of instructors (Michael Shoemaker, Will Russell, Anna Geller, Juan Manuel Perafan, Arsalan Noorafkan, and others). It is sponsored by Kestra, Bruin, and dltHub.

16. Quick Reference

Default credentials & ports

Caution

These are development-only credentials shipped for learning. Never reuse them, or the bundled docker-compose files, for anything exposed to the internet. See the Admin Guide for hardening guidance.

Everyday commands

# Start / stop a module's stack
docker compose up -d
docker compose down

# Run a Python script in its environment
uv run python <script>.py

# Provision / tear down cloud infra
terraform apply -var="project=<your-gcp-project-id>"
terraform destroy

# Run the dbt project
dbt build

Key links

• Repository: github.com/DataTalksClub/data-engineering-zoomcamp
• Sign up: Airtable form
• Video playlist: YouTube
• FAQ: datatalks.club/faq
• Slack: datatalks.club/slack.html

For architecture, complete configuration reference, the reference stacks each module deploys, security considerations, and how the pipelines fit together, see the Admin Guide.