# Apache Airflow User Guide

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. You define your pipelines as Python code; Airflow's scheduler then runs your tasks on a pool of workers while honoring the dependencies you declared. A rich web UI lets you visualize pipelines in production, watch progress, inspect logs, and retry or backfill work when something goes wrong.

This guide is written for the people who **use** Airflow day to day: data engineers and analysts who author DAGs, and operators who trigger, monitor, and troubleshoot runs from the command line or the web interface. It covers installation, core concepts, the complete `airflow` command-line interface, the everyday tasks you perform against a running environment, and a deep troubleshooting and FAQ reference.

> [!NOTE]
> Architecture deep dives, the full `airflow.cfg` configuration reference, deployment topologies, executor selection, database tuning, and security hardening live in the companion **Admin Guide**. This guide links there whenever a topic crosses into operations.

---

## 1. What Airflow Is (and Is Not)

Airflow orchestrates **batch** workflows. You express a pipeline as a **DAG** (Directed Acyclic Graph) — a collection of **tasks** with directional dependencies — written in Python. The scheduler reads your DAG files, decides which task instances are ready to run, and hands them to an **executor**, which runs them on workers. Everything that happens is recorded in a metadata database and surfaced in the UI and REST API.

Airflow's design philosophy, taken directly from the project:

- **Dynamic** — Pipelines are defined in Python, so DAGs can be generated and parameterized in code.
- **Extensible** — A large library of built-in operators ships in *provider* packages, and you can write your own.
- **Flexible** — Airflow leans on the [Jinja](https://jinja.palletsprojects.com) templating engine for rich, runtime-parameterized commands.

> [!IMPORTANT]
> Airflow works best with workflows that are **mostly static and slowly changing** — where the DAG structure is similar from one run to the next. Tasks should be **idempotent** (re-running produces the same result without duplicating data) and should not pass large volumes of data between each other. For passing small bits of metadata between tasks, use Airflow's **XCom** feature; for high-volume data, delegate to an external system (a warehouse, Spark, etc.) and let Airflow orchestrate it.

> [!CAUTION]
> Airflow is **not** a streaming solution. It is frequently used to process near-real-time data by pulling from streams in batches, but it is not an event-by-event stream processor.

### Versioning at a glance

Airflow follows strict [SemVer](https://semver.org/). Crucially, **core Airflow and each provider version independently**: an `amazon` or `google` provider release has its own MAJOR/MINOR cycle and can be installed against multiple Airflow versions. The Helm chart and the language API clients also version independently. This guide documents Airflow core **3.3.0**.

---

## 2. The Airflow Family at a Glance

Airflow is developed as a single repository that ships **several separately installed, separately consumed products**. As a user you will almost always start with the core platform; the rest are extensions you add when you need them.

| Product | PyPI / artifact | What it is | When you need it |
|---|---|---|---|
| **Apache Airflow (core)** | `apache-airflow` | The scheduler, API server, DAG processor, triggerer, UI, and the `airflow` CLI | Always — this is the platform |
| **Providers** (87 of them) | `apache-airflow-providers-*` | Operators, hooks, sensors, transfers, secrets backends, and executors for external systems (AWS, Google, Postgres, Slack, Kubernetes, …) | Whenever a task talks to an external system |
| **Task SDK** | `apache-airflow-task-sdk` | The stable, lightweight authoring API (`airflow.sdk`) used inside your DAG and task code | Authoring DAGs against a forward-compatible interface |
| **Airflow CLI (remote)** | `apache-airflow-ctl` (`airflowctl`) | A standalone client that drives a remote Airflow instance over its REST API | Operating a remote/managed Airflow from your laptop or CI |
| **Helm chart** | `apache-airflow` chart | Production Kubernetes deployment | Running Airflow on Kubernetes (see Admin Guide) |
| **Edge SDKs** | `go-sdk`, `java-sdk` | Experimental SDKs for running tasks on edge workers in Go and Java/Kotlin | Polyglot/edge task execution (advanced) |

> [!TIP]
> Most readers only need three of these: **core** (run the platform), **providers** (connect to your systems), and the **Task SDK** (author your DAGs). Start there. The remainder of this guide is organized that way.

---

## 3. Quick Start

The fastest way to see Airflow run end to end is the all-in-one `standalone` command. It initializes a local SQLite metadata database, creates an admin user, and starts the scheduler, the API server (with the UI), the DAG processor, and the triggerer together in one process group.

=== "pip (virtualenv)"

    ```bash
    # 1. Create and activate a virtual environment (Python 3.10-3.14)
    python -m venv airflow-venv
    source airflow-venv/bin/activate

    # 2. Install Airflow with a known-good constraint set
    pip install 'apache-airflow==3.3.0' \
      --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt"

    # 3. Choose where Airflow keeps its home directory (dags, logs, config, db)
    export AIRFLOW_HOME=~/airflow

    # 4. Run everything at once
    airflow standalone
    ```

=== "Docker image"

    ```bash
    # Pull the official image and run the all-in-one quick start
    docker run -it --rm -p 8080:8080 \
      apache/airflow:3.3.0 standalone
    ```

When `standalone` finishes booting it prints the generated admin credentials:

```text
standalone | Airflow is ready
standalone | Login with username: admin  password: <generated-password>
```

Open <http://localhost:8080> in your browser, log in with those credentials, and you will see the **Dags** view populated with the bundled example DAGs.

> [!WARNING]
> `airflow standalone` is for **local exploration and development only**. It uses SQLite and a single-process LocalExecutor, neither of which is safe for production. For real deployments, see the Admin Guide's deployment chapter.

> [!TIP]
> The admin password is written to `$AIRFLOW_HOME/simple_auth_manager_passwords.json.generated`. If you lose the printed value, read it from there.

---

## 4. Installation

### 4.1 Requirements

Airflow 3.3.0 is tested against the following. POSIX-compliant operating systems only — on Windows, run Airflow inside WSL2 or a Linux container.

| Component | Supported versions |
|---|---|
| **Python** | 3.10, 3.11, 3.12, 3.13, 3.14 |
| **Platform** | AMD64 / ARM64 |
| **Kubernetes** | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 |
| **PostgreSQL** | 14, 15, 16, 17, 18 |
| **MySQL** | 8.0, 8.4, Innovation |
| **SQLite** | 3.15.0+ (development/testing only) |

> [!CAUTION]
> SQLite and the default LocalExecutor are fine for trying Airflow but must **not** be used in production. MariaDB is explicitly **not** tested or recommended. Use PostgreSQL or MySQL for any shared or production environment.

### 4.2 Installing from PyPI with constraints

Airflow is simultaneously a *library* (you import it in your DAGs) and an *application* (you run its services). Because of that dual nature, its dependencies are kept deliberately open in `pyproject.toml`, which means a bare `pip install apache-airflow` can occasionally resolve to an unusable combination. The project therefore publishes **constraint files** — pinned, known-to-work dependency sets — per Airflow version and per Python minor version.

> [!IMPORTANT]
> Always install Airflow **with a `--constraint` URL**. The URL encodes both the Airflow version and your Python version. Only `pip` is officially supported; Poetry and pip-tools are not.

```bash
# Plain core install, Python 3.10
pip install 'apache-airflow==3.3.0' \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt"
```

```text
Successfully installed apache-airflow-3.3.0 ...
```

### 4.3 Installing with extras (providers)

"Extras" pull in the provider packages and their dependencies in one step. Specify them in brackets:

```bash
# Core + PostgreSQL + Google providers
pip install 'apache-airflow[postgres,google]==3.3.0' \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt"
```

Common extras include `postgres`, `mysql`, `celery`, `cncf.kubernetes`, `amazon`, `google`, `microsoft.azure`, `slack`, `http`, and `ssh`. You can also install any provider directly:

```bash
pip install apache-airflow-providers-amazon
```

> [!NOTE]
> Providers version independently from core. Installing a provider does not change your Airflow version, and you can upgrade a provider without upgrading Airflow.

### 4.4 First-run setup (manual)

If you are not using `standalone`, initialize the environment explicitly:

```bash
export AIRFLOW_HOME=~/airflow

# Create/upgrade the metadata database schema
airflow db migrate
```

```text
DB: sqlite:////home/you/airflow/airflow.db
Performing upgrade to the metadata database ...
Database migrating done!
```

Then start the services you need in separate terminals or as managed processes (see [§6, The Components You Run](#6-the-components-you-run)).

### 4.5 Verifying the install

```bash
airflow version
```

```text
3.3.0
```

```bash
airflow info
```

`airflow info` prints a full environment report — Airflow version, Python version, executor, configured paths, installed providers, and system details. Run it first whenever you file a bug report.

---

## 5. Core Concepts You Need Before Authoring

You can use most of Airflow knowing just a handful of terms. (Architecture-level internals are in the Admin Guide.)

| Term | Meaning |
|---|---|
| **Dag** | A workflow: a Python file defining tasks and their dependencies. (Airflow 3 spells it "Dag".) |
| **Task** | A single unit of work in a DAG — one node in the graph. |
| **Operator** | A template for a task (e.g. run a Bash command, call a Python function, run a SQL query). Operators come from providers or the standard set. |
| **Task instance** | A specific run of a task for a specific logical date/run. |
| **Dag run** | A specific execution of a whole DAG. |
| **Sensor** | A special task that waits for a condition (a file, a partition, a time) before succeeding. |
| **Hook** | A reusable interface to an external system (a database, an API), used inside operators. |
| **XCom** | "Cross-communication" — a small key/value channel for passing metadata between tasks. |
| **Connection** | Stored, named credentials/endpoints for an external system, referenced by `conn_id`. |
| **Variable** | A stored, named runtime value (often JSON), referenced from DAG code. |
| **Pool** | A concurrency limit shared across tasks, used to cap parallel access to a resource. |
| **Asset** | A logical data object; DAGs can be scheduled to run when assets are updated (data-aware scheduling). |
| **Bundle** | A versioned source of DAG files that the DAG processor reads (a "DAG bundle"). |

---

## 6. The Components You Run

A production Airflow is several long-running services. As a user you mostly interact with the CLI and UI, but it helps to know what each process does because you start them with the `airflow` CLI.

| Command | Process | Responsibility |
|---|---|---|
| `airflow scheduler` | Scheduler | Decides what runs and when; creates DAG runs and schedules task instances |
| `airflow api-server` | API server | Serves the REST API **and** the React web UI (FastAPI + uvicorn) |
| `airflow dag-processor` | DAG processor | Parses DAG files from bundles and writes serialized DAGs to the database |
| `airflow triggerer` | Triggerer | Runs asynchronous *deferrable* operators and sensors efficiently |
| `airflow standalone` | All of the above | One-process quick start for local use |

> [!NOTE]
> In Airflow 3 the old `webserver` command is gone — the UI is served by **`api-server`**. DAG parsing has also been split into its own **`dag-processor`** service. If you are migrating from Airflow 2, update your service definitions accordingly.

Minimal manual startup for development:

```bash
airflow scheduler &
airflow dag-processor &
airflow triggerer &
airflow api-server --port 8080
```

---

## 7. Authoring Your First DAG

DAGs are Python files placed in your **dags folder** (default `$AIRFLOW_HOME/dags`). Author them against the **Task SDK** (`airflow.sdk`), the stable public authoring interface.

### 7.1 A complete example

```python
from __future__ import annotations

import pendulum

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator


with DAG(
    dag_id="example_etl",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example", "etl"],
) as dag:

    extract = BashOperator(
        task_id="extract",
        bash_command="echo 'pulling data' && sleep 1",
    )

    @task
    def transform() -> dict:
        return {"rows": 42}

    @task
    def load(payload: dict) -> None:
        print(f"loaded {payload['rows']} rows")

    # Dependency wiring: extract -> transform -> load
    extract >> load(transform())
```

```text
# Save as $AIRFLOW_HOME/dags/example_etl.py
# Within ~30s the DAG processor parses it and it appears in the UI.
```

### 7.2 What the pieces mean

- **`dag_id`** — Unique name for the DAG. It is how you reference the DAG everywhere in the CLI and UI.
- **`schedule`** — When the DAG runs. Accepts cron strings (`"0 6 * * *"`), shorthand presets (`"@daily"`, `"@hourly"`), `timedelta` objects, or an **asset** expression for data-aware scheduling.
- **`start_date`** — The first logical date the scheduler considers.
- **`catchup`** — If `False` (recommended for most DAGs), Airflow does **not** run all missed intervals between `start_date` and now when you first enable the DAG.
- **`tags`** — Labels for filtering DAGs in the UI.
- **`>>`** — The dependency operator. `a >> b` means "run `a`, then `b`".
- **`@task`** — The TaskFlow decorator: turns a plain Python function into a task, and its return value becomes an XCom automatically passed to downstream tasks.

> [!TIP]
> Prefer importing from `airflow.sdk` (the Task SDK) rather than deep internal modules. The SDK is the supported, forward-compatible surface; internal imports can break between minor releases.

### 7.3 Validating a DAG before you ship it

```bash
# Parse the file and report import errors without scheduling anything
airflow dags list-import-errors
```

```bash
# Run the whole DAG once locally, in-process, for a date — great for development
airflow dags test example_etl 2026-01-01
```

```text
[2026-01-01T00:00:00+0000] INFO - Marking run <DagRun example_etl @ 2026-01-01> successful
```

`airflow dags test` executes the DAG end to end without needing a running scheduler — the single most useful authoring-loop command.

---

## 8. The `airflow` Command-Line Interface

The `airflow` CLI is your primary control surface. It is organized into **command groups** (each managing one kind of object) plus a handful of top-level **action commands** (mostly to start services). This section documents the full user-facing command tree of Airflow 3.3.0.

> [!NOTE]
> Every command accepts `-h/--help`. Many object commands accept `--output` (`table`, `json`, `yaml`, `plain`) so you can pipe results into scripts. Service commands accept `--pid`, `--daemon`, `--stdout`, `--stderr`, and `--log-file` for process management.

### 8.1 Top-level map

| Group / command | Purpose |
|---|---|
| `airflow dags` | Manage DAGs (list, trigger, pause, test, inspect) |
| `airflow tasks` | Manage and test individual tasks |
| `airflow backfill` | Create and manage backfills |
| `airflow partitions` | Manage DAG run partition metadata |
| `airflow assets` | Inspect and materialize assets |
| `airflow pools` | Manage concurrency pools |
| `airflow variables` | Manage variables |
| `airflow connections` | Manage connections |
| `airflow teams` | Manage teams |
| `airflow jobs` | Inspect/health-check running jobs |
| `airflow db` | Database operations (migrate, check, clean) |
| `airflow config` | View and lint configuration |
| `airflow providers` | Inspect installed providers and their extensions |
| `airflow state-store` | Manage task and asset state storage |
| `airflow db-manager` | Manage externally connected database managers |
| `airflow connections`, `airflow variables`, `airflow pools` | Object management (above) |
| `airflow scheduler` / `api-server` / `triggerer` / `dag-processor` | Start core services |
| `airflow kerberos` | Start a Kerberos ticket renewer |
| `airflow standalone` | Run an all-in-one copy of Airflow |
| `airflow rotate-fernet-key` | Re-encrypt secrets after a key rotation |
| `airflow version` | Print the Airflow version |
| `airflow info` | Print an environment report |
| `airflow plugins` | Dump information about loaded plugins |
| `airflow cheat-sheet` | Print a CLI cheat sheet |

### 8.2 `airflow dags` — manage DAGs

| Subcommand | What it does |
|---|---|
| `list` | List all DAGs known to the environment |
| `details` | Show details for a single DAG |
| `list-runs` | List the runs of a DAG (filter by state, date) |
| `list-jobs` | List jobs associated with DAG runs |
| `list-import-errors` | Show DAG files that failed to parse |
| `report` | Show a parse-time report of all DAG files |
| `state` | Show the state of a specific DAG run |
| `next-execution` | Show the next scheduled run time(s) |
| `pause` | Pause a DAG (stop scheduling new runs) |
| `unpause` | Resume a paused DAG |
| `trigger` | Manually start a new DAG run |
| `delete` | Delete a DAG and all its metadata |
| `show` | Render the DAG structure (e.g. to a graph) |
| `show-dependencies` | Render cross-DAG/asset dependencies |
| `test` | Execute a DAG run in-process for a date (no scheduler needed) |
| `reserialize` | Re-serialize all DAGs into the database |

Common examples:

```bash
# List DAGs as JSON
airflow dags list --output json

# Trigger a run now, passing a JSON configuration to the run
airflow dags trigger example_etl --conf '{"region": "us-east-1"}'

# Pause / resume
airflow dags pause example_etl
airflow dags unpause example_etl

# Inspect recent runs that failed
airflow dags list-runs --dag-id example_etl --state failed

# When was this DAG going to run next?
airflow dags next-execution example_etl
```

```text
# airflow dags trigger output
Created <DagRun example_etl @ 2026-06-04T12:00:00+00:00: manual__2026-06-04T12:00:00+00:00, state:queued>
```

> [!TIP]
> `airflow dags trigger --conf '{...}'` passes a run-time configuration dictionary that your tasks read via the runtime context (`{{ dag_run.conf }}` in templates). This is the standard way to parameterize manual runs.

### 8.3 `airflow tasks` — manage and test tasks

| Subcommand | What it does |
|---|---|
| `list` | List the tasks in a DAG |
| `state` | Show the state of a single task instance |
| `failed-deps` | Explain why a task instance is not runnable (which dependencies fail) |
| `render` | Render a task's templated fields for a date (see the actual command that would run) |
| `test` | Run a single task instance in-process for a date, without scheduling |
| `clear` | Clear task instances so they will be re-run |
| `states-for-dag-run` | Show the state of every task in a given DAG run |

```bash
# List tasks in a DAG
airflow tasks list example_etl

# Dry-run a single task for a date (prints logs to your terminal)
airflow tasks test example_etl transform 2026-01-01

# Why is this task stuck?
airflow tasks failed-deps example_etl load 2026-01-01

# See the exact templated command/SQL that would execute
airflow tasks render example_etl extract 2026-01-01
```

> [!IMPORTANT]
> `airflow tasks test` runs the task **immediately and in-process**, ignoring dependencies, pools, and the scheduler. It is a development tool — it does not record a normal task instance state the way a scheduled run does.

### 8.4 `airflow backfill` — run historical intervals

Backfills run a DAG across a **range of past dates**. In Airflow 3, backfill is a first-class object you create and manage rather than a flag on `dags`.

```bash
airflow backfill create \
  --dag-id example_etl \
  --from-date 2026-01-01 \
  --to-date 2026-01-31
```

> [!NOTE]
> Use backfills deliberately. They can create many DAG runs at once; combine with **pools** (§8.6) to cap how much they hit downstream systems.

### 8.5 `airflow connections` — manage external credentials

Connections store the endpoints and credentials your operators use, keyed by `conn_id`.

| Subcommand | What it does |
|---|---|
| `list` | List stored connections |
| `get` | Show one connection |
| `add` | Create a connection |
| `delete` | Remove a connection |
| `export` | Export connections to a file (JSON/YAML/env) |
| `import` | Import connections from a file |
| `test` | Test that a connection actually works |
| `create-default-connections` | Seed the default example connections |

```bash
# Add a Postgres connection via a connection URI
airflow connections add my_pg \
  --conn-uri 'postgresql://user:[email removed]:5432/analytics'

# List and test
airflow connections list --output table
airflow connections test my_pg

# Back up all connections to a file
airflow connections export connections.json
```

```text
# airflow connections test my_pg
Connection success!
```

> [!CAUTION]
> Connection secrets are encrypted at rest using the Fernet key. Treat exported files (which may contain plaintext credentials) as sensitive and delete them after use. See the Admin Guide's security chapter for secrets backends (Vault, AWS Secrets Manager, etc.).

### 8.6 `airflow pools` — limit concurrency

Pools cap how many task instances can run against a shared resource at once.

```bash
# Create a pool of 5 slots for a rate-limited API
airflow pools set api_pool 5 "Limit concurrent calls to the vendor API"

# List and inspect
airflow pools list --output table
airflow pools get api_pool

# Import/export pools as JSON
airflow pools export pools.json
airflow pools import pools.json
```

In a task, assign it to the pool with `pool="api_pool"`.

### 8.7 `airflow variables` — runtime values

```bash
# Set a JSON variable
airflow variables set feature_flags '{"new_pipeline": true}'

# Get it back (optionally a nested key)
airflow variables get feature_flags

# Bulk import/export
airflow variables export variables.json
airflow variables import variables.json
```

In a DAG, read it with `Variable.get("feature_flags", deserialize_json=True)` from the Task SDK.

> [!TIP]
> Reading a Variable at the top level of a DAG file runs on **every parse** and hammers the database. Read Variables inside tasks (or use Jinja templating `{{ var.json.feature_flags }}`) instead of at module scope.

### 8.8 `airflow assets` — data-aware scheduling

| Subcommand | What it does |
|---|---|
| `list` | List assets known to the environment |
| `details` | Show details for one asset |
| `materialize` | Trigger the DAG that produces an asset |

Assets let a DAG be scheduled to run when another DAG updates a dataset, rather than on a clock. Schedule a consumer DAG with `schedule=[my_asset]`.

### 8.9 `airflow config` — view configuration

```bash
# Print the full effective configuration
airflow config list

# Read a single value
airflow config get-value core executor

# Lint config for deprecated/removed options
airflow config lint
```

```text
# airflow config get-value core executor
LocalExecutor
```

> [!NOTE]
> By default the API server does **not** expose configuration in the UI (`expose_config = False`) for security. The CLI always reads the local config. The full option reference is in the Admin Guide.

### 8.10 `airflow providers` — inspect installed providers

The `providers` group introspects what each installed provider contributes. Useful subcommands:

| Subcommand | Shows |
|---|---|
| `list` | All installed providers and versions |
| `get` | Details for one provider |
| `hooks` | All hooks registered by providers |
| `links` | Extra operator links |
| `widgets` | Connection-form widgets |
| `behaviours` | Connection-type behaviours |
| `logging` | Logging handlers contributed |
| `secrets` | Secrets backends contributed |
| `executors` | Executors contributed |
| `notifications` | Notification classes |
| `configs` | Provider-contributed config |
| `auth-managers` | Available auth managers |

```bash
airflow providers list --output table
airflow providers hooks
```

### 8.11 `airflow db` — database operations

Mostly an operator concern, but two are commonly run by users on dev environments:

```bash
# Create/upgrade the schema (run after install and after upgrades)
airflow db migrate

# Verify connectivity and schema health
airflow db check

# Open an interactive DB shell
airflow db shell

# Purge old metadata (use with care — see Admin Guide)
airflow db clean --clean-before-timestamp 2026-01-01
```

> [!WARNING]
> `airflow db clean`, `db reset`, and `db downgrade` are destructive. `db reset` drops all metadata. Read the Admin Guide before running any of them against a shared environment.

### 8.12 Other top-level commands

```bash
airflow version          # Print version (3.3.0)
airflow info             # Full environment report (attach to bug reports)
airflow cheat-sheet      # Print a quick command cheat sheet
airflow plugins          # List loaded plugins
airflow rotate-fernet-key  # Re-encrypt secrets after rotating the Fernet key
```

---

## 9. The Web UI

The web interface is served by `airflow api-server` (default <http://localhost:8080>). It is the fastest way to observe and intervene in running pipelines.

| View | What it gives you |
|---|---|
| **Dags** | The list of all DAGs, with toggle to pause/unpause, run counts, and quick actions |
| **Grid** | A time-by-task grid of a DAG's runs — the workhorse for spotting failures across runs |
| **Graph** | The DAG's task dependency graph for a single run, color-coded by state |
| **Code** | The source of a DAG (read-only) |
| **Home / Dashboard** | Summary statistics and health of the environment |
| **Assets** | Assets and their producer/consumer dependencies |
| **Backfill** | Create and watch backfills over a date range |

Typical UI workflow when a task fails:

1. Open **Grid**, find the red (failed) task instance.
2. Click it and open **Logs** to read the traceback.
3. Fix the cause (data, connection, code).
4. **Clear** the task instance (or the downstream subtree) to re-run it — equivalent to `airflow tasks clear`.

> [!TIP]
> Clearing a task in the Grid view re-runs it and, by default, its downstream tasks. This is the most common recovery action and mirrors the `airflow tasks clear` CLI command.

---

## 10. Using Providers

Providers are how Airflow talks to the outside world. Each provider ships operators, hooks, sensors, and sometimes transfers, secrets backends, or executors. Airflow 3.3.0 has **87 providers** spanning cloud platforms, databases, messaging, and more.

Pattern for using a provider operator:

```python
from airflow.sdk import DAG
from airflow.providers.amazon.aws.operators.s3 import S3CreateBucketOperator

with DAG(dag_id="s3_example", schedule=None, catchup=False) as dag:
    create_bucket = S3CreateBucketOperator(
        task_id="create_bucket",
        bucket_name="my-airflow-bucket",
        aws_conn_id="aws_default",   # references a stored Connection
    )
```

The `*_conn_id` argument points at a Connection you created with `airflow connections add`. The operator's hook reads that connection to authenticate.

> [!NOTE]
> Discover what an installed provider offers with `airflow providers hooks`, `airflow providers list`, and the provider's own documentation. Operators are imported from `airflow.providers.<provider>....`.

A small sample of widely used providers:

| Provider | Extra | Representative use |
|---|---|---|
| `amazon` | `[amazon]` | S3, Athena, ECS, EMR, Lambda, Redshift |
| `google` | `[google]` | GCS, BigQuery, Dataflow, Cloud Composer |
| `cncf.kubernetes` | `[cncf.kubernetes]` | `KubernetesPodOperator`, Kubernetes executor |
| `postgres` / `mysql` | `[postgres]` / `[mysql]` | SQL execution, transfers |
| `http` | `[http]` | REST calls and sensors |
| `slack` | `[slack]` | Notifications |
| `ssh` | `[ssh]` | Remote command execution |
| `standard` | (bundled) | `BashOperator`, `PythonOperator`, core sensors |

---

## 11. Driving a Remote Airflow with `airflowctl`

`airflowctl` (the `apache-airflow-ctl` package) is a **standalone** command-line client that operates a remote Airflow instance over its REST API. Unlike the `airflow` CLI — which must run on a host with the metadata database — `airflowctl` only needs network access to an API server and an auth token.

```bash
pip install apache-airflow-ctl

# Point it at your environment and authenticate
export AIRFLOW_CLI_ENVIRONMENT=prod
airflowctl auth login --api-url https://airflow.example.com --token "$AIRFLOW_CLI_TOKEN"

# Now drive the remote instance
airflowctl dags list
airflowctl dags trigger example_etl
```

| Environment variable | Purpose |
|---|---|
| `AIRFLOW_CLI_TOKEN` | API auth token |
| `AIRFLOW_CLI_ENVIRONMENT` | Named environment to target |
| `AIRFLOW_CLI_API_RETRIES` | Retry attempts on API failure (default `3`) |
| `AIRFLOW_CLI_API_RETRY_WAIT_MIN` | Min backoff seconds (default `1`) |
| `AIRFLOW_CLI_API_RETRY_WAIT_MAX` | Max backoff seconds (default `10`) |
| `AIRFLOW_CLI_DEBUG_MODE` | Enable verbose debug output |

> [!TIP]
> Use `airflowctl` from CI pipelines and operator laptops to manage managed/remote Airflow deployments where you have no shell access to the scheduler host.

---

## 12. Common Recipes

### 12.1 Re-run a failed task and everything after it

```bash
airflow tasks clear example_etl \
  --task-regex 'transform' \
  --downstream \
  --start-date 2026-06-01 --end-date 2026-06-04 --yes
```

### 12.2 Trigger a parameterized one-off run

```bash
airflow dags trigger example_etl --conf '{"full_refresh": true}'
```

### 12.3 Promote connections/variables between environments

```bash
# On source
airflow connections export conns.json
airflow variables export vars.json

# On target
airflow connections import conns.json
airflow variables import vars.json
```

### 12.4 Back up before an upgrade

```bash
airflow db export-archived --output-path ./archive   # archive old rows first if desired
airflow connections export conns.json
airflow variables export vars.json
airflow pools export pools.json
```

### 12.5 Throttle a busy DAG

Assign its expensive tasks to a small pool, then cap the DAG itself with `max_active_runs` and `max_active_tasks` in the DAG definition.

---

## 13. Troubleshooting

Format: the exact symptom, the cause, the fix, and how to prevent recurrence. These are grounded in real, community-reported issues and Airflow's own error surface.

### Error: DAG does not appear in the UI

**Cause:** The DAG processor hasn't parsed it, or the file has an import error.
**Solution:**
```bash
airflow dags list-import-errors
airflow dags reserialize
```
If the file is valid, confirm it is inside the configured **dags folder** and that the `dag-processor` service is running.
**Prevention:** Run `airflow dags test <dag_id> <date>` before deploying; keep DAG files importable (no heavy top-level work).

### Error: `airflow command error: the following arguments are required: ...`

**Cause:** A required positional/option argument was omitted.
**Solution:** Run the command with `-h` to see required args, e.g. `airflow dags trigger -h`.
**Prevention:** Use `--output json` and scripted wrappers for repeated commands.

### Error: Task stuck in `queued` or `scheduled` and never runs

**Cause:** No worker capacity, an exhausted pool, or a stopped scheduler.
**Solution:**
```bash
airflow jobs check --job-type SchedulerJob   # is a scheduler alive?
airflow tasks failed-deps <dag_id> <task_id> <date>
airflow pools list                            # are slots available?
```
**Prevention:** Monitor scheduler health (`scheduler.enable_health_check`, see Admin Guide) and size pools to real capacity.

### Error: DAG code syntax highlighting is wrong with f-strings (UI display)

**Cause:** Known UI rendering quirk in the **Code** view for some f-string patterns (community issue #67986). It is a *display* artifact only — execution is unaffected.
**Solution:** None required; the code runs correctly. Verify with `airflow tasks render`.
**Prevention:** N/A — cosmetic.

### Error: `KubernetesPodOperator` fails behind a corporate proxy

**Cause:** A `no_proxy` regression in older Kubernetes client versions used by the `cncf.kubernetes` provider (community issue #67845).
**Solution:** Upgrade the `cncf.kubernetes` provider to a build that supports Kubernetes client `>=36.0.1`.
**Prevention:** Keep providers current; pin provider versions via constraints.

### Error: Deep links redirect to the homepage after login (SimpleAuthManager)

**Cause:** The `next` parameter was ignored during auto-login on first visit (community issue #67476).
**Solution:** Upgrade to a release where the fix is applied; re-navigate to the deep link after authenticating.
**Prevention:** Keep core current.

### Error: Duplicate DAG runs under multiple schedulers (HA)

**Cause:** A race condition for certain partitioned/asset timetables when running multiple schedulers in high-availability (community issue #68045).
**Solution:** If you observe duplicates, reduce to a single scheduler for the affected timetable or apply the latest patch release.
**Prevention:** Track the fix in your upgrade cadence; test HA timetables before relying on them.

### Error: `sqlite` / "database is locked" or poor concurrency

**Cause:** Running SQLite (the default) with concurrent tasks.
**Solution:** Switch `database.sql_alchemy_conn` to PostgreSQL or MySQL (Admin Guide).
**Prevention:** Never use SQLite beyond local experimentation.

### Error: `pip install apache-airflow` produces a broken environment

**Cause:** Installed without constraints, so dependencies resolved to an incompatible set.
**Solution:** Reinstall with the matching `--constraint` URL for your Airflow and Python version.
**Prevention:** Always pass `--constraint` (see §4.2).

---

## 14. Frequently Asked Questions

**Q: Which Python versions are supported?**
Python 3.10 through 3.14 for Airflow 3.3.0.

**Q: Can I run Airflow on Windows?**
Not natively. Use WSL2 or a Linux container. Only Linux is a supported production environment.

**Q: What's the difference between `airflow standalone` and running the services separately?**
`standalone` runs the scheduler, API server, DAG processor, and triggerer together with SQLite and a generated admin user — for local use only. Production runs each service separately against PostgreSQL/MySQL.

**Q: Where did `airflow webserver` go?**
In Airflow 3 the UI and REST API are served by `airflow api-server`. There is no `webserver` command.

**Q: Where did `airflow users` and `airflow roles` go?**
User/role management is no longer in core. It is provided by the auth manager you configure. The default `SimpleAuthManager` uses a simple username/password scheme; the FAB provider supplies full RBAC. See the Admin Guide.

**Q: How do I pass data between tasks?**
Small metadata via **XCom** (TaskFlow return values do this automatically). Large data should be written to external storage and referenced, not pushed through XCom.

**Q: How do I parameterize a manual run?**
`airflow dags trigger <dag_id> --conf '{"key": "value"}'`, then read `{{ dag_run.conf }}` (or `context["dag_run"].conf`) in your tasks.

**Q: What is `catchup` and should I enable it?**
`catchup=True` makes Airflow run every missed interval back to `start_date` when a DAG is first enabled. Most users set `catchup=False` and use explicit **backfills** when they need historical runs.

**Q: How do I install operators for AWS / Google / Kubernetes?**
Install the relevant provider, e.g. `pip install apache-airflow-providers-amazon`, or use an extra like `apache-airflow[amazon]`.

**Q: What is a "bundle"?**
A DAG bundle is a versioned source of DAG files that the DAG processor reads. It lets Airflow track which version of your DAG code produced a given run.

**Q: How do I test a single task quickly?**
`airflow tasks test <dag_id> <task_id> <date>` runs it in-process and prints logs, without touching the scheduler.

**Q: How do I find out what's installed and where?**
`airflow info` prints the full environment report (version, Python, executor, paths, providers).

**Q: Why shouldn't I read a Variable at the top of my DAG file?**
The DAG file is parsed frequently; top-level reads hit the database on every parse. Read Variables inside tasks or via Jinja templating.

**Q: How do I drive a remote/managed Airflow without shell access?**
Use `airflowctl` (the `apache-airflow-ctl` package) against the REST API with a token.

---

## 15. Quick Reference

### Service commands

| Command | Starts |
|---|---|
| `airflow scheduler` | Scheduler |
| `airflow api-server` | REST API + web UI |
| `airflow dag-processor` | DAG parser |
| `airflow triggerer` | Deferrable task runner |
| `airflow standalone` | Everything (local only) |

### Most-used object commands

| Command | Action |
|---|---|
| `airflow dags list` | List DAGs |
| `airflow dags trigger <id> [--conf JSON]` | Start a run |
| `airflow dags test <id> <date>` | Run a DAG locally |
| `airflow dags pause/unpause <id>` | Toggle scheduling |
| `airflow tasks test <id> <task> <date>` | Run one task locally |
| `airflow tasks clear <id> ...` | Re-run task instances |
| `airflow connections add/list/test` | Manage connections |
| `airflow variables set/get` | Manage variables |
| `airflow pools set <name> <slots>` | Manage concurrency |
| `airflow db migrate` | Apply schema migrations |
| `airflow config get-value <sec> <key>` | Read one setting |

### Key environment variables

| Variable | Default | Meaning |
|---|---|---|
| `AIRFLOW_HOME` | `~/airflow` | Root for dags, logs, config, and (default) the SQLite db |
| `AIRFLOW_CONFIG` | `$AIRFLOW_HOME/airflow.cfg` | Path to the config file |
| `AIRFLOW__<SECTION>__<KEY>` | — | Override any config option via environment (e.g. `AIRFLOW__CORE__EXECUTOR`) |
| `AIRFLOW_CLI_TOKEN` | — | Token for `airflowctl` |

> [!TIP]
> Any `airflow.cfg` setting can be overridden with an environment variable named `AIRFLOW__SECTION__KEY` (double underscores, uppercased). This is the most common way to configure Airflow in containers. The full option list is in the Admin Guide.

---

_Generated June 2026 from apache/airflow at 0dd0189. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._