# Apache Airflow Administrator & Architecture Guide This guide is the operational and architectural reference for Apache Airflow **3.3.0**. It is written for the people who deploy, configure, secure, scale, and maintain Airflow — platform engineers, SREs, data-platform architects, and developers extending the system. Where the User Guide covers *using* Airflow, this guide covers *running* it: how the components fit together, every configuration surface that matters, deployment topologies, executor selection, the metadata database, security, observability, and the internals you need to operate (or replicate) the platform with confidence. > [!NOTE] > Airflow is a large, multi-product monorepo. This guide centers on the **flagship deployable** — the `apache-airflow` core platform — and treats the Task SDK, `airflowctl`, the Helm chart, the 87 provider packages, and the experimental Go/Java edge SDKs as the extensions they are. Each gets focused coverage; the core gets depth. --- ## 1. Executive Summary Apache Airflow is a Python-based workflow orchestration platform. Workflows ("DAGs") are authored as code; a set of long-running services parse them, schedule task instances, execute those tasks via a pluggable **executor**, and record all state in a relational metadata database. A FastAPI-based API server exposes both a REST API and a React 19 web UI. **What you are operating:** - A **scheduler** (the brain — creates DAG runs and schedules tasks). - An **API server** (FastAPI + uvicorn — serves the REST API and UI). - A **DAG processor** (parses DAG files from bundles into the database). - A **triggerer** (runs asynchronous deferrable tasks efficiently). - An **executor** (where tasks actually run — Local, Celery, Kubernetes, Edge, or cloud-native). - A **metadata database** (PostgreSQL or MySQL in production; SQLite for dev only). - Optional **workers** (for Celery) and **edge workers** (for the Edge executor). **Key 3.x architectural facts an operator must internalize:** | Fact | Implication | |---|---| | `webserver` → `api-server` | The UI is served by `airflow api-server`; update service definitions and health checks | | DAG parsing is its own service | You must run `airflow dag-processor`; it is not folded into the scheduler | | Task execution is decoupled via an **Execution API** | Tasks talk to Airflow through a versioned API, enabling the Task SDK and remote/edge workers | | Auth is pluggable (**auth managers**) | Default is `SimpleAuthManager`; full RBAC comes from the FAB provider | | Executors live in providers | Core ships only `LocalExecutor`; Celery/Kubernetes/Edge/cloud executors are provider packages | | Secrets are encrypted with a **Fernet key** | Losing the key means losing all stored connection/variable secrets | **Maturity & support:** Airflow 3 is the current major line (3.x first released Apr 2025, in active maintenance). Airflow 2.x is in limited maintenance with EOL Apr 22, 2026. Run the latest 3.x patch. --- ## 2. Architecture ### 2.1 Component topology ``` ┌──────────────────────────────────────────────┐ │ Metadata Database │ │ (PostgreSQL / MySQL; SQLite = dev) │ │ DAG runs · task instances · connections · │ │ variables · pools · serialized DAGs · XComs │ └───────▲───────────▲───────────▲───────────▲────┘ │ │ │ │ ┌─────────────────────────┘ │ │ └────────────────────┐ │ │ │ │ ┌──────┴───────┐ ┌───────┴──────┐ ┌──┴────────────┐ ┌───────┴────────┐ │ Scheduler │ │ DAG │ │ API Server │ │ Triggerer │ │ - create │ │ Processor │ │ (FastAPI + │ │ - async │ │ DAG runs │ │ - parse DAG │ │ uvicorn) │ │ deferrable │ │ - schedule │ │ bundles │ │ - REST API │ │ operators │ │ task inst │ │ - serialize │ │ - React UI │ │ │ │ - heartbeat │ │ to DB │ │ - Execution │ └────────────────┘ └──────┬───────┘ └──────────────┘ │ API │ │ hand off workloads └───────▲───────┘ ▼ │ Execution API (versioned) ┌──────────────────────────────────────────────┐ │ │ Executor │ ┌────────┴─────────────────────────┐ │ Local (in-proc) │ Celery │ Kubernetes │ Edge │ │ Task runtime (Task SDK) │ │ │ AWS ECS/Batch/Lambda │◄────┤ workers / pods / edge nodes run │ └──────────────────────────────────────────────┘ │ task code, call back via API │ └───────────────────────────────────┘ ▲ ▲ │ reads DAG files from │ ┌──────┴───────────────┐ ┌───────┴────────┐ │ DAG Bundles │ │ Providers │ │ (versioned sources │ │ (87 packages: │ │ of dags folder) │ │ operators, │ └──────────────────────┘ │ hooks, etc.) │ └────────────────┘ ``` ### 2.2 Data and control flow 1. The **DAG processor** reads DAG files from configured **bundles**, parses them, and writes **serialized DAGs** to the metadata DB. (Parsing is isolated from scheduling so a bad DAG can't take down the scheduler.) 2. The **scheduler** reads serialized DAGs, creates **DAG runs** when schedules fire (or assets update), and creates **task instances**, scheduling those whose dependencies are met. It uses **row-level locking** to allow multiple schedulers (HA). 3. The scheduler hands runnable workloads to the **executor**, which places task execution onto workers/pods/processes. 4. Task code runs against the **Task SDK** and communicates with Airflow only through the versioned **Execution API** exposed by the API server — never touching the database directly. This decoupling is what enables remote and edge workers and forward-compatible authoring. 5. **Deferrable** operators yield control to the **triggerer**, which polls external conditions asynchronously and resumes the task when ready — freeing worker slots while waiting. 6. The **API server** serves the REST API and React UI for human and programmatic interaction. ### 2.3 Why the split matters operationally - **Scheduler HA:** Multiple schedulers can run concurrently (`scheduler.use_row_level_locking = True`). Watch for timetable-specific race conditions (e.g. partitioned/asset timetables, community issue #68045) and patch promptly. - **DAG processor isolation:** Parse errors and slow imports affect only parsing throughput, not scheduling decisions already in the DB. - **Execution API boundary:** Because tasks never hit the DB directly, you can run workers outside your trusted network (edge), and the Task SDK stays stable across core upgrades. --- ## 3. Component Inventory | Component | Start command | Default port | Scales by | Notes | |---|---|---|---|---| | Scheduler | `airflow scheduler` | health: `8974` | Run N schedulers (HA) | Heartbeat-driven; row-level locking | | API server | `airflow api-server` | `8080` | Run N + load balancer | FastAPI/uvicorn; serves UI + REST + Execution API | | DAG processor | `airflow dag-processor` | health: internal | `parsing_processes` | Reads bundles; serializes DAGs | | Triggerer | `airflow triggerer` | log: `8794` | `--capacity`, run N | Async deferrable execution | | Worker (Celery) | provider-supplied | log: `8793` | Add workers | Only with CeleryExecutor | | Edge worker | `go-sdk` / provider | — | Add edge nodes | With EdgeExecutor (experimental) | | Kerberos renewer | `airflow kerberos` | — | One per host | Optional, for Kerberized environments | --- ## 4. System Requirements | Component | Supported | Production guidance | |---|---|---| | Python | 3.10–3.14 | Match the constraint file to your Python minor | | OS | POSIX (Linux/macOS) | Production = Linux only; Debian Bookworm is the reference image base | | Metadata DB | PostgreSQL 14–18, MySQL 8.0/8.4/Innovation | **PostgreSQL strongly recommended**; never SQLite in prod; MariaDB unsupported | | Kubernetes | 1.30–1.35 | For the Helm chart / KubernetesExecutor | | Platform | AMD64 / ARM64 | Both first-class | > [!CAUTION] > SQLite + LocalExecutor are single-writer and cannot support concurrent task execution safely. They exist for `standalone` and local development only. --- ## 5. Installation & Deployment ### 5.1 Installation methods | Method | Best for | Notes | |---|---|---| | **PyPI + constraints** | VMs, bare metal, custom images | `pip install 'apache-airflow==3.3.0' --constraint ` (always use constraints) | | **Official Docker image** | Containers, docker-compose, custom builds | `apache/airflow:3.3.0`; Debian Bookworm base | | **Helm chart** | Kubernetes | The production-grade path on K8s; versions independently of core | > [!IMPORTANT] > The dual library/application nature of Airflow makes unconstrained installs unreliable. Always pin with the constraint file for your exact Airflow **and** Python version: `https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt`. ### 5.2 Reference single-node (non-Kubernetes) deployment 1. Provision PostgreSQL; create a database and user. 2. Install Airflow + needed providers into a virtualenv with constraints. 3. Set configuration (env vars or `airflow.cfg`): ```bash export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN='postgresql+psycopg2://airflow:***@db:5432/airflow' export AIRFLOW__CORE__EXECUTOR='LocalExecutor' # or CeleryExecutor for multi-node export AIRFLOW__CORE__FERNET_KEY='' export AIRFLOW__API__SECRET_KEY='' ``` 4. Initialize the schema: `airflow db migrate`. 5. Run each service under a process supervisor (systemd, supervisord): - `airflow scheduler` - `airflow dag-processor` - `airflow triggerer` - `airflow api-server` 6. Put a TLS-terminating reverse proxy in front of the API server. > [!TIP] > Generate a Fernet key once and store it in your secret manager: > ```python > from cryptography.fernet import Fernet; print(Fernet.generate_key().decode()) > ``` > Reuse the **same** key across all services. Rotating it later requires `airflow rotate-fernet-key`. ### 5.3 Kubernetes (Helm chart) The official `apache-airflow` Helm chart deploys scheduler, API server, DAG processor, triggerer, optional Celery or Kubernetes executors, PgBouncer (connection pooling), and supporting resources. It versions independently of core; pin both. Detailed values are in the chart's own documentation — operationally, the chart is the supported way to run Airflow on Kubernetes. > [!NOTE] > Community issue #67951 tracks adding template (`tpl`) support to `.Values.volumes`, `.Values.volumeMounts`, and `pgbouncer.configSecretName` — relevant if you template chart values dynamically. ### 5.4 Deployment checklist - [ ] PostgreSQL/MySQL provisioned and reachable from all services - [ ] `sql_alchemy_conn` set via env/secret (not committed to `airflow.cfg`) - [ ] Fernet key generated and shared across all components - [ ] `api.secret_key` set and consistent across API server replicas - [ ] Executor chosen and its provider installed - [ ] DAG bundles configured (`dag_processor.dag_bundle_config_list`) - [ ] `airflow db migrate` run before first start and after every upgrade - [ ] Auth manager selected and admin access provisioned - [ ] Remote logging configured (so logs survive worker churn) - [ ] Reverse proxy / TLS in front of the API server - [ ] Health checks wired to scheduler (`8974`) and API server (`8080`) - [ ] Backups of the metadata DB scheduled --- ## 6. Configuration Mastery Airflow's configuration lives in `airflow.cfg` (under `$AIRFLOW_HOME`), but **every option can be set via an environment variable** named `AIRFLOW__
__` (uppercase, double underscores). Environment variables win over the file, which is why containerized deployments configure Airflow almost entirely through env. There is also a `_CMD` / `_SECRET` suffix convention to source a value from a shell command or a secrets backend instead of inlining it. The authoritative option set is defined in `airflow-core/src/airflow/config_templates/config.yml`, which contains **28 sections** and roughly 600 options, each carrying a default and a `version_added`. The sections: `core`, `database`, `logging`, `metrics`, `traces`, `secrets`, `api`, `workers`, `sdk`, `api_auth`, `execution_api`, `lineage`, `operators`, `email`, `smtp`, `sentry`, `scheduler`, `connection_test`, `triggerer`, `kerberos`, `sensors`, `dag_processor`, `state_store`, `profiling`, `callbacks`. The following subsections document the options operators touch most. Defaults are taken directly from `config.yml`. ### 6.1 `[core]` — fundamentals | Key | Default | Purpose | |---|---|---| | `dags_folder` | `{AIRFLOW_HOME}/dags` | Where DAG files live | | `executor` | `LocalExecutor` | The executor class (or provider path) | | `auth_manager` | `...simple.SimpleAuthManager` | Pluggable authentication/authorization | | `parallelism` | `32` | Max task instances running across the whole environment | | `max_active_tasks_per_dag` | `16` | Per-DAG concurrent task cap | | `max_active_runs_per_dag` | `16` | Per-DAG concurrent run cap | | `max_consecutive_failed_dag_runs_per_dag` | `0` | Auto-pause a DAG after N consecutive failures (0 = off) | | `dags_are_paused_at_creation` | `True` | New DAGs start paused | | `load_examples` | `True` | Load bundled example DAGs (disable in prod) | | `fernet_key` | `{FERNET_KEY}` | Symmetric key encrypting secrets at rest | | `default_timezone` | `utc` | Timezone for naive datetimes | | `plugins_folder` | `{AIRFLOW_HOME}/plugins` | Where plugins are loaded from | | `dagbag_import_timeout` | `30.0` | Seconds before a DAG file import is killed | | `hostname_callable` | `airflow.utils.net.getfqdn` | How hostnames are resolved | | `simple_auth_manager_users` | `admin:admin` | Default users for SimpleAuthManager | > [!WARNING] > Set `load_examples = False` and change the `simple_auth_manager_users` default away from `admin:admin` before any non-local deployment. ### 6.2 `[database]` — metadata store | Key | Default | Purpose | |---|---|---| | `sql_alchemy_conn` | `sqlite:///{AIRFLOW_HOME}/airflow.db` | DB connection string — **change for production** | | `sql_alchemy_pool_enabled` | `True` | Use SQLAlchemy connection pooling | | `sql_alchemy_pool_size` | `5` | Pool size per process | | `sql_alchemy_max_overflow` | `10` | Extra connections beyond pool size | | `sql_alchemy_pool_recycle` | `1800` | Recycle connections after N seconds | | `sql_alchemy_pool_pre_ping` | `True` | Test connections before use | | `max_db_retries` | `3` | Retries on transient DB errors | | `check_migrations` | `True` | Verify schema is migrated on startup | | `migration_batch_size` | `10000` | Rows per batch in migrations | | `external_db_managers` | `~` | Externally managed DB managers | > [!TIP] > With many scheduler/worker processes, the sum of `pool_size + max_overflow` across all processes can exhaust DB connections. On Kubernetes, front PostgreSQL with **PgBouncer** (the Helm chart includes it). ### 6.3 `[api]` — API server & UI | Key | Default | Purpose | |---|---|---| | `host` | `0.0.0.0` | Bind address | | `port` | `8080` | Bind port | | `workers` | `1` | uvicorn worker processes | | `worker_timeout` | `120` | Worker timeout (s) | | `server_type` | `uvicorn` | ASGI server | | `secret_key` | `{SECRET_KEY}` | Session signing key — must be identical across replicas | | `base_url` | `~` | Public URL of the UI | | `expose_config` | `False` | Whether the UI exposes config (keep False) | | `expose_stacktrace` | `False` | Whether errors expose stack traces | | `enable_swagger_ui` | `True` | Serve interactive API docs | | `access_control_allow_origins` | `""` | CORS origins | | `ssl_cert` / `ssl_key` | `""` | Direct TLS (prefer a proxy) | | `maximum_page_limit` | `100` | Max page size for list endpoints | | `require_confirmation_dag_change` | `False` | Require confirm before DAG state changes in UI | > [!IMPORTANT] > When running multiple API server replicas, set a fixed `api.secret_key` (and `core.fernet_key`) so sessions and encrypted values are consistent across all of them. ### 6.4 `[scheduler]` — scheduling behavior | Key | Default | Purpose | |---|---|---| | `scheduler_heartbeat_sec` | `5` | Scheduler loop heartbeat | | `use_row_level_locking` | `True` | Enables multi-scheduler HA | | `max_tis_per_query` | `16` | Task instances examined per query | | `max_dagruns_to_create_per_loop` | `10` | New DAG runs created per loop | | `max_dagruns_per_loop_to_schedule` | `20` | DAG runs scheduled per loop | | `catchup_by_default` | `False` | Default `catchup` for DAGs | | `enable_health_check` | `False` | Expose a scheduler health endpoint | | `scheduler_health_check_server_port` | `8974` | Health endpoint port | | `scheduler_health_check_threshold` | `30` | Seconds before a scheduler is "unhealthy" | | `task_instance_heartbeat_timeout` | `300` | Mark a task as failed if no heartbeat | | `task_queued_timeout` | `600.0` | Fail tasks stuck queued this long | | `parsing_cleanup_interval` | `60` | Cleanup of stale parse data | > [!TIP] > Enable `scheduler.enable_health_check` and point your orchestrator's liveness probe at port `8974`. Tune `max_tis_per_query` and `max_dagruns_*` together when scaling to thousands of tasks. ### 6.5 `[dag_processor]` — DAG parsing | Key | Default | Purpose | |---|---|---| | `dag_bundle_config_list` | (built-in) | The list of DAG bundles to read | | `dag_bundle_storage_path` | `~` | Where bundle sources are materialized | | `refresh_interval` | `300` | Seconds between bundle refreshes | | `parsing_processes` | `2` | Parallel DAG-parsing processes | | `min_file_process_interval` | `30` | Min seconds between re-parsing a file | | `dag_file_processor_timeout` | `50` | Kill a file parse after N seconds | | `stale_dag_threshold` | `50` | Mark DAGs stale after N seconds absent | | `disable_bundle_versioning` | `False` | Turn off versioned bundles | | `parsing_pre_import_modules` | `True` | Pre-import modules to speed parsing | ### 6.6 `[logging]` — task and system logs | Key | Default | Purpose | |---|---|---| | `base_log_folder` | `{AIRFLOW_HOME}/logs` | Local log root | | `remote_logging` | `False` | Ship logs to remote storage | | `remote_base_log_folder` | `""` | Remote URI (e.g. `s3://...`, `gs://...`) | | `remote_log_conn_id` | `""` | Connection used for remote logging | | `delete_local_logs` | `False` | Delete local logs after upload | | `logging_level` | `INFO` | Root log level | | `worker_log_server_port` | `8793` | Port serving task logs to the UI | | `trigger_log_server_port` | `8794` | Port serving triggerer logs | | `log_filename_template` | `dag_id=.../run_id=.../task_id=.../...` | Log path layout | > [!IMPORTANT] > In any deployment where workers are ephemeral (Kubernetes, Celery autoscaling, Edge), enable **remote logging** to durable storage. Otherwise task logs vanish when the worker/pod is recycled. ### 6.7 `[secrets]` — secrets backends | Key | Default | Purpose | |---|---|---| | `backend` | `""` | Alternative secrets backend class (Vault, AWS SM, GCP SM) | | `backend_kwargs` | `""` | JSON kwargs for the backend | | `use_cache` | `False` | Cache secrets lookups | | `cache_ttl_seconds` | `900` | Secrets cache TTL | A secrets backend lets connections and variables be resolved from an external store instead of the metadata DB. The lookup order is: secrets backend → environment variables → metadata DB. ### 6.8 Other notable sections - **`[metrics]`** — StatsD/OpenTelemetry metrics emission. - **`[traces]`** — OpenTelemetry distributed tracing. - **`[workers]` / `[sdk]` / `[execution_api]`** — Task-runtime and Execution-API tuning (worker API retries, execution-API behavior). - **`[api_auth]`** — API authentication backends/tokens. - **`[smtp]` / `[email]`** — Email/alert delivery. - **`[sentry]`** — Sentry error reporting. - **`[kerberos]`** — Kerberos integration for secured Hadoop/enterprise environments. - **`[triggerer]`** — Deferrable-task capacity and behavior. - **`[state_store]`** — Backend for task/asset state storage. > [!TIP] > Audit your effective configuration with `airflow config list`, and catch deprecated options after an upgrade with `airflow config lint`. --- ## 7. Executors The executor is the single most consequential operational choice. Core ships only `LocalExecutor`; everything else is a provider you install and select via `core.executor`. | Executor | Package | Model | Use when | |---|---|---|---| | **LocalExecutor** | core | Runs tasks as subprocesses on the scheduler host | Single-node deployments, small/medium load | | **CeleryExecutor** | `apache-airflow-providers-celery` | Distributes tasks to Celery workers via a broker (Redis/RabbitMQ) | Multi-node, horizontally scaled workers | | **KubernetesExecutor** | `apache-airflow-providers-cncf-kubernetes` | Launches one pod per task instance | Elastic, isolated, K8s-native workloads | | **LocalKubernetesExecutor** | cncf.kubernetes | Routes tasks to Local **or** K8s by queue | Mixed workloads on K8s | | **CeleryKubernetesExecutor** | celery | Routes tasks to Celery **or** K8s by queue | Hybrid scaling strategies | | **EdgeExecutor** | `apache-airflow-providers-edge3` | Dispatches tasks to remote edge workers over the Execution API | Geographically distributed / on-prem edge execution (experimental) | | **AWS ECS/Batch/Lambda executors** | `apache-airflow-providers-amazon` | Run tasks as ECS tasks, Batch jobs, or Lambda invocations | AWS-native serverless/containerized execution | > [!NOTE] > Airflow supports **multiple executors simultaneously** — assign tasks to an executor via the task's `queue`/executor configuration. Start with LocalExecutor (single node) or CeleryExecutor/KubernetesExecutor (scaled), and add specialized executors only where they earn their complexity. > [!CAUTION] > CeleryExecutor adds a broker (Redis/RabbitMQ) and result backend as additional operational dependencies. KubernetesExecutor trades per-task pod startup latency for isolation and elasticity. Choose deliberately based on task volume, duration, and isolation needs. --- ## 8. The Metadata Database The metadata DB is the **source of truth** for all Airflow state: DAG runs, task instances, serialized DAGs, XComs, connections, variables, pools, and job heartbeats. ### 8.1 Schema management ```bash airflow db migrate # apply pending migrations (run after install & every upgrade) airflow db check # verify connectivity and that migrations are current airflow db check-migrations # block until migrations are applied (useful in init containers) airflow db downgrade # revert to an earlier revision (dangerous) airflow db reset # DROP everything and recreate (destructive — dev only) airflow db shell # open an interactive DB shell ``` Migrations are Alembic-based (`alembic.ini`, `database.alembic_ini_file_path`). ### 8.2 Maintenance Airflow accumulates history. Periodically archive/purge old rows: ```bash # Archive rows older than a cutoff into archive tables, then optionally drop them airflow db clean --clean-before-timestamp 2026-01-01 airflow db export-archived --output-path ./archive airflow db drop-archived ``` > [!WARNING] > `db clean`, `db drop-archived`, `db downgrade`, and `db reset` mutate or destroy data. Always back up the database first and test on a copy. Run cleanup during low-traffic windows. ### 8.3 HA and connections - Multiple schedulers rely on **row-level locking** (`scheduler.use_row_level_locking = True`) — supported on PostgreSQL and MySQL, **not** SQLite. - Use connection pooling (`PgBouncer`) to keep connection counts sane as you scale processes. - Set `sql_alchemy_pool_recycle` below your DB/proxy idle-timeout to avoid stale connections. --- ## 9. Security ### 9.1 Authentication & authorization (auth managers) Airflow 3 makes auth pluggable via `core.auth_manager`: | Auth manager | Source | Provides | |---|---|---| | `SimpleAuthManager` (default) | core | Simple username/password auth; users defined in config (`simple_auth_manager_users`, default `admin:admin`) or a passwords file | | FAB auth manager | `apache-airflow-providers-fab` | Full RBAC (roles, permissions), the classic Flask-AppBuilder security model | | Keycloak / AWS / Google managers | provider packages | Enterprise SSO/IdP integration | > [!CAUTION] > The default `SimpleAuthManager` with `admin:admin` is for local use. For production, either configure SimpleAuthManager with a real passwords file (`core.simple_auth_manager_passwords_file`) and strong credentials, or install the FAB auth manager for RBAC, or integrate an external IdP (Keycloak/OIDC). Note community issue #67476: older builds ignored the `next` deep-link parameter during SimpleAuthManager auto-login — keep core patched. ### 9.2 Secrets at rest — the Fernet key Connection passwords and variable values are encrypted with the **Fernet key** (`core.fernet_key`). - Generate one key, store it in your secret manager, and use it across **all** components. - To rotate: add the new key, then run `airflow rotate-fernet-key` to re-encrypt existing secrets. - Losing the key means losing the ability to decrypt all stored secrets. ### 9.3 Secrets backends Move secrets out of the metadata DB entirely with a `[secrets]` backend (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). Connections and variables are then resolved from the external store. This is the recommended posture for regulated environments. ### 9.4 Network & transport - Terminate **TLS** at a reverse proxy in front of the API server (or use `api.ssl_cert`/`ssl_key` directly). - Keep `api.expose_config = False` and `api.expose_stacktrace = False` in production. - Restrict CORS via `api.access_control_allow_origins`. - Use `[api_auth]` tokens for programmatic/REST access; `airflowctl` authenticates with a token (`AIRFLOW_CLI_TOKEN`). ### 9.5 Execution isolation Because task code reaches Airflow only through the versioned **Execution API**, workers do not need database credentials. This lets you run workers (especially **edge** workers) in lower-trust networks while keeping the metadata DB private. Combine with `core.default_impersonation` / run-as-user controls where supported. > [!IMPORTANT] > DAG files are **executed Python** during parsing. Treat the dags folder/bundles as a code-trust boundary: only deploy DAGs from sources you trust, since a malicious DAG file runs with the DAG processor's privileges. --- ## 10. Observability & Operations ### 10.1 Health checks | Target | How | |---|---| | Scheduler | Enable `scheduler.enable_health_check`; probe port `8974` | | API server | HTTP probe on `8080` (health endpoint) | | Jobs | `airflow jobs check --job-type SchedulerJob --hostname ` | | Environment | `airflow info` | ### 10.2 Metrics & traces - **`[metrics]`** emits StatsD/OpenTelemetry metrics (scheduler loop timing, pool usage, task duration). Wire to your metrics stack. - **`[traces]`** emits OpenTelemetry spans for distributed tracing across components. - **`[sentry]`** integrates error reporting. ### 10.3 Logging operations - Default logs under `base_log_folder`; the UI fetches task logs over `worker_log_server_port` (`8793`) and triggerer logs over `8794`. - Enable `remote_logging` to S3/GCS/Azure/Elasticsearch (via the relevant provider) for ephemeral workers. ### 10.4 Scaling playbook | Symptom | Lever | |---|---| | Tasks queue but don't start | Add workers (Celery) / increase `parallelism`; check pools | | Scheduler loop slow | Add schedulers (HA); tune `max_tis_per_query`, `max_dagruns_*` | | DAGs slow to appear/update | Increase `dag_processor.parsing_processes`; lower `min_file_process_interval` | | DB connection exhaustion | Add PgBouncer; tune pool sizes | | Deferrable sensors pile up | Add triggerer capacity (`--capacity`) or more triggerers | | Logs lost on pod churn | Enable remote logging | ### 10.5 Upgrades 1. Read the release notes for breaking changes. 2. Back up the metadata DB. 3. Install the new core + providers with the matching constraint file. 4. Run `airflow db migrate`. 5. Run `airflow config lint` to catch deprecated options. 6. Roll services (DAG processor, scheduler, triggerer, API server). --- ## 11. Provider & Plugin Internals ### 11.1 Providers Providers are independently versioned Python packages (`apache-airflow-providers-*`) discovered at runtime via a provider-discovery mechanism. Each declares, through a `provider.yaml`/metadata, the operators, hooks, sensors, transfers, connection types, secrets backends, executors, logging handlers, and auth managers it contributes. Inspect what's installed: ```bash airflow providers list airflow providers hooks airflow providers secrets airflow providers executors airflow providers auth-managers ``` 87 providers ship in this repository, covering AWS (`amazon`), Google (`google`), Azure (`microsoft.azure`), Kubernetes (`cncf.kubernetes`), the common databases, messaging systems, and many SaaS integrations. ### 11.2 Plugins Plugins (in `core.plugins_folder`) extend Airflow with custom operator links, UI elements (React plugins), macros, and more. List loaded plugins with `airflow plugins`. The repository includes React plugin tooling (`dev/react-plugin-tools`) for building UI plugins against the new React UI. --- ## 12. Repository & Build Internals (for contributors and replicators) This section documents the codebase well enough that a developer — or a future AI — could navigate, build, and extend it. ### 12.1 Monorepo layout | Path | Contents | |---|---| | `airflow-core/src/airflow/` | The core platform: `cli/` (the `airflow` CLI, argparse-based in `cli/cli_config.py`), `models/` (ORM: DAG, TaskInstance, DagRun, …), `api_fastapi/` (the FastAPI app: `core_api`, `execution_api`, `auth/managers`), `config_templates/config.yml` (the config schema), `utils/`, `ui/` (the React 19 UI) | | `task-sdk/` | The Task SDK (`airflow.sdk`) — the stable authoring/runtime interface | | `airflow-ctl/` | `airflowctl`, the remote REST CLI client | | `providers/` | 87 provider packages, each `apache-airflow-providers-` | | `chart/` | The official Helm chart | | `go-sdk/`, `java-sdk/` | Experimental edge-worker SDKs (Go: Cobra/Viper; Java/Kotlin) | | `shared/` | Cross-cutting shared libraries (logging, serialization, timezones, config) | | `dev/breeze/` | **Breeze**, the containerized developer/CI environment and tooling | | `scripts/`, `docker-stack-docs/`, `docker-context-files/` | Image build and CI scripts | | `clients/python` | Generated REST API client | ### 12.2 The two CLIs — don't confuse them - The **`airflow` CLI** (`airflow-core/src/airflow/cli/cli_config.py`) is the user/operator surface, built on **argparse** with a `GroupCommand`/`ActionCommand` structure. This is what the User Guide documents. - **Breeze** (`dev/breeze`, a large Click application) is the **contributor** tool for building images, running tests, managing releases, and CI. Its ~500 Click commands dominate naive CLI extraction but are **not** end-user commands. Operators of a production Airflow do not run Breeze. > [!NOTE] > Frontend/UI work is React 19 + Vite + Chakra UI + TanStack Query, with an OpenAPI-generated client (`openapi-gen`) bound to the FastAPI API. The FAB provider still ships a legacy Flask/webpack UI for its auth-manager screens. ### 12.3 Building & testing Development uses **Breeze** (Dockerized): `breeze start-airflow` boots a full stack with hot reload; `breeze testing` runs the suite; `breeze ci-image`/`prod-image` build images; `breeze release-management` handles the elaborate ASF release process. Dependency resolution is constraint-driven (`generate-constraints`), and provider releases are independently versioned and tagged. --- ## 13. Improvements & Technical Debt (observed) Grounded in the source and the project's own issue tracker: - **Multi-scheduler timetable races:** Duplicate DAG runs can occur for partitioned/asset timetables under HA (issue #68045). If you run HA with such timetables, validate carefully and track the fix. - **Kubernetes client/proxy regression:** Older `cncf.kubernetes` client versions broke `KubernetesPodOperator` behind corporate proxies (issue #67845, resolved by supporting client `>=36.0.1`). Keep providers current. - **UI polish items:** DAG code syntax highlighting with f-strings (#67986) and DAG code readability improvements (#67359) are cosmetic UI items. - **SimpleAuthManager deep-link redirect (#67476):** auto-login dropped the `next` parameter in older builds. - **Chart templating gaps (#67951):** request to add `tpl` support to chart volumes/secret names. - **UI dependency consolidation (#61834):** redundant UI dependencies to be consolidated. General operational debt to plan around: the experimental status of the Go/Java edge SDKs and EdgeExecutor; the per-task pod startup latency of KubernetesExecutor; and the broker operational burden of CeleryExecutor. --- ## 14. Operational Troubleshooting ### Symptom: Scheduler appears alive but nothing schedules **Diagnose:** `airflow jobs check --job-type SchedulerJob`; confirm the DAG processor is running and DAGs are serialized (`airflow dags reserialize`); check `parallelism` and pool exhaustion. **Resolve:** Restart the stalled service; raise `parallelism`/pool slots; verify DB connectivity (`airflow db check`). ### Symptom: "Too many connections" on the database **Cause:** `(pool_size + max_overflow) × processes` exceeds DB limits. **Resolve:** Introduce PgBouncer; reduce per-process pool sizes; raise the DB connection limit. ### Symptom: Task logs missing in the UI **Cause:** Ephemeral workers without remote logging, or the log server port blocked. **Resolve:** Enable `logging.remote_logging` to durable storage; ensure ports `8793`/`8794` are reachable from the API server. ### Symptom: Migrations fail or schema mismatch on startup **Cause:** Upgraded binaries without running migrations. **Resolve:** `airflow db migrate`; use `airflow db check-migrations` as an init-container gate before starting services. ### Symptom: Deferrable sensors never resume **Cause:** No triggerer running, or triggerer at capacity. **Resolve:** Start/scale `airflow triggerer`; raise `--capacity`. ### Symptom: Secrets fail to decrypt after redeploy **Cause:** Fernet key mismatch across components or after rotation without `rotate-fernet-key`. **Resolve:** Ensure a single shared `core.fernet_key`; run `airflow rotate-fernet-key` only with both keys configured. --- ## 15. Quick Reference ### Production configuration minimums (env vars) | Variable | Set to | |---|---| | `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` | PostgreSQL/MySQL connection string | | `AIRFLOW__CORE__EXECUTOR` | `LocalExecutor` / `CeleryExecutor` / `KubernetesExecutor` | | `AIRFLOW__CORE__FERNET_KEY` | A generated Fernet key (shared) | | `AIRFLOW__CORE__LOAD_EXAMPLES` | `False` | | `AIRFLOW__API__SECRET_KEY` | A random, replica-consistent secret | | `AIRFLOW__LOGGING__REMOTE_LOGGING` | `True` (with a remote base folder + conn id) | | `AIRFLOW__CORE__AUTH_MANAGER` | Your chosen auth manager | ### Service ports | Port | Service | |---|---| | `8080` | API server (REST + UI) | | `8974` | Scheduler health check | | `8793` | Worker log server | | `8794` | Triggerer log server | ### Database lifecycle commands | Command | Use | |---|---| | `airflow db migrate` | Apply migrations | | `airflow db check` | Health check | | `airflow db check-migrations` | Block until migrated | | `airflow db clean` | Purge old metadata | | `airflow db downgrade` | Revert schema (dangerous) | | `airflow db reset` | Drop & recreate (dev only) | ### Config introspection | Command | Use | |---|---| | `airflow config list` | Dump effective config | | `airflow config get-value
` | Read one value | | `airflow config lint` | Find deprecated options | --- _Generated June 2026 from apache/airflow at 0dd0189. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._