# TiDB Administrator & Architecture Guide

This guide is the operational and architectural reference for **TiDB**, the open-source distributed SQL database by PingCAP. It is written for administrators, SREs, architects, and developers who deploy, configure, secure, operate, and extend TiDB. Every fact here is derived directly from the source at commit `5e0c630`.

It covers the system architecture, a complete configuration reference for `tidb-server`, security hardening, operations (startup, shutdown, monitoring, backup), and a deep dive into the internals — the DDL job engine, the Distributed eXecution Framework (DXF), and the code layout — sufficient for a reader to reason about, extend, or replicate the system.

> [!NOTE]
> **Scope of this repository.** `pingcap/tidb` is the **SQL layer** plus the ecosystem tools (BR, TiDB Lightning, Dumpling). The distributed KV store **TiKV**, the cluster controller **PD** (Placement Driver), and the columnar engine **TiFlash** are separate projects that TiDB communicates with over gRPC. This guide documents how TiDB *uses* them, not their internals.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Architecture](#2-architecture)
3. [System Requirements](#3-system-requirements)
4. [Configuration Mastery](#4-configuration-mastery)
5. [Security](#5-security)
6. [Operations](#6-operations)
7. [Backup, Restore & Disaster Recovery](#7-backup-restore--disaster-recovery)
8. [Observability](#8-observability)
9. [Internals: How TiDB Works](#9-internals-how-tidb-works)
10. [Repository Structure](#10-repository-structure)
11. [Code Quality & Conventions](#11-code-quality--conventions)
12. [Troubleshooting & Diagnostics](#12-troubleshooting--diagnostics)
13. [Potential Improvements](#13-potential-improvements)
14. [Quick Reference](#14-quick-reference)

---

## 1. Executive Summary

TiDB separates **compute** from **storage**:

- **`tidb-server`** — a stateless SQL engine. It terminates MySQL-protocol connections, parses and optimizes SQL, and executes plans by issuing key-value and coprocessor requests to the storage layer. Because it holds no durable state, you scale SQL throughput by adding more `tidb-server` instances behind a load balancer, and any instance can fail without data loss.
- **TiKV** — a distributed, transactional, Raft-replicated key-value store. It holds the actual data, sharded into "regions" and replicated for high availability.
- **PD (Placement Driver)** — the cluster brain: it allocates timestamps (the global clock for MVCC/transactions), stores cluster metadata, and schedules region placement and balancing.
- **TiFlash** (optional) — a columnar replica of selected tables, kept in sync from TiKV, for analytical (OLAP) queries. This is what makes TiDB **HTAP**.

Key operational properties:

| Property | Mechanism |
|----------|-----------|
| Strong consistency | Raft majority commit in TiKV; global timestamps from PD |
| ACID transactions | Percolator-style two-phase commit; optimistic and pessimistic modes |
| Online schema change | Asynchronous, job-based DDL with a cluster-wide schema-version protocol |
| Horizontal scale | Add `tidb-server` (compute) and TiKV (storage) independently, online |
| MySQL compatibility | MySQL 8.0 wire protocol, error codes, system variables, metadata schemas |

---

## 2. Architecture

### 2.1 Cluster topology

```text
                         ┌──────────────────────────────┐
        MySQL clients ──▶│  Load balancer (L4/L7)        │
                         └───────────────┬──────────────┘
                                         │  MySQL protocol :4000
        ┌────────────────────────────────┼────────────────────────────────┐
        ▼                                 ▼                                 ▼
 ┌─────────────┐                  ┌─────────────┐                   ┌─────────────┐
 │ tidb-server │                  │ tidb-server │   ... (stateless) │ tidb-server │
 │  :4000 SQL  │                  │  :4000 SQL  │                   │  :4000 SQL  │
 │  :10080 HTTP│                  │  :10080 HTTP│                   │  :10080 HTTP│
 └──────┬──────┘                  └──────┬──────┘                   └──────┬──────┘
        │  gRPC (KV + coprocessor)       │                                 │
        └────────────────┬───────────────┴─────────────────┬───────────────┘
                         ▼                                  ▼
                 ┌───────────────┐                   ┌───────────────┐
                 │     PD        │  timestamps,       │   TiKV nodes  │  row store
                 │  :2379 /2380  │  metadata,         │   :20160      │  (Raft groups)
                 │ (Raft, 3+)    │  scheduling        │   (3+ replicas)│
                 └───────────────┘                   └───────┬───────┘
                                                             │ Multi-Raft Learner
                                                             ▼
                                                     ┌───────────────┐
                                                     │   TiFlash     │  columnar
                                                     │ (optional)    │  (HTAP/OLAP)
                                                     └───────────────┘
```

### 2.2 The `tidb-server` process

`tidb-server` is a single Go binary whose `main()` (`cmd/tidb-server/main.go`) performs a deterministic startup sequence:

1. **Parse flags** (`initFlagSet`) and **load config** (`config.InitializeConfig`), merging file + flags.
2. **Register store drivers** (`registerStores`) — TiKV, MockTiKV, and the embedded UniStore.
3. **Initialize globals** — logging (`setupLog`), metrics (`setupMetrics`), memory hooks, CPU profiling, extensions, statement-summary, tracing.
4. **Create the store, DDL owner manager, and Domain** (`createStoreDDLOwnerMgrAndDomain`):
   - Connect to the store (`kvstore.MustInitStorage`).
   - Verify the PD/TiDB **kernel type** matches (classic vs next-gen) — a mismatch aborts startup.
   - Start the **DDL owner manager** (`ddl.StartOwnerManager`).
   - **Bootstrap a session** (`session.BootstrapSession`) which loads/creates the system schema and returns the **Domain**.
5. **Create the SQL server** (`createServer`) — wraps the storage with a `TiDBDriver`, builds the MySQL-protocol `server.Server`, and attaches background handles (expensive-query killer, memory-usage alarm, server-memory-limit enforcement).
6. **Install signal handlers** for graceful shutdown and **run** (`svr.Run(dom)`).

### 2.3 The Domain

The **Domain** (`pkg/domain`) is the per-instance hub that ties together:

- the **InfoSchema** (the in-memory snapshot of all databases/tables/indexes) and its synchronizer,
- the **DDL** subsystem and its owner election,
- statistics handling, the timer/TTL subsystems, plan replayer, and the info-syncer that publishes this server's presence to the cluster.

Every `tidb-server` has exactly one Domain; it is the root object from which the executor, planner, and session layers obtain schema and cluster state.

### 2.4 Request lifecycle

A SQL statement flows: **MySQL protocol** (`pkg/server`) → **session** (`pkg/session`) → **parser** (`pkg/parser`, builds an AST) → **planner** (`pkg/planner`, logical + physical optimization, `pkg/planner/core`) → **executor** (`pkg/executor`, the volcano-style operator tree) → **distsql / store** (`pkg/distsql`, `pkg/store`) which pushes coprocessor requests to TiKV and reads back results. Expression evaluation lives in `pkg/expression`; type handling in `pkg/types`; row/key encoding in `pkg/tablecodec` and `pkg/kv`.

### 2.5 Kernel types: classic vs next-gen

The source distinguishes two **kernel types** (`pkg/config/kerneltype`):

- **Classic** — the traditional architecture (PD + TiKV; no distributed file system).
- **Next-gen** — a multi-tenant, keyspace-aware architecture that adds a **TiKV-Worker** component and S3-compatible **DFS** (distributed file system) integration. Next-gen requires a `--keyspace-name` (or standby mode) and forbids manual version overrides.

A `tidb-server` checks at startup that its kernel type matches PD's; a mismatch is a fatal `kernel type mismatch` error.

---

## 3. System Requirements

### 3.1 Build

| Item | Requirement |
|------|-------------|
| Go | `1.25.10` (root module); `1.25` for `pkg/parser` |
| Build system | `make` (targets: `make server`, `make build_for_br_integration_test`, etc.) |
| Platform | Linux/macOS for full functionality; Windows lacks secure bootstrap |

### 3.2 Runtime ports

| Component | Port(s) | Purpose |
|-----------|---------|---------|
| `tidb-server` SQL | `4000` | MySQL protocol |
| `tidb-server` status | `10080` | HTTP: health, metrics, schema, pprof |
| PD | `2379` (client), `2380` (peer) | Timestamps, metadata, scheduling |
| TiKV | `20160` (service), `20180` (status) | KV storage |
| TiKV-Worker (next-gen) | `19000` | Coprocessor offload |

### 3.3 External storage (for BR / Lightning / DFS)

TiDB's tooling integrates with object storage via vendored SDKs (from `go.mod`): **Amazon S3** (`aws-sdk-go-v2`), **Google Cloud Storage** (`cloud.google.com/go/storage`), **Azure Blob** (`azure-sdk-for-go`), and **Aliyun OSS**. Storage URLs use schemes `s3://`, `gcs://`, `azure://`, `local://`.

---

## 4. Configuration Mastery

`tidb-server` is configured by a **TOML file** (`--config`) deserialized into the `Config` struct (`pkg/config/config.go`). Command-line flags override matching keys. This section documents the structure and the load-bearing defaults; defaults are the constants defined alongside the struct.

> [!TIP]
> Validate before deploying: `tidb-server --config tidb.toml --config-check`. Add `--config-strict` in CI to fail on unknown keys. Unknown keys are otherwise warned-about, not fatal.

### 4.1 Top-level keys

| TOML key | Default | Meaning |
|----------|---------|---------|
| `host` | `0.0.0.0` | SQL bind address |
| `advertise-address` | (derived) | Address advertised to the cluster |
| `port` | `4000` | SQL port |
| `store` | `unistore` | `tikv`, `unistore`, or `mocktikv` |
| `path` | `/tmp/tidb` | Storage path, or PD endpoints when `store=tikv` |
| `socket` | `/tmp/tidb-{Port}.sock` | Unix socket |
| `lease` | `45s` | **Schema lease** — interval for schema-version sync. **Do not change casually.** |
| `token-limit` | `1000` | Max concurrent executing sessions (cap `1048576`) |
| `split-table` | — | Pre-split a region per new table to avoid hotspots |
| `temp-dir` | `/tmp/tidb` | Temp directory |
| `tmp-storage-path` | (encoded per host/port) | Spill-to-disk location for large queries |
| `tmp-storage-quota` | — | Byte quota for spill storage; startup fails if it exceeds capacity |
| `max-index-length` | `3072` | Max index length in bytes (MySQL-compatible; max `12288`) |
| `index-limit` | `64` | Max indexes per table (max `512`) |
| `table-column-count-limit` | `1017` | Max columns per table (max `4096`) |
| `max-allowed-packet` | `67108864` (64 MiB) | Max packet size (starter mode) |
| `new_collations_enabled_on_first_bootstrap` | — | Enables new collation framework (first-bootstrap only) |
| `enable-table-lock` | — | Enable `LOCK TABLES` support |
| `enable-global-kill` | — | Cluster-wide `KILL <connid>` |
| `graceful-wait-before-shutdown` | `0` | Seconds to report unhealthy before shutdown (drains LBs) |
| `repair-mode` / `repair-table-list` | false / — | Table-meta repair mode |
| `deploy-mode` / `keyspace-name` | — | Next-gen deployment/keyspace selection |

### 4.2 Config sections

The `Config` struct nests these sections (each a TOML `[section]`):

| Section | Purpose | Notable keys |
|---------|---------|--------------|
| `[log]` | Logging | `level`, `format`, `slow-query-file`, `file.filename`, `file.max-days`, `file.max-size` |
| `[instance]` | Instance-scoped settings (the modern home for many options) | `tidb_enable_ddl`, `max_connections`, `tidb_general_log`, `tidb_slow_log_threshold`, `tidb_memory_usage_alarm_ratio`, `plugin_load`, `plugin_dir` |
| `[security]` | TLS, auth, bootstrap | `ssl-ca`, `ssl-cert`, `ssl-key`, `cluster-ssl-ca/cert/key`, `secure-bootstrap`, `skip-grant-table`, `disconnect-on-expired-password`, `auto-tls` |
| `[status]` | Status/HTTP server | `report-status`, `status-host`, `status-port`, `metrics-addr`, `metrics-interval` |
| `[performance]` | Resource tuning | `max-procs`, `server-memory-quota`, `txn-total-size-limit`, `txn-entry-size-limit`, `stats-lease`, `gogc`, `cross-join`, `distinct-agg-push-down`, `enforce-mpp` |
| `[prepared-plan-cache]` | Prepared-statement plan cache | `enabled`, `capacity`, `memory-guard-ratio` |
| `[pd-client]` | PD client tuning | timeouts, retry, request concurrency |
| `[tikv-client]` | TiKV client tuning | `grpc-connection-count`, `commit-timeout`, `region-cache-ttl`, `store-liveness-timeout`, `copr-cache` |
| `[pessimistic-txn]` | Pessimistic locking | `max-retry-count`, `deadlock-history-capacity`, `constraint-check-in-place-pessimistic` |
| `[isolation-read]` | Engine routing | `engines` (e.g. `["tikv","tiflash"]`) |
| `[proxy-protocol]` | PROXY protocol | `networks`, `header-timeout`, `fallbackable` |
| `[opentracing]` | Jaeger tracing | `enable`, sampler/reporter settings |
| `[top-sql]` | Top-SQL CPU attribution | `receiver-address` |
| `[transaction-summary]` | Transaction recording | `transaction-summary-capacity`, `transaction-id-digest-min-duration` |
| `[experimental]` | Experimental flags | gated features |
| `[standby]` | Standby/serverless activation | `standby`, `activation-timeout`, `max-idle-seconds` |

### 4.3 Key tuning defaults (from source constants)

| Constant | Default | Notes |
|----------|---------|-------|
| `DefSchemaLease` | `45s` | Schema-version lease |
| `DefTxnEntrySizeLimit` | `6 MiB` | Single-entry size cap; hard max `120 MiB` |
| `DefTxnTotalSizeLimit` | `100 MiB` | When left at default, a large per-session memory tracker effectively governs txn size instead |
| `DefMemoryUsageAlarmRatio` | `0.8` | Fraction of memory that triggers an alarm dump |
| `DefExpensiveQueryTimeThreshold` | `60s` | Logs queries exceeding this |
| `DefExpensiveTxnTimeThreshold` | `600s` | Logs long-running transactions |
| `DefDDLSlowOprThreshold` | `300ms` | Logs slow DDL operations |
| `MaxLogFileSize` | `4096 MB` | Per-log-file cap |
| `DefMaxAllowedPacket` | `64 MiB` | — |

### 4.4 Deprecated / moved options

The source maintains a `sectionMovedToInstance` map: several options that historically lived at the top level or in `[log]`/`[performance]`/`[plugin]` have moved to the `[instance]` section and/or become **system variables**. Examples:

| Old config key | New home |
|----------------|----------|
| `run-ddl` | `[instance] tidb_enable_ddl` |
| `max-server-connections` | `[instance] max_connections` |
| `check-mb4-value-in-utf8` | `[instance] tidb_check_mb4_value_in_utf8` |
| `log.enable-slow-log` | `[instance] tidb_enable_slow_log` |
| `log.slow-threshold` | `[instance] tidb_slow_log_threshold` |
| `performance.force-priority` | `[instance] tidb_force_priority` |
| `plugin.load` / `plugin.dir` | `[instance] plugin_load` / `plugin_dir` |

TiDB still reads the old keys for upgrade compatibility but logs them as deprecated. Migrate to the new locations.

> [!IMPORTANT]
> Many runtime behaviors are governed by **system variables** (`SET GLOBAL tidb_...`), not config keys — they are stored in the cluster and persist across restarts. Use `--initialize-sql-file` to set GLOBAL variables on first bootstrap, since they are not read from the config file. Config keys generally set process/instance defaults; system variables set cluster-wide behavior.

---

## 5. Security

TiDB's security surface spans network TLS, authentication, authorization, the status port, and an optional Security Enhanced Mode.

### 5.1 Transport security (TLS)

There are **two independent TLS channels**, configured separately:

| Channel | Config keys | Flags |
|---------|-------------|-------|
| **Client ↔ TiDB** (SQL) | `[security] ssl-ca`, `ssl-cert`, `ssl-key` | `--sql-ca`, `--sql-cert`, `--sql-key` |
| **TiDB ↔ TiKV/PD** (cluster) | `[security] cluster-ssl-ca`, `cluster-ssl-cert`, `cluster-ssl-key` | `--cluster-ca`, `--cluster-cert`, `--cluster-key` |

The startup code enforces that `cert` and `key` are set **together**, and that a `ca` requires both. Corresponding environment variables exist (`SQL_CA`, `SQL_CERT`, `SQL_KEY`, `CLUSTER_CA`, …) for containerized deployments.

> [!CAUTION]
> Enable cluster TLS in any multi-host deployment. Without it, TiDB↔TiKV↔PD gRPC traffic — including data and timestamps — crosses the network in plaintext.

### 5.2 Authentication & bootstrap

- **Insecure bootstrap (default):** `--initialize-insecure=true` creates a `root` account with an **empty password**. Convenient for testing, unsafe for production.
- **Secure bootstrap:** `--initialize-secure=true` uses Unix-socket authentication for the initial setup (not available on Windows).
- **Skip-grant mode:** `[security] skip-grant-table` (or the `SkipWithGrant` path) disables privilege checks entirely — an emergency-recovery escape hatch only. Never run this exposed.
- **Password lifecycle:** `disconnect-on-expired-password` (default true) disconnects clients with expired passwords; disabling it puts such sessions into a sandbox mode. Errors `privilege:1862 Your password has expired` surface this.
- **LDAP / JWT / auth tokens:** Pluggable auth methods exist (`pkg/privilege/privileges/ldap`, JWT via `lestrrat-go/jwx`), including a configurable auth-token refresh interval (`DefAuthTokenRefreshInterval = 1h`).

### 5.3 Authorization

Privileges follow the MySQL model (`pkg/privilege`): `GRANT`/`REVOKE`, user@host matching, roles, and dynamic privileges. Access failures map to MySQL codes (`1045` access denied, `1142` command denied, `1141` no such grant, `3530` role not granted). Privilege data lives in the `mysql.*` system tables and is cached in the Domain.

### 5.4 Status port hardening

The status HTTP server (`:10080`) exposes profiling (`/debug/pprof/*`), a diagnostics bundle (`/debug/zip`), config (`/config`), and schema introspection. Restrict it:

- Bind `status-host` to an internal interface, not `0.0.0.0`, in production.
- Firewall port `10080` from client networks.
- Set `skip-register-to-dashboard` if you do not want a server advertised to TiDB Dashboard.

### 5.5 Security Enhanced Mode (SEM)

TiDB ships a **Security Enhanced Mode** (`pkg/util/sem`, set up via `setupSEM`) that, when enabled, restricts powerful operations and hides sensitive system-table contents even from privileged users — useful for managed/multi-tenant deployments where the operator must protect data from `root`.

### 5.6 Log redaction

Both `tidb-server` and BR support **log redaction** (`redact-info-log`, the `--redact` collect-log subcommand) to strip sensitive values from logs before they leave the host — important when shipping logs to third parties.

### 5.7 Handling untrusted SQL workloads

Resource-control features (resource groups, runaway-query detection under `pkg/resourcegroup`) let you cap the resources a tenant or query can consume, mitigating noisy-neighbor and accidental-DoS risks from untrusted SQL.

---

## 6. Operations

### 6.1 Starting a production instance

```bash
tidb-server \
  --store=tikv \
  --path="pd1:2379,pd2:2379,pd3:2379" \
  --config=/etc/tidb/tidb.toml \
  --host=0.0.0.0 --port=4000 \
  --status=10080 \
  --advertise-address="$(hostname -i)" \
  --log-file=/var/log/tidb/tidb.log
```

Deployment checklist:

- [ ] PD endpoints reachable from this host
- [ ] Cluster TLS certs in place (`cluster-ssl-*`) if multi-host
- [ ] `advertise-address` set to a cluster-reachable IP (auto-derived if `host=0.0.0.0`)
- [ ] Config validated (`--config-check`)
- [ ] Log directory writable; rotation configured (`log.file.max-days`, `max-size`)
- [ ] `tmp-storage-path` on a volume with adequate free space ≥ `tmp-storage-quota`
- [ ] Load balancer health-checks `GET /status`

### 6.2 Graceful shutdown

On `SIGTERM`/`SIGINT`, `tidb-server`: closes the SQL server (`svr.Close()`), stops the resource manager, cleans up the Domain and storage, stops the CPU profiler, flushes logs, then exits. To drain connections gracefully, set `graceful-wait-before-shutdown` so the status endpoint reports unhealthy first and the load balancer stops sending new connections before the process stops accepting work.

> [!NOTE]
> A `SIGINT`-driven shutdown returns exit code `130` (`128 + SIGINT`) so deployment scripts can distinguish a forced/standby shutdown from a normal exit.

### 6.3 Rolling upgrades & online DDL during upgrade

Because DDL is owner-driven and version-synchronized, upgrades are rolling: replace `tidb-server` instances one at a time behind the load balancer. The source includes explicit machinery to **pause user DDL while upgrading** (design doc `2023-04-11-pause-user-ddl-when-upgrading.md`) so schema changes don't race the bootstrap-version migration. Let the cluster finish its bootstrap upgrade before resuming heavy DDL.

### 6.4 Memory & OOM protection

TiDB has layered memory protection:

- **Per-query / global trackers** — `server-memory-quota` and the global memory tracker bound total query memory; queries spill to `tmp-storage-path` when `EnableTmpStorageOnOOM` is on.
- **Memory-usage alarm** — at `memory-usage-alarm-ratio` (default `0.8`) of available memory, TiDB writes an alarm + profile for diagnosis.
- **Server memory limit handle** — a background goroutine (`ServerMemoryLimitHandle`) enforces `tidb_server_memory_limit` by cancelling the most memory-hungry queries.
- **Expensive-query killer** — queries exceeding `expensive-query-time-threshold` are logged and can be killed.

### 6.5 Resource control

Resource groups (`pkg/resourcegroup`) let you allocate Request Units (RUs) per workload and detect/limit "runaway" queries — essential for multi-tenant clusters. RU weighting is tunable via the `[ru-v2]` config (the source ships experimentally-fitted default weights in `DefaultRUV2Config`).

### 6.6 Time-to-live (TTL) tables

TiDB has a built-in TTL subsystem (`pkg/ttl`) — define a row-expiry policy on a table and background TTL workers delete expired rows on a schedule, governed by the timer framework (`pkg/timer`). This avoids external cron jobs for data-retention enforcement.

---

## 7. Backup, Restore & Disaster Recovery

The end-user mechanics of BR (`br`) are in the **User Guide §8**. This section covers the operational/DR posture.

### 7.1 Backup strategy

| Layer | Tool | Cadence |
|-------|------|---------|
| Periodic full snapshot | `br backup full` | Daily/weekly to object storage |
| Continuous change capture | `br log start` (log backup) | Always-on, streams to storage |
| Point-in-time recovery | `br restore point` | On demand: base snapshot + log replay to a target TS |

The combination of a base full backup and a continuous log backup gives **point-in-time recovery** to any timestamp within the retention window.

### 7.2 BR internals worth knowing

- BR runs as a client that talks to PD/TiKV directly; during backup it **disables its own memory-limit tuner** because server memory is measured on the TiDB node, not the BR host (`br/cmd/br/cmd.go`).
- BR computes a **memory limit** for itself based on free memory (`calculateMemoryLimit`) and starts a heap-dump-on-pressure monitor (`BR_HEAP_DUMP_DIR`, default `/tmp/br_heap_dumps`).
- During backup BR sets `AdvertiseAddress` to an unavailable IP and disables coprocessor caching so it doesn't pollute cluster state.
- `br operator` provides tidb-operator integration: `prepare-for-snapshot-backup` (pause GC/schedulers), `base64ify`, migration listing/applying, and checksum subcommands.

### 7.3 Verifying backups

```bash
br debug backupmeta validate --storage <url>   # checks key ranges & rewrite rules
br debug checksum            --storage <url>   # recomputes & compares checksums
```

A failed verification raises `BR:Backup:ErrBackupChecksumMismatch`. Treat such a backup as unusable.

### 7.4 GC and backup safepoints

TiDB's garbage collection reclaims old MVCC versions. Backups coordinate with GC via **safepoints** so data needed by an in-progress backup isn't collected; a mis-set safepoint surfaces as `BR:Backup:ErrBackupGCSafepointExceeded`. Don't shorten the GC lifetime below your backup window.

---

## 8. Observability

### 8.1 Metrics

Every `tidb-server` exposes Prometheus metrics at `GET /metrics` (`:10080`). It can also **push** to a Pushgateway (`--metrics-addr`, `--metrics-interval`, default `15s`; `0` disables). Metric definitions live in `pkg/metrics`; pre-built Grafana dashboards are generated from jsonnet in `pkg/metrics/grafana`.

### 8.2 Status / introspection API

The status server (`pkg/server/http_status.go`) is the operational control plane. Highlights:

| Endpoint | Use |
|----------|-----|
| `/status` | Health/version (load-balancer probe) |
| `/info`, `/info/all` | This node / whole cluster topology |
| `/schema`, `/schema/{db}/{table}` | Live schema introspection |
| `/ddl/history`, `/ddl/owner/resign` | DDL job history; force owner re-election |
| `/config` | Effective configuration |
| `/tables/{db}/{table}/regions` | Region layout (hotspot analysis) |
| `/regions/hot`, `/regions/meta` | Hot/region metadata |
| `/mvcc/key/{db}/{table}/{handle}` | Inspect MVCC versions of a row |
| `/debug/pprof/*` | Go profiling |
| `/debug/zip` | One-shot diagnostics bundle for support |
| `/dxf/task/active`, `/dxf/schedule/status` | Distributed-task framework state |

### 8.3 Logs

| Log | Config | Content |
|-----|--------|---------|
| Server log | `--log-file`, `[log]` | Operational events; level via `-L` |
| Slow-query log | `--log-slow-query`, `tidb_slow_log_threshold` | Queries over the threshold; also in `information_schema.slow_query` |
| General log | `--log-general`, `tidb_general_log` | Every statement (debugging only — high volume) |

### 8.4 In-SQL observability

- **Top-SQL** (`[top-sql]`) attributes CPU to specific SQL digests.
- **Statement summary** (`pkg/util/stmtsummary`) aggregates per-digest latency/row stats, queryable via `INFORMATION_SCHEMA`.
- **Continuous profiling** integrates with Grafana Pyroscope (`PYROSCOPE_SERVER_ADDRESS` env var + related auth vars) and an internal CPU profiler.
- **Cluster tables** — `information_schema.cluster_info`, `cluster_log`, `slow_query`, and the `metrics_schema` expose cluster-wide state from any node.

---

## 9. Internals: How TiDB Works

This section is the developer/architect deep dive — enough to extend or reason about the system. It is grounded in the in-tree architecture notes under `docs/agents/` and the package layout.

### 9.1 Online DDL: the job engine

TiDB DDL is **job-based** and **owner-driven**. A DDL SQL statement does not mutate schema in place; it becomes a persistent **job** that the cluster's single **DDL owner** schedules and runs through a state machine. Flow (`docs/agents/ddl`):

```text
Session ─▶ DDLExec (pkg/executor/ddl.go)
        ─▶ DDL Executor (pkg/ddl/executor.go)   converts AST → model.Job
        ─▶ JobSubmitter (pkg/ddl/job_submitter.go)  allocates IDs, INSERTs into mysql.tidb_ddl_job, notifies owner
        ─▶ DDL Owner / jobScheduler (pkg/ddl/job_scheduler.go)  dispatches to worker pools
        ─▶ Workers (pkg/ddl/job_worker.go)  run job step-by-step, advancing schema states
        ─▶ Syncer (pkg/ddl/schemaver/syncer.go)  bumps global schema version, waits for all nodes to sync (etcd)
        ─▶ job moved to history; submitting session unblocks
```

The schema-state machine for an index/column change is `delete only → write only → reorganization → public`. This staged progression — combined with the cluster-wide schema-version sync and the schema **lease** (`45s`) — is what makes DDL safe to run **online** while other nodes still serve the old schema. A reorg/backfill step (e.g. `ADD INDEX`) scans and writes data with checkpoints so it can resume after owner failover.

> [!WARNING]
> The most common mistake when extending TiDB is implementing DDL behavior directly in the executor layer (`pkg/executor`), bypassing job persistence, the schema state machine, version sync, and MDL/lease safety. New DDL must go through the job framework.

Key system tables: `mysql.tidb_ddl_job` (queue), `mysql.tidb_ddl_reorg` (reorg progress), plus MDL (metadata-lock) tables. Inspect from SQL with `ADMIN SHOW DDL [JOBS]`, or force owner re-election via `POST /ddl/owner/resign`.

### 9.2 Distributed eXecution Framework (DXF)

DXF (`pkg/dxf`) is TiDB's framework for running large work **across all nodes** instead of on a single `tidb-server`. It powers distributed `ADD INDEX` backfill and `IMPORT INTO`. Structure (`docs/agents/dxf`):

- `pkg/dxf/framework/proto` — task/step/state enums.
- `pkg/dxf/framework/storage` — task & subtask persistence (the dist-task tables).
- `pkg/dxf/framework/scheduler` — **owner-side** scheduler that admits tasks, reserves slots, and advances steps. Runs only on the DDL owner node.
- `pkg/dxf/framework/taskexecutor` — **node-side** executor manager that runs subtasks on every node and reports progress.

Runtime: business logic submits a task → framework persists it → every node runs an executor-manager loop → the owner runs the scheduler loop that advances steps and dispatches subtasks → executors run subtasks and report state. Task rank (priority, create_time, id) drives both scheduling and preemption; slot reservation governs concurrency (`MaxConcurrentTask`).

To add a new distributed task type: define task/step enums (keeping step constants backward-compatible), implement a `scheduler.Extension` and a `taskexecutor.Extension`, and register both. A minimal skeleton lives in `pkg/dxf/example`.

### 9.3 `IMPORT INTO` and conflict handling

`IMPORT INTO` (a DXF app, `pkg/dxf/importinto`) has two paths:

- **Local sort:** `Init → Import → PostProcess → Done`.
- **Global sort:** `Init → EncodeAndSort → MergeSort → WriteAndIngest → CollectConflicts → ConflictResolution → PostProcess → Done`.

On global sort, duplicate keys are detected and written to `conflicted-rows/<task-id>/<subtask-id>-<uuid>` — deliberately **outside** the task's cleanup path so they survive for inspection. If the in-memory conflict-handle set exceeds its bound (`TooManyConflictsFromIndex`), the final checksum verification is skipped. Cleanup (`clean_up.go`) switches the table mode back to normal, deletes task-scoped global-sort artifacts (but not the conflict rows), and, in next-gen, emits metering.

### 9.4 Transactions

TiDB implements Percolator-style **two-phase commit** over TiKV with global timestamps from PD. Both **optimistic** (conflicts detected at commit) and **pessimistic** (locks acquired during execution) modes are supported; the pessimistic path is tuned under `[pessimistic-txn]` (retry count, deadlock-history capacity, in-place constraint checks). Transaction size is bounded by `txn-entry-size-limit` (6 MiB default, 120 MiB hard max) and `txn-total-size-limit`. A deadlock-history ring buffer aids diagnosis (`information_schema.deadlocks`).

### 9.5 Query optimizer

The planner (`pkg/planner`, `pkg/planner/core`, ~2.7 MB of source — the single largest subsystem along with the executor and expression engine) performs logical optimization (rule-based: predicate pushdown, column pruning, join reordering) and physical optimization (cost-based, using statistics from `pkg/statistics`). A **cascades**-style optimizer (`pkg/planner/cascades`) is also present. Plan and execution can be inspected with `EXPLAIN` / `EXPLAIN ANALYZE`, the plan cache (`[prepared-plan-cache]`), SQL plan bindings (`pkg/bindinfo`), and plan replayer dumps (`/plan_replayer/dump`).

### 9.6 Storage abstraction

`pkg/kv` defines the transactional KV interface; `pkg/store` provides drivers — the real `tikv` driver (`pkg/store/driver`, wrapping `tikv/client-go`), the embedded `unistore` (`pkg/store/mockstore/unistore`), and a mock. `pkg/tablecodec` encodes rows and indexes into KV keys; `pkg/distsql` builds and dispatches coprocessor requests that push computation down to TiKV.

---

## 10. Repository Structure

The codebase is a large Go monorepo. The highest-signal directories (by size and importance):

| Path | Size¹ | Responsibility |
|------|------|----------------|
| `pkg/expression` | ~3.9 MB | SQL expression & built-in function evaluation |
| `pkg/executor` | ~3.3 MB | Physical operator execution (volcano model) |
| `pkg/ddl` | ~3.0 MB | Online schema-change engine (job-based) |
| `pkg/planner/core` | ~2.7 MB | Query optimizer (logical + physical) |
| `pkg/infoschema` | ~627 KB | In-memory schema snapshot & sync |
| `pkg/session` | ~597 KB | Session lifecycle, bootstrap, SQL entry |
| `dumpling/export` | ~454 KB | Dumpling logical-export engine |
| `pkg/server` | ~432 KB | MySQL protocol server + status HTTP API |
| `pkg/statistics` | ~366 KB | Table statistics & auto-analyze |
| `pkg/ttl` | ~368 KB | TTL (row-expiry) workers |
| `pkg/parser` | ~298 KB | SQL parser (separate Go module) |
| `pkg/domain` | ~264 KB | Per-instance hub (schema, DDL, stats, syncers) |
| `pkg/errno` | ~234 KB | Error code definitions (MySQL-compatible) |
| `pkg/config` | ~178 KB | Configuration struct & loader |
| `br/` | — | Backup & Restore tool |
| `lightning/`, `pkg/lightning` | — | TiDB Lightning importer |
| `dumpling/` | — | Dumpling exporter |
| `cmd/tidb-server` | — | The server `main()` entry point |
| `pkg/dxf` | — | Distributed eXecution Framework |
| `docs/design`, `docs/agents` | — | Design docs & internal navigation maps |

¹ Approximate source size of the HIGH-priority modules as triaged during analysis.

Entry points (`func main`): `cmd/tidb-server/main.go`, `br/cmd/br/main.go`, `lightning/cmd/tidb-lightning/main.go`, `lightning/cmd/tidb-lightning-ctl`, `dumpling/cmd/dumpling/main.go`, plus dev utilities under `cmd/` (`benchdb`, `benchkv`, `benchraw`, `importer`, `ddltest`, `mirror`, `pluginpkg`) and `tools/`.

---

## 11. Code Quality & Conventions

- **Language & style:** Go 1.25, enforced by an extensive linter set (golangci-lint, revive, gosec, custom analyzers in `tools/`). License headers are mandatory (checked by skywalking-eyes).
- **Error handling:** Errors are namespaced `Component:Subsystem:ErrName` and auto-documented into `errors.toml` (3,623 lines) by `errdoc-gen`. User-facing errors carry MySQL-compatible numeric codes (`pkg/errno`).
- **Testing:** Heavy use of unit tests, integration tests (shell-driven under `br/tests`, `lightning/tests`, `dumpling/tests`), and real-TiKV tests (`tests/realtikvtest`). Determinism is aided by `failpoint` injection. There is even an LLM-assisted test harness (`tests/llmtest`).
- **Internal docs:** `docs/agents/*` provide "read-first" navigation maps for the trickiest subsystems (DDL, DXF, IMPORT INTO, planner) with the explicit rule that code is the source of truth and docs must be updated when they drift.
- **Build:** Bazel and `make` are both supported; the parser is a separate Go module (`pkg/parser/go.mod`) replaced locally.

> [!NOTE]
> The dependency set is large because the tooling integrates with many clouds (AWS/GCS/Azure/Aliyun SDKs), observability stacks (Prometheus, OpenTelemetry, Jaeger, Pyroscope), and storage engines (`cockroachdb/pebble`, `pingcap/badger`). The core SQL path depends mainly on `tikv/client-go`, `tikv/pd/client`, and `etcd`.

---

## 12. Troubleshooting & Diagnostics

### Diagnostic-first workflow

1. **Health:** `curl :10080/status` on each node; check `information_schema.cluster_info`.
2. **Metrics:** Grafana dashboards (from `pkg/metrics/grafana`) or `/metrics`.
3. **Bundle:** `GET /debug/zip` collects config, schema, profiles, and recent logs into one archive for support.
4. **Slow queries:** `information_schema.slow_query` + the slow log.
5. **DDL stuck?** `ADMIN SHOW DDL JOBS`; check the owner with `ADMIN SHOW DDL`; inspect `mysql.tidb_ddl_job`.

### Common operational errors

### Error: `kernel type mismatch`

- **Cause:** `tidb-server`'s kernel type (classic/next-gen) disagrees with PD's.
- **Solution:** Run a build/config matching the cluster's kernel type. Next-gen requires `--keyspace-name`.

### Error: `invalid config: keyspace name or standby mode is required for nextgen tidb`

- **Cause:** Next-gen kernel started without a keyspace or standby mode.
- **Solution:** Pass `--keyspace-name <name>` (or run with `--standby`).

### Error: `value of [tmp-storage-quota] (...) exceeds the capacity of the [...] directory`

- **Cause:** Spill-storage quota exceeds free space on the temp volume.
- **Solution:** Lower `tmp-storage-quota`, move `tmp-storage-path`, or add disk.

### Error: `cannot set txn entry size limit larger than 120M`

- **Cause:** `txn-entry-size-limit` set above the 120 MiB hard cap.
- **Solution:** Keep it ≤ `120 MiB`; restructure very large single writes.

### Error: `BR:Common:ErrFailedToConnect` (`failed to make gRPC channels`)

- **Cause:** BR cannot reach PD/TiKV.
- **Solution:** Verify `--pd` reachability and TLS material.

### Error: `admin:8223 data inconsistency in table ...`

- **Cause:** Index/record mismatch detected by `ADMIN CHECK`.
- **Solution:** Freeze writes to the table, capture `/debug/zip`, engage support; do not blindly rebuild.

### Memory-usage alarm firing repeatedly

- **Cause:** Queries near `memory-usage-alarm-ratio` (0.8) of available memory.
- **Solution:** Set/lower `tidb_server_memory_limit`, enable spill-to-disk, identify the offending query via the alarm's profile dump, add resource-group limits.

---

## 13. Potential Improvements

Observations drawn from the codebase that operators and contributors may weigh:

- **Archived dependency cleanup.** `go.mod` carries a `replace` for `sourcegraph.com/sourcegraph/appdash` with a TODO noting the upstream is archived and should be removed.
- **Deprecated config keys persist.** Several options remain at the top level only for upgrade compatibility (`EnableBatchDML`, `MemQuotaQuery`, `OOMAction`, `OOMUseTmpStorage`, the moved-to-`[instance]` set). A future major version could drop them; operators should migrate now to avoid surprise.
- **Hidden but powerful subcommands.** `br debug` and `br operator` are marked hidden yet are load-bearing for support and tidb-operator. Clear external documentation of their contract would reduce reliance on reading source.
- **Large subsystems.** `pkg/expression`, `pkg/executor`, `pkg/ddl`, and `pkg/planner/core` are each multi-megabyte. The in-tree `docs/agents/*` navigation maps are an effective mitigation; extending that pattern to the executor and expression engines would help new contributors.
- **Default insecure bootstrap.** The default `--initialize-insecure=true` (empty root password) is convenient but a foot-gun if a test config reaches production. Consider defaulting deployment tooling to secure bootstrap.

> [!NOTE]
> These are framed from a single-snapshot reading of the source at `5e0c630`; the TiDB roadmap (`roadmap.md`) and design docs (`docs/design`) reflect the maintainers' own prioritized direction.

---

## 14. Quick Reference

### Ports

| Component | Port |
|-----------|------|
| TiDB SQL | `4000` |
| TiDB status/HTTP | `10080` |
| PD | `2379` / `2380` |
| TiKV | `20160` / `20180` |
| TiKV-Worker (next-gen) | `19000` |

### Essential config keys

```toml
host = "0.0.0.0"
port = 4000
store = "tikv"
path = "pd1:2379,pd2:2379,pd3:2379"
lease = "45s"                 # schema lease — do not change casually

[log]
level = "info"
slow-query-file = "/var/log/tidb/slow.log"

[security]
ssl-ca = "/etc/tidb/certs/ca.pem"
ssl-cert = "/etc/tidb/certs/tidb.pem"
ssl-key = "/etc/tidb/certs/tidb-key.pem"
cluster-ssl-ca = "/etc/tidb/certs/ca.pem"
cluster-ssl-cert = "/etc/tidb/certs/tidb.pem"
cluster-ssl-key = "/etc/tidb/certs/tidb-key.pem"

[status]
status-port = 10080
metrics-addr = "pushgateway:9091"

[performance]
server-memory-quota = 0       # 0 = use system memory
```

### Operational SQL

| Task | Statement |
|------|-----------|
| DDL owner & state | `ADMIN SHOW DDL;` |
| DDL jobs | `ADMIN SHOW DDL JOBS;` |
| Integrity check | `ADMIN CHECK TABLE t;` |
| Cluster nodes | `SELECT * FROM information_schema.cluster_info;` |
| Slow queries | `SELECT * FROM information_schema.slow_query ORDER BY query_time DESC;` |
| Deadlocks | `SELECT * FROM information_schema.deadlocks;` |
| TiFlash replica | `ALTER TABLE t SET TIFLASH REPLICA 1;` |

### Diagnostic endpoints

| URL (`:10080`) | Use |
|----------------|-----|
| `/status` | Health |
| `/info/all` | Cluster topology |
| `/config` | Effective config |
| `/ddl/history` | DDL job history |
| `/tables/{db}/{tbl}/regions` | Hotspot analysis |
| `/debug/zip` | Support bundle |

### Tool entry points

| Binary | Source |
|--------|--------|
| `tidb-server` | `cmd/tidb-server/main.go` |
| `br` | `br/cmd/br/main.go` |
| `tidb-lightning` | `lightning/cmd/tidb-lightning/main.go` |
| `dumpling` | `dumpling/cmd/dumpling/main.go` |

---

_Generated June 2026 from pingcap/tidb at 5e0c630. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._