# etcd — Administrator & Developer Guide

This guide is for the people who run etcd in production and the developers who extend or embed it: operators, SREs, platform architects, and contributors. It documents etcd's architecture, the complete server configuration surface, deployment and scaling strategies, day-2 operations, performance tuning, security hardening, and the internal module layout.

It complements the **etcd User Guide** (install + `etcdctl`/`etcdutl` usage). Where the User Guide shows you how to *use* a command, this guide explains how etcd *works* and how to operate it safely. All facts are drawn from the source at commit `7d330a1` (the v3.7 development line, `v3.7.0-beta.0`).

> [!NOTE]
> etcd is the consistent, distributed key-value store at the heart of many systems — most prominently as the backing store for Kubernetes. It is optimized for storing a relatively small amount of critical metadata with strong consistency and high availability, not for bulk data.

---

## 1. Architecture Overview

etcd is a replicated state machine. Clients issue requests to any member over a gRPC API; mutating requests are proposed to the **Raft** consensus module, which replicates them to a quorum of members and commits them in a total order. Each member then applies committed entries to its local **MVCC** key-value store, which is persisted in an embedded **bbolt** database. A **write-ahead log (WAL)** records every Raft entry to stable storage before it is applied, and periodic **snapshots** compact the log.

```
                          ┌──────────────────────────────────────────────┐
   etcdctl / clientv3 ───▶│  gRPC API (api/etcdserverpb)                   │
   (gRPC, port 2379)      │  KV · Watch · Lease · Cluster · Maintenance ·  │
                          │  Auth      [+ optional gRPC-gateway → HTTP]    │
                          └───────────────┬────────────────────────────────┘
                                          │
                                ┌─────────▼──────────┐
                                │   EtcdServer       │  server/etcdserver
                                │   (request router, │
                                │    apply loop)     │
                                └───┬───────────┬────┘
                  propose mutating │           │ apply committed
                                   ▼           ▼
                        ┌──────────────┐   ┌───────────────────────┐
   peer traffic ◀──────▶│  Raft node   │   │  MVCC store           │ server/storage/mvcc
   (port 2380, rafthttp)│ go.etcd.io/  │   │  (revisions, watch,   │
                        │   raft/v3    │   │   lease, auth)        │
                        └──────┬───────┘   └──────────┬────────────┘
                               │ persist             │ persist
                       ┌───────▼────────┐    ┌────────▼───────────┐
                       │  WAL + snapshot │    │  bbolt backend     │ server/storage/backend
                       │ server/storage/ │    │  (data-dir/*.db)   │ (go.etcd.io/bbolt)
                       │  wal, snap      │    └────────────────────┘
                       └─────────────────┘
```

### Component inventory

| Component | Path | Role |
|---|---|---|
| Entry point | `server/main.go` → `server/etcdmain` | Parses flags/config, dispatches to server, gateway, or grpc-proxy |
| Embeddable server | `server/embed` | `Config`, `Etcd`, `StartEtcd()` — the API to run etcd in-process |
| Core server | `server/etcdserver` | `EtcdServer`: request routing, the Raft apply loop, bootstrap, membership, corruption check |
| Consensus | `go.etcd.io/raft/v3` (external) | Leader election, log replication, membership changes, Pre-Vote |
| MVCC store | `server/storage/mvcc` | Multi-version key-value store, watchable store, revisions, compaction |
| Backend | `server/storage/backend` | bbolt wrapper: batched transactions, defrag, quota enforcement |
| WAL + snapshots | `server/storage/wal`, `server/etcdserver/api/snap` | Durable Raft log + periodic snapshots |
| Schema | `server/storage/schema` | On-disk schema, storage version, migrations |
| Lease subsystem | `server/lease` | TTL leases, keep-alive, checkpointing |
| Auth subsystem | `server/auth` | Users, roles, permissions, token (simple/jwt) management |
| Membership | `server/etcdserver/api/membership` | Cluster member registry, learners, cluster version |
| Peer transport | `server/etcdserver/api/rafthttp` | HTTP transport for Raft peer messages |
| gRPC API surface | `api/etcdserverpb` | Protobuf service/message definitions for the v3 API |
| Client library | `client/v3` (`clientv3`) | The official Go client used by `etcdctl` and applications |
| CLI client | `etcdctl` | Network client (see User Guide) |
| Offline tool | `etcdutl` | Operates on data-dir files and snapshots (see User Guide) |

### Technology stack

| Layer | Technology | Version (from `go.mod`) |
|---|---|---|
| Language | Go | toolchain `go1.26` |
| Consensus | `go.etcd.io/raft/v3` | `v3.7.0-rc.1` |
| Storage backend | `go.etcd.io/bbolt` | `v1.5.0-rc.0` |
| RPC | `google.golang.org/grpc` | `v1.81.0` |
| HTTP/JSON bridge | `grpc-ecosystem/grpc-gateway/v2` | `v2.29.0` |
| CLI framework | `spf13/cobra` / `spf13/pflag` | `v1.10.2` / `v1.0.10` |
| Logging | `go.uber.org/zap` (+ lumberjack rotation) | `v1.27.1` |
| Metrics | `prometheus/client_golang` | `v1.23.2` |
| Tracing | `go.opentelemetry.io/otel` (OTLP) | `v1.43.0` |
| Auth tokens | `golang-jwt/jwt/v5`, `golang.org/x/crypto` (bcrypt) | `v5.3.1` |

> [!NOTE]
> etcd is split into independently-versioned Go modules — `go.etcd.io/etcd/api/v3`, `client/v3`, `server/v3`, `etcdctl/v3`, `etcdutl/v3`, `pkg/v3`, and `cache/v3` — all currently at `v3.7.0-beta.0`. This lets applications depend on just the client (`client/v3`) without pulling in the server. The server module even uses `gomodguard` to forbid importing the umbrella `go.etcd.io/etcd` module, keeping the dependency graph clean.

### Data flow: a write

1. A client sends `Put` to any member over gRPC (port 2379).
2. `EtcdServer` checks the request against the backend **quota** (`server/storage/quota.go`); an oversized keyspace returns `etcdserver: mvcc: database space exceeded` and raises a `NOSPACE` alarm.
3. The request becomes an `InternalRaftRequest` and is **proposed** to the Raft node.
4. Raft replicates the entry to peers (port 2380 via `rafthttp`). Once a **quorum** persists it to their WALs, the entry is **committed**.
5. Each member's apply loop applies the committed entry to the **MVCC store**, creating a new **revision**. The write is persisted in the bbolt backend in a batched transaction.
6. The response (with the new revision in its header) returns to the client. Linearizable reads go through a `ReadIndex` confirmation so they observe all prior committed writes.

---

## 2. System Requirements

| Resource | Guidance |
|---|---|
| OS / arch | 64-bit Linux (primary target), macOS, Windows |
| CPU | 2–4 cores for small clusters; etcd is not CPU-bound but benefits from headroom during compaction/defrag |
| Memory | 2–8 GB typical. Memory scales with keyspace size, watch count, and history. The default DB quota is 2 GiB; do not exceed 8 GiB |
| **Disk** | **Dedicated SSD/NVMe.** etcd `fsync`s every Raft entry — disk write latency (`fdatasync`) is the single biggest performance factor |
| Network | Low, stable latency between members. Election timing assumes round-trips well under `--election-timeout` |
| File descriptors | Raise `LimitNOFILE` (e.g. 40000) under systemd — many client and peer connections |
| Cluster size | Odd number of voting members: 3 (tolerates 1 failure) or 5 (tolerates 2). More members slow writes |
| Time sync | Keep clocks loosely in sync (NTP); large skews affect lease/TTL reasoning and log timestamps |

> [!IMPORTANT]
> Put etcd's data directory on storage with consistently low write latency. A shared or network disk with variable `fsync` latency will cause leader elections, `etcdserver: request timed out` errors, and apply backpressure even when CPU and memory look healthy.

---

## 3. Configuration Reference

etcd is configured by command-line flags, by environment variables (`ETCD_` prefix, upper-cased, dashes→underscores), or by a single config file (`--config-file`). Flags and the config file map to the same `embed.Config` struct.

> [!IMPORTANT]
> If `--config-file` is provided, **all other command-line flags and environment variables are ignored.** Also note: duration fields in the JSON/YAML config file must be given as **integer nanoseconds** (e.g. `600000000000` for 10 minutes); human-readable strings like `"10m"` are only accepted on the command line. This is a Go standard-library limitation around `time.Duration` unmarshalling.

### 3.1 Member flags

| Flag | Default | Description |
|---|---|---|
| `--name` | `default` | Human-readable member name |
| `--data-dir` | `${name}.etcd` | Path to the data directory |
| `--wal-dir` | `` | Dedicated WAL directory (often a separate disk) |
| `--snapshot-count` | `10000` | Committed transactions between snapshots |
| `--snapshot-catchup-entries` | (server default) | Raft entries kept for a slow follower to catch up |
| `--heartbeat-interval` | `100` (ms) | Leader heartbeat interval |
| `--election-timeout` | `1000` (ms) | Election timeout; must be ≥ 5× heartbeat and ≤ 50000 |
| `--initial-election-tick-advance` | `true` | Fast-forward initial election ticks for faster first election |
| `--listen-peer-urls` | `http://localhost:2380` | URLs to listen on for peer (Raft) traffic |
| `--listen-client-urls` | `http://localhost:2379` | URLs to listen on for client gRPC+HTTP traffic |
| `--listen-client-http-urls` | `` | URLs for HTTP-only client traffic (splits HTTP off `--listen-client-urls`) |
| `--max-snapshots` | `5` | Max snapshot files to retain (0 = unlimited). Deprecated in v3.6, removed in v3.8 |
| `--max-wals` | `5` | Max WAL files to retain (0 = unlimited) |
| `--quota-backend-bytes` | `0` (→ 2 GiB) | Max backend DB size; exceeding it raises a `NOSPACE` alarm and read-only mode |
| `--backend-bbolt-freelist-type` | `map` | bbolt freelist type: `array` or `map` |
| `--backend-batch-interval` | `` | Max time before committing a backend transaction |
| `--backend-batch-limit` | `0` | Max operations before committing a backend transaction |
| `--max-txn-ops` | `128` | Max operations permitted in a single transaction |
| `--max-request-bytes` | `1572864` | Max client request size in bytes (~1.5 MiB) |
| `--max-concurrent-streams` | `math.MaxUint32` | Max concurrent gRPC streams per client |
| `--grpc-keepalive-min-time` | `5s` | Minimum interval a client must wait before pinging the server |
| `--grpc-keepalive-interval` | `2h` | Server→client keepalive ping frequency (0 disables) |
| `--grpc-keepalive-timeout` | `20s` | Wait before closing a non-responsive connection (0 disables) |
| `--socket-reuse-port` | `false` | Set `SO_REUSEPORT` on listeners |
| `--socket-reuse-address` | `false` | Set `SO_REUSEADDR` on listeners |
| `--enable-grpc-gateway` | `true` | Expose the gRPC API as JSON/HTTP via grpc-gateway |
| `--raft-read-timeout` | (rafthttp default) | Read timeout on each peer connection |
| `--raft-write-timeout` | (rafthttp default) | Write timeout on each peer connection |
| `--memory-mlock` | `false` | mlock etcd's pages (esp. bbolt) into RAM to avoid swap-induced latency |
| `--feature-gates` | `` | `key=value` pairs enabling alpha/beta features (see §3.9) |

### 3.2 Clustering flags

| Flag | Default | Description |
|---|---|---|
| `--initial-advertise-peer-urls` | `http://localhost:2380` | This member's peer URLs advertised to the cluster |
| `--advertise-client-urls` | `http://localhost:2379` | This member's client URLs advertised to clients |
| `--initial-cluster` | `default=http://localhost:2380` | Static bootstrap cluster map (`name=peerURL,...`) |
| `--initial-cluster-state` | `new` | `new` to bootstrap, `existing` to join. Ignored after successful init |
| `--initial-cluster-token` | `etcd-cluster` | Bootstrap token; isolates co-located clusters |
| `--strict-reconfig-check` | `true` | Reject membership changes that would lose quorum |
| `--pre-vote` | `true` | Raft Pre-Vote: prevents a rejoining partitioned node from disrupting the cluster |
| `--max-learners` | `1` | Max non-voting learner members allowed |
| `--discovery-srv` | `` | DNS SRV domain for cluster bootstrap discovery |
| `--discovery-srv-name` | `` | Suffix for the queried SRV name |
| `--discovery-token`, `--discovery-endpoints` | `` | v3 discovery: token + gRPC endpoints of a discovery service |
| `--discovery-dial-timeout` | `2s` | v3 discovery dial timeout |
| `--discovery-request-timeout` | `5s` | v3 discovery request timeout |
| `--discovery-keepalive-time` / `--discovery-keepalive-timeout` | `2s` / `6s` | v3 discovery keepalive |
| `--discovery-insecure-transport` | `true` | v3 discovery: disable transport security |
| `--discovery-insecure-skip-tls-verify` | `false` | v3 discovery: skip cert verification (testing only) |
| `--discovery-cert` / `--discovery-key` / `--discovery-cacert` | `` | v3 discovery TLS material |
| `--discovery-user` / `--discovery-password` | `` | v3 discovery auth |
| `--v2-deprecation` | `write-only` (enforced) | Phase of v2store deprecation; value is enforced and slated for removal in v3.8 |

> [!WARNING]
> The legacy v2 `--discovery` URL bootstrap and `--discovery-proxy` are deprecated (planned removal in v3.8). Use static `--initial-cluster`, DNS SRV (`--discovery-srv`), or the v3 discovery service for new clusters.

### 3.3 Security (TLS) flags

| Flag | Default | Description |
|---|---|---|
| `--cert-file` / `--key-file` | `` | Client-facing server TLS certificate and key |
| `--client-cert-auth` | `false` | Require and verify client certificates (mTLS) |
| `--client-cert-file` / `--client-key-file` | `` | Explicit client cert/key when client auth is required |
| `--client-crl-file` | `` | Client certificate revocation list |
| `--client-cert-allowed-hostname` | `` | Allowed SAN hostnames for client cert auth |
| `--trusted-ca-file` | `` | CA bundle to verify client certificates |
| `--auto-tls` | `false` | Generate self-signed client certs automatically |
| `--peer-cert-file` / `--peer-key-file` | `` | Peer (inter-member) TLS certificate and key |
| `--peer-client-cert-auth` | `false` | Require and verify peer client certificates |
| `--peer-client-cert-file` / `--peer-client-key-file` | `` | Explicit peer client cert/key |
| `--peer-trusted-ca-file` | `` | CA bundle to verify peer certificates |
| `--peer-auto-tls` | `false` | Generate self-signed peer certs automatically |
| `--peer-cert-allowed-cn` | `` | Allowed CNs for inter-peer auth |
| `--peer-cert-allowed-hostname` | `` | Allowed SAN hostnames for inter-peer auth |
| `--peer-crl-file` | `` | Peer certificate revocation list |
| `--peer-skip-client-san-verification` | `false` | Skip SAN verification for peer client certs |
| `--self-signed-cert-validity` | `1` | Validity (years) of auto-generated certs |
| `--cipher-suites` | `` | Allowed TLS cipher suites (empty = Go default) |
| `--tls-min-version` | `TLS1.2` | Minimum TLS version (`TLS1.2`, `TLS1.3`) |
| `--tls-max-version` | `` | Maximum TLS version (empty = Go default) |
| `--cors` | `*` | CORS origin allowlist |
| `--host-whitelist` | `*` | Acceptable HTTP `Host` values when the server is insecure (anti-DNS-rebinding) |

### 3.4 Auth flags

| Flag | Default | Description |
|---|---|---|
| `--auth-token` | `simple` | Token type/options: `simple` or `jwt` |
| `--bcrypt-cost` | bcrypt default (10) | Cost factor for password hashing (between bcrypt min/max) |
| `--auth-token-ttl` | `300` | Lifetime (seconds) of the simple auth token |

### 3.5 Profiling & monitoring flags

| Flag | Default | Description |
|---|---|---|
| `--enable-pprof` | `false` | Expose Go pprof at client-URL `/debug/pprof/` |
| `--metrics` | `basic` | Metric detail; `extensive` adds server-side gRPC histograms |
| `--listen-metrics-urls` | `` | Extra URLs serving only `/metrics` and `/health` |

### 3.6 Logging flags

| Flag | Default | Description |
|---|---|---|
| `--logger` | `zap` | Only `zap` is supported (`capnslog` removed in v3.5) |
| `--log-outputs` | `default` | `stdout`, `stderr`, or comma-separated file targets; `default` uses journald under systemd |
| `--log-level` | `info` | `debug`, `info`, `warn`, `error`, `panic`, `fatal` |
| `--log-format` | `json` | `json` or `console` |
| `--enable-log-rotation` | `false` | Rotate a single file log target |
| `--log-rotation-config-json` | `{"maxsize":100,"maxage":0,"maxbackups":0,"localtime":false,"compress":false}` | lumberjack rotation config |

### 3.7 Distributed tracing flags

| Flag | Default | Description |
|---|---|---|
| `--enable-distributed-tracing` | `false` | Enable OpenTelemetry tracing |
| `--distributed-tracing-address` | `localhost:4317` | OTLP collector address |
| `--distributed-tracing-service-name` | `etcd` | Service name (keep identical across instances) |
| `--distributed-tracing-instance-id` | `` | Unique per-instance ID |
| `--distributed-tracing-sampling-rate` | `0` | Samples collected per million spans |

### 3.8 Feature, maintenance & unsafe flags

| Flag | Default | Description |
|---|---|---|
| `--auto-compaction-mode` | `periodic` | `periodic` (duration-based) or `revision` (revision-count-based) |
| `--auto-compaction-retention` | `0` | Retention window/count; `0` disables auto-compaction |
| `--corrupt-check-time` | `0s` | Interval between cluster corruption checks (0 disables) |
| `--compact-hash-check-time` | `1m` | Interval the leader checks followers' compaction hashes |
| `--compaction-batch-limit` | `1000` | Max revisions deleted per compaction batch |
| `--compaction-sleep-interval` | (server default) | Sleep between compaction batches |
| `--watch-progress-notify-interval` | `10m` | Periodic watch progress notification interval |
| `--warning-apply-duration` | `100ms` | Warn if applying a request exceeds this |
| `--warning-unary-request-duration` | `300ms` | Warn if a unary request exceeds this |
| `--bootstrap-defrag-threshold-megabytes` | `0` | Defrag on boot if it would free at least this many MB (0 disables) |
| `--downgrade-check-time` | `5s` | Interval between downgrade status checks |
| `--force-new-cluster` | `false` | **Unsafe.** Force a new single-member cluster from existing data |
| `--unsafe-no-fsync` | `false` | **Unsafe.** Disable fsync — *will cause data loss* |

> [!CAUTION]
> `--force-new-cluster` and `--unsafe-no-fsync` break the guarantees of the consensus protocol. `--force-new-cluster` is a disaster-recovery escape hatch to recover a quorum-loss situation from one surviving member; `--unsafe-no-fsync` is only for throwaway benchmarking. Never run either in normal production.

### 3.9 Feature gates

`--feature-gates` takes `Feature=true/false` pairs. The gates defined in this version:

| Feature gate | Default | Stage | Effect |
|---|---|---|---|
| `TxnModeWriteWithSharedBuffer` | `true` | Beta | Write txns use a shared buffer for read-only checks |
| `FastLeaseKeepAlive` | `true` | Beta | Lease renewal skips waiting for the applied index |
| `StopGRPCServiceOnDefrag` | `false` | Alpha | Stop serving client requests during defragmentation |
| `InitialCorruptCheck` | `false` | Alpha | Verify data integrity before serving any traffic |
| `CompactHashCheck` | `false` | Alpha | Leader periodically checks followers' compaction hashes |
| `LeaseCheckpoint` | `false` | Alpha | Leader sends regular lease checkpoints to prevent TTL reset on leader change |
| `LeaseCheckpointPersist` | `false` | Alpha | Persist remaining TTL (deprecated; always on in v3.6) |
| `SetMemberLocalAddr` | `false` | Alpha | Use a non-loopback local address from initial-advertise-peer-urls when dialing peers |
| `PriorityRequest` | `false` | Alpha | Give certain requests (e.g. `LeaseRevoke`) higher priority under overload |

> [!NOTE]
> `LeaseCheckpointPersist` requires `LeaseCheckpoint` to also be enabled — etcd refuses to start otherwise (`enabling feature gate LeaseCheckpointPersist requires enabling feature gate LeaseCheckpoint`).

### 3.10 YAML configuration file

The config file uses the JSON keys from `embed.Config` (e.g. `name`, `data-dir`, `initial-cluster`). TLS is nested under `client-transport-security` / `peer-transport-security`:

```yaml
name: infra1
data-dir: /var/lib/etcd
listen-client-urls: https://10.0.0.10:2379
advertise-client-urls: https://10.0.0.10:2379
listen-peer-urls: https://10.0.0.10:2380
initial-advertise-peer-urls: https://10.0.0.10:2380
initial-cluster: infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380
initial-cluster-token: prod-etcd
initial-cluster-state: new
client-transport-security:
  cert-file: /etc/etcd/server.pem
  key-file: /etc/etcd/server-key.pem
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ca.pem
peer-transport-security:
  cert-file: /etc/etcd/peer.pem
  key-file: /etc/etcd/peer-key.pem
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ca.pem
auto-compaction-mode: periodic
auto-compaction-retention: "1h"
quota-backend-bytes: 8589934592
```

Run with: `etcd --config-file /etc/etcd/etcd.conf.yaml`.

---

## 4. Deployment & Scaling

### 4.1 Static bootstrap (recommended)

Every member is told the full membership up front. Start all members with `--initial-cluster-state new`:

```bash
etcd --name infra1 \
  --data-dir /var/lib/etcd \
  --listen-client-urls https://10.0.0.10:2379 \
  --advertise-client-urls https://10.0.0.10:2379 \
  --listen-peer-urls https://10.0.0.10:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-cluster-token prod-etcd \
  --initial-cluster-state new
```

Deployment checklist:

- [ ] Odd number of members (3 or 5), spread across failure domains
- [ ] Unique `--name`, `--initial-advertise-peer-urls`, `--advertise-client-urls` per member
- [ ] Identical `--initial-cluster` and `--initial-cluster-token` on every member
- [ ] Dedicated fast disk for `--data-dir`
- [ ] TLS configured for both client and peer traffic
- [ ] `--quota-backend-bytes` and auto-compaction set
- [ ] `LimitNOFILE` raised in the unit file

### 4.2 DNS SRV discovery

Instead of a static list, point members at a DNS domain holding `_etcd-server-ssl._tcp` (peer) and `_etcd-client-ssl._tcp` (client) SRV records:

```bash
etcd --name infra1 --discovery-srv example.com \
  --initial-advertise-peer-urls https://infra1.example.com:2380 \
  --advertise-client-urls https://infra1.example.com:2379 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --initial-cluster-token prod-etcd \
  --initial-cluster-state new
```

### 4.3 systemd (multi-node)

A production unit file (adapted from the project's `contrib/systemd/etcd3-multinode` example):

```ini
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
ExecStart=/usr/local/bin/etcd --name %H \
    --data-dir /var/lib/etcd \
    --listen-client-urls https://${IP}:2379 \
    --advertise-client-urls https://${IP}:2379 \
    --listen-peer-urls https://${IP}:2380 \
    --initial-advertise-peer-urls https://${IP}:2380 \
    --initial-cluster infra1=https://${IP_1}:2380,infra2=https://${IP_2}:2380,infra3=https://${IP_3}:2380 \
    --initial-cluster-token prod-etcd \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now etcd.service
sudo journalctl -u etcd.service -f
```

Prepare the data directory first:

```bash
sudo mkdir -p /var/lib/etcd
sudo chown -R etcd:etcd /var/lib/etcd
```

### 4.4 Scaling: adding/removing members safely

etcd does not scale horizontally for throughput — adding members **increases** write latency (every write needs quorum). Add members for **fault tolerance**, not performance. The safe sequence uses learners:

1. `etcdctl member add <name> --peer-urls=<peerURL>` (or `member add --learner` where supported) — for a learner, it replicates but doesn't vote.
2. Start the new member with the printed `ETCD_*` variables and `--initial-cluster-state existing`.
3. Wait until `etcdctl endpoint status` shows the new member caught up (matching raft index).
4. Promote it to a voter (`etcdctl member promote <id>`) only when in sync — otherwise you get `can only promote a learner member which is in sync with the leader`.
5. To remove: `etcdctl member remove <id>` **after** confirming the cluster still has quorum without it.

> [!WARNING]
> `--strict-reconfig-check` (default true) rejects reconfigurations that would cause quorum loss. Leave it on. Change membership one member at a time and verify health between steps.

---

## 5. Operations

### 5.1 Start / stop / health

```bash
sudo systemctl start etcd
sudo systemctl stop etcd
etcdctl endpoint --cluster health
etcdctl -w table endpoint --cluster status
```

### 5.2 Backup and disaster recovery

**Back up** from a live cluster (writes an integrity hash):

```bash
etcdctl snapshot save /backups/etcd-$(date +%F).db
etcdutl snapshot status /backups/etcd-$(date +%F).db -w table
```

**Restore** a new cluster from one snapshot — run `etcdutl snapshot restore` once per member, each with its own name/peer URLs but the same `--initial-cluster`, then start all with `--initial-cluster-state new`:

```bash
etcdutl snapshot restore /backups/etcd.db \
  --name infra1 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --data-dir /var/lib/etcd-new
```

> [!CAUTION]
> Recovering a cluster from a single surviving member after permanent quorum loss uses `etcd --force-new-cluster` (it rewrites membership to a one-member cluster), then you re-add members. This is destructive and discards uncommitted state — take a snapshot first and only do it when quorum is truly unrecoverable.

- [ ] Snapshots scheduled (cron/systemd timer)
- [ ] Snapshots stored off-box
- [ ] Restore procedure tested in a sandbox
- [ ] `snapshot status` verified after each backup

### 5.3 Compaction and defragmentation

History (MVCC revisions) grows forever until **compacted**; the DB file does not shrink until **defragmented**. Operate both:

```bash
# automatic compaction (set on the server; preferred)
etcd ... --auto-compaction-mode=periodic --auto-compaction-retention=1h

# manual compaction to a recent revision
etcdctl compaction "$(etcdctl endpoint status -w json | grep -o '\"revision\":[0-9]*' | head -1 | cut -d: -f2)"

# reclaim disk (one member at a time; blocks that member)
etcdctl defrag --cluster
```

> [!NOTE]
> Compaction and defrag are independent. Compaction removes old revisions inside the database; defrag returns the freed pages to the filesystem. Run compaction frequently (auto), defrag occasionally and never on all members simultaneously.

### 5.4 Alarms

```bash
etcdctl alarm list      # e.g. alarm:NOSPACE  or  alarm:CORRUPT
etcdctl alarm disarm
```

A `NOSPACE` alarm means the backend exceeded its quota and etcd is read-only. Recovery: compact → defrag → disarm. A `CORRUPT` alarm (raised by the corruption checker) means a member's data diverged — investigate before disarming.

### 5.5 Monitoring (Prometheus)

etcd exposes Prometheus metrics on the client URLs (or `--listen-metrics-urls`). Key series to alert on:

| Metric | Why it matters |
|---|---|
| `etcd_server_has_leader` | 0 means the member has no leader — no writes |
| `etcd_server_leader_changes_seen_total` | Frequent changes → instability (often slow disk) |
| `etcd_disk_wal_fsync_duration_seconds` | WAL fsync latency — the canary for disk health |
| `etcd_disk_backend_commit_duration_seconds` | Backend commit latency |
| `etcd_mvcc_db_total_size_in_bytes` | DB size vs. `--quota-backend-bytes` |
| `etcd_server_proposals_failed_total` | Failed Raft proposals |
| `etcd_network_peer_round_trip_time_seconds` | Peer link latency |

Set `--metrics=extensive` to also export per-method gRPC histograms. The project ships Grafana dashboards and Prometheus alert rules in `contrib/mixin`.

### 5.6 Upgrades and downgrades

etcd supports rolling upgrades one minor version at a time; members agree on a **cluster version** (the lowest member minor version), and new members refuse to join a cluster newer than themselves. Downgrade is the reverse and is **experimental** (v3.6+):

```bash
etcdctl downgrade validate 3.6
etcdctl downgrade enable 3.6
# replace each member's binary with the older version, one at a time
# watch the leader log for "the cluster has been downgraded"
```

`etcdutl migrate --data-dir <dir> --target-version X.Y` migrates an **offline** data directory's schema (minimum target 3.5).

> [!IMPORTANT]
> Always take a snapshot before an upgrade or downgrade. For downgrade, confirm each member logs `The server is ready to downgrade` before swapping its binary, and use `etcdctl downgrade cancel` to abort if anything looks wrong.

---

## 6. Performance Tuning

| Area | Lever | Guidance |
|---|---|---|
| Disk | dedicated NVMe; `--wal-dir` on its own device | The #1 factor. Watch `etcd_disk_wal_fsync_duration_seconds` (target p99 < 10ms) |
| Raft timing | `--heartbeat-interval`, `--election-timeout` | Set heartbeat ≈ RTT between members; election timeout ≥ 5× heartbeat. Higher for cross-DC links |
| Snapshots | `--snapshot-count`, `--snapshot-catchup-entries` | More frequent snapshots trim the WAL but cost I/O; keep enough catch-up entries for slow followers |
| Keyspace | `--auto-compaction-*`, periodic `defrag` | Keep history bounded; defrag to reclaim space |
| Quota | `--quota-backend-bytes` | Default 2 GiB; up to 8 GiB suggested max. Larger quotas degrade performance |
| Backend batching | `--backend-batch-interval`, `--backend-batch-limit` | Larger batches raise throughput at the cost of latency |
| Memory | `--memory-mlock` | Pin pages in RAM where swap would otherwise add tail latency |
| Network | `--max-request-bytes`, `--max-concurrent-streams` | Keep requests small; cap streams to bound resource use |
| Reads | serializable reads (`--consistency=s` client-side) | Offload read-heavy, latency-sensitive workloads from the quorum path |

> [!TIP]
> Use the built-in `etcdctl check perf` against your real hardware to find the workload tier your cluster sustains, and `etcdctl check datascale` to estimate memory for a given keyspace before you grow it.

> [!WARNING]
> Raising `--quota-backend-bytes` above 8 GiB triggers a warning (`quota exceeds the maximum value`) because large databases increase boot time, snapshot size, defrag duration, and memory use. Prefer aggressive compaction over a huge quota.

---

## 7. Security

### 7.1 Transport security (TLS)

etcd has two independent TLS domains: **client↔server** and **peer↔peer**. Configure both for production. Mutual TLS is enabled with `--client-cert-auth` / `--peer-client-cert-auth` plus a `--trusted-ca-file` / `--peer-trusted-ca-file`.

```bash
etcd --name infra1 ... \
  --cert-file=/etc/etcd/server.pem --key-file=/etc/etcd/server-key.pem \
  --client-cert-auth --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-cert-file=/etc/etcd/peer.pem --peer-key-file=/etc/etcd/peer-key.pem \
  --peer-client-cert-auth --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --tls-min-version=TLS1.2
```

`--auto-tls` / `--peer-auto-tls` generate self-signed certificates for quick setups (validity set by `--self-signed-cert-validity`, default 1 year). Restrict identities with `--client-cert-allowed-hostname`, `--peer-cert-allowed-cn`, and `--peer-cert-allowed-hostname`.

> [!CAUTION]
> When TLS is **not** configured, etcd serves clients over plain HTTP and logs `serving client traffic insecurely; this is strongly discouraged!`. An insecure server is also exposed to DNS-rebinding attacks (CVE-2018-5702), which is why `--host-whitelist` defaults to `*` but should be restricted to known hostnames for any insecure HTTP listener.

### 7.2 Authentication & RBAC

Once `auth enable` is run, every request must authenticate. The model is users → roles → per-key/prefix permissions (`read`, `write`, `readwrite`). A `root` user with the `root` role is mandatory before enabling. Token type is set by `--auth-token`:

- **`simple`** (default) — in-memory tokens with TTL `--auth-token-ttl` (300s). Simple, but tokens don't survive restarts and don't work well behind L4 proxies.
- **`jwt`** — signed JWT tokens; configure signing key/method via the `--auth-token` options string. Stateless and proxy-friendly.

Passwords are hashed with **bcrypt** at cost `--bcrypt-cost`. See the User Guide §4.10 for the full command sequence.

> [!WARNING]
> Authentication is an authorization layer on top of, not a replacement for, TLS. Without client TLS, credentials and data still cross the wire in cleartext. Always combine RBAC with TLS. There is no root-password recovery — losing it means restoring from a snapshot.

### 7.3 Data integrity

etcd can detect silent data divergence between members:

- `--corrupt-check-time` runs periodic cluster corruption checks; a mismatch raises a `CORRUPT` alarm and logs `found data inconsistency with peers`.
- The `InitialCorruptCheck` feature gate verifies integrity **before** serving any traffic at startup.
- The `CompactHashCheck` feature gate (with `--compact-hash-check-time`, default 1m) has the leader compare followers' compaction hashes.
- `etcdctl endpoint hashkv` and `etcdutl hashkv` compute the KV history hash for manual comparison.

### 7.4 Operational security checklist

- [ ] Client and peer TLS enabled, with mTLS (`--client-cert-auth`, `--peer-client-cert-auth`)
- [ ] `--host-whitelist` restricted if any insecure HTTP listener exists
- [ ] Authentication enabled with least-privilege roles scoped by prefix
- [ ] `--auth-token=jwt` behind proxies
- [ ] Corruption checking enabled (`--corrupt-check-time`, `InitialCorruptCheck`)
- [ ] `--enable-pprof` left **off** in production (it exposes profiling data)
- [ ] Peer and client ports firewalled to known hosts
- [ ] Dependencies kept patched (the project tracks CVE bumps, e.g. `golang.org/x/crypto` — issues #21896, #21270)

---

## 8. Developer Internals

### 8.1 Repository structure

| Path | Contents |
|---|---|
| `api/` | Protobuf API definitions: `etcdserverpb` (the v3 service), `mvccpb` (KV/Event), `authpb`, `membershippb`, `versionpb` |
| `client/v3/` | The official Go client (`clientv3`); concurrency helpers, namespacing, ordering, mirror, leasing live in subpackages |
| `client/pkg/` | Shared client utilities: transport/TLS, file utils, types, logging |
| `server/embed/` | Embeddable server: `Config`, `Etcd`, `StartEtcd()` |
| `server/etcdmain/` | The `etcd` binary: flag/config parsing, gateway and grpc-proxy subcommands, usage/help |
| `server/etcdserver/` | `EtcdServer` core: apply loop, bootstrap, raft glue, membership, corruption check, apply/, txn/, version/ |
| `server/storage/` | `backend` (bbolt), `mvcc`, `wal`, `schema`, `datadir`, `quota`, snapshot recovery |
| `server/lease/` | Lease subsystem (`lessor`), lease persistence (`leasepb`), `leasehttp` |
| `server/auth/` | RBAC store, simple-token and JWT providers |
| `server/proxy/` | gRPC L7 proxy and TCP pass-through proxy |
| `etcdctl/`, `etcdutl/` | The two CLI tools |
| `pkg/` | Cross-cutting libraries: `flags`, `featuregate`, `adt` (interval tree), `schedule`, `wait`, `traceutil`, `netutil`, etc. |
| `tools/` | `benchmark`, `etcd-dump-db`, `etcd-dump-logs`, `etcd-dump-metrics`, `rw-heatmaps`, and others |
| `tests/` | `integration`, `e2e`, `robustness`, `antithesis`, and shared `framework` |
| `contrib/` | `systemd` units, `mixin` (Grafana/Prometheus), `raftexample`, lock example |
| `Documentation/` | Contributor and internals docs |

### 8.2 Architecture deep dive

**Request path.** `EtcdServer` (`server/etcdserver/server.go`) is the hub. Read requests with linearizable consistency go through a `ReadIndex` round trip; serializable reads are served locally. Mutating requests are marshalled into an `InternalRaftRequest` and proposed to the embedded Raft node. The apply loop consumes committed entries and dispatches them to the appropriate subsystem (`server/etcdserver/apply`, `txn/`).

**Consensus.** Raft (`go.etcd.io/raft/v3`) handles leader election, log replication, and membership changes (`ConfChange` entries for add/remove/learner). Pre-Vote (`--pre-vote`) stops a node that was partitioned away from incrementing terms and disrupting the cluster when it returns. Membership entries are reconstructed from the snapshot + WAL via `GetEffectiveNodeIDsFromWALEntries` (`server/storage/util.go`).

**Durability.** Every Raft entry is written to the **WAL** (`server/storage/wal`) and `fsync`ed before being applied — this is the durability guarantee and the dominant latency cost. The `Storage` interface (`server/storage/storage.go`) wraps WAL + snapshotter: `Save` persists entries, `SaveSnap` writes a snapshot file *before* the WAL snapshot record (so a snapshot file may be orphaned but a WAL record never lacks its file), and `Release` frees old WAL locks and `.snap.db` files.

**State machine.** Committed entries are applied to the **MVCC store** (`server/storage/mvcc`). Every change bumps a global **revision**; each key tracks create/mod revisions and a version. The watchable store streams events to watchers and supports replay from any uncompacted revision. The store is persisted in the **bbolt backend** (`server/storage/backend`), where keys live in buckets (key, lease, auth, meta, ...). A `consistentIndex` is stored in the backend so recovery knows which entries are already applied (`RecoverSnapshotBackend` replaces the DB with a leader-sent snapshot when `snapshot.Index > consistentIndex`).

**Quota & alarms.** `server/storage/quota.go` charges each mutating request a cost (`kvOverhead=256` per key, `leaseOverhead=64` per lease, plus key+value bytes) and rejects writes once `backend.Size()` would exceed the limit, raising a `NOSPACE` alarm. The default quota is 2 GiB (`DefaultQuotaBytes`), and the backend mmap is sized at quota + 10% to leave room to disarm.

**Leases.** `server/lease` tracks TTL leases and the keys attached to them, expiring keys when leases lapse. The `LeaseCheckpoint` feature persists remaining TTL so a leader change doesn't reset long-lived leases; `FastLeaseKeepAlive` (beta, default on) speeds renewal by skipping the applied-index wait.

**Schema & versioning.** `server/storage/schema` records the on-disk **storage version** and runs migrations; `DetectSchemaVersion`/`Migrate` drive `etcdutl migrate`. The cluster version (lowest member minor version) is negotiated at runtime and gates joins/downgrades.

### 8.3 The v3 gRPC API surface

Defined in `api/etcdserverpb/rpc.proto`. Six services:

| Service | RPCs |
|---|---|
| **KV** | `Range`, `RangeStream`, `Put`, `DeleteRange`, `Txn`, `Compact` |
| **Watch** | `Watch` (bidirectional stream) |
| **Lease** | `LeaseGrant`, `LeaseRevoke`, `LeaseKeepAlive` (stream), `LeaseTimeToLive`, `LeaseLeases` |
| **Cluster** | `MemberAdd`, `MemberRemove`, `MemberUpdate`, `MemberList`, `MemberPromote` |
| **Maintenance** | `Alarm`, `Status`, `Defragment`, `Hash`, `HashKV`, `Snapshot` (stream), `MoveLeader`, `Downgrade` |
| **Auth** | `AuthEnable`, `AuthDisable`, `AuthStatus`, `Authenticate`, `UserAdd/Get/List/Delete/ChangePassword/GrantRole/RevokeRole`, `RoleAdd/Get/List/Delete/GrantPermission/RevokePermission` |

When `--enable-grpc-gateway=true` (default), the grpc-gateway exposes these as JSON/HTTP.

### 8.4 Code patterns and conventions

- **Modular Go modules** with `replace` directives during development and `gomodguard` rules forbidding the umbrella module — keeps `client/v3` lightweight for downstream users.
- **Protobuf everywhere** — the wire API, internal Raft requests, lease/auth/membership persistence, and WAL records are all protobuf.
- **`gofail` injection points** — comments like `// gofail: var raftBeforeWALSaveSnaphot struct{}` mark failpoints the robustness tests use to inject faults.
- **Structured logging** via zap, with `--warning-apply-duration` / `--warning-unary-request-duration` emitting slow-path warnings.
- **Feature gates** (`pkg/featuregate`) gate alpha/beta behavior behind `--feature-gates`.
- **Cobra command trees** for all CLIs with generated shell completion.

### 8.5 Testing strategy

etcd carries an unusually deep test suite, reflecting its role as critical infrastructure:

- **`tests/integration`** — in-process multi-member clusters exercising the client and server APIs.
- **`tests/e2e`** — spawn real `etcd`/`etcdctl` binaries (via `pkg/expect`).
- **`tests/robustness`** — linearizability checking with `porcupine`, driving failpoints (`gofail`) and validating history against a formal model.
- **`tests/antithesis`** — deterministic fault-injection testing.
- **`tests/framework`** — a shared abstraction so the same test runs under integration and e2e.

### 8.6 Bundled tools

| Tool | Purpose |
|---|---|
| `tools/benchmark` | Official load generator (`put`, `range`, `txn-put`, `watch`, `stm`, `lease-keepalive`, ...) |
| `tools/etcd-dump-db` | Inspect a bbolt database file (`list-bucket`, `iterate-bucket`, `scan-keys`, `hash`) |
| `tools/etcd-dump-logs` | Decode WAL log entries |
| `tools/etcd-dump-metrics` | Dump a binary's exported metrics |
| `tools/rw-heatmaps` | Render read/write performance heatmaps from CSV |
| `server/etcdmain gateway` | Stateless TCP pass-through proxy |
| `server/etcdmain grpc-proxy` | Stateless L7 gRPC reverse proxy (connection coalescing, caching) |

### 8.7 Potential improvements & known issues

From the project's own issue tracker at this commit:

- **Memory usage with 3.6 event reuse** (#21355) — increased memory reported under event reuse; relevant when sizing nodes.
- **Promote member with auth enabled** (#20757) — promoting a follower can fail when auth is enabled.
- **Dependency hygiene** (#21270, #21896, #21885) — archived-dependency cleanup and CVE-driven bumps (`golang.org/x/crypto`, Go toolchain).
- **Exit-code classification** (#21032) — an effort to classify all etcdserver exit codes.
- **v3.8 performance ideas** (#19806) — ongoing performance work.
- **Deprecations to track:** v2 store/discovery (`--v2-deprecation`, `--discovery`), `--max-snapshots`, and the `LeaseCheckpointPersist` gate are all on removal paths (v3.7/v3.8).

### 8.8 Contributing

The project is Go-module-based with build/test scripts under `scripts/` and `./scripts/build.sh` producing the three binaries. Contributor documentation lives in `Documentation/contributor-guide`. Protobuf changes are validated by `tools/proto-annotations` (which enforces `etcd_version` annotations on API fields). The robustness and e2e suites gate behavioral changes.

---

## 9. Integration Points

| Integration | How |
|---|---|
| **Go applications** | Import `go.etcd.io/etcd/client/v3` (`clientv3`). Concurrency primitives (`concurrency` package: sessions, mutexes, elections), namespacing, and mirroring are subpackages |
| **HTTP/JSON clients** | The grpc-gateway (`--enable-grpc-gateway`) exposes the v3 API as REST/JSON |
| **Kubernetes** | etcd is Kubernetes' backing store; `client/v3/kubernetes` provides a tailored access path |
| **Prometheus** | Scrape `/metrics` on client URLs or `--listen-metrics-urls`; `--metrics=extensive` for gRPC histograms; dashboards/alerts in `contrib/mixin` |
| **OpenTelemetry** | `--enable-distributed-tracing` exports OTLP traces to `--distributed-tracing-address` |
| **systemd** | `Type=notify` units; example in `contrib/systemd` |
| **Service discovery / coordination** | Leases + watches power service registries; `concurrency` package and `etcdctl lock`/`elect` power distributed locks and leader election |
| **Proxies** | The bundled `etcd gateway` (L4) and `etcd grpc-proxy` (L7) front a cluster |

> [!NOTE]
> For application developers, prefer the `clientv3` library over shelling out to `etcdctl`. The library handles endpoint balancing, retries, and watch reconnection; `etcdctl` is a thin wrapper over the same library intended for humans and scripts.

---

_Generated June 2026 from etcd-io/etcd at 7d330a1. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._