# etcd — User Guide

etcd is a distributed, reliable key-value store for the most critical data of a distributed system. It keeps a small amount of data consistent and highly available by replicating it across a cluster of machines using the Raft consensus algorithm. If a majority of the cluster is alive, etcd keeps serving — even through machine failures, including leader failures.

This guide is for the people who *use* etcd: application developers storing configuration and coordination data, operators running `etcdctl` against a live cluster, and anyone who needs to install etcd, store and watch keys, manage leases, take backups, or set up authentication. It covers the `etcd` server binary, the `etcdctl` client, and the `etcdutl` offline maintenance tool as they behave in the source at commit `7d330a1` (the v3.7 development line).

> [!NOTE]
> etcd speaks a **gRPC API (the v3 API)**. `etcdctl` uses the v3 API by default. The old v2 key-value API and its `etcdctl` v2 mode are deprecated and on a removal path; everything in this guide is v3.

---

## 1. Quick Start

etcd ships as three self-contained binaries:

- **`etcd`** — the server. Run one for local development, or three/five for a fault-tolerant cluster.
- **`etcdctl`** — the command-line client for talking to a running cluster (put/get keys, watch, leases, membership, auth, backups).
- **`etcdutl`** — an offline administration tool that operates directly on data-dir files and snapshots (defrag, restore, migrate, hash) without a running server.

### Key features

- **Strong consistency** — linearizable reads and writes by default, backed by Raft consensus.
- **Watch streams** — subscribe to changes on a key or prefix and receive every event, optionally replayed from a past revision.
- **Leases (TTL keys)** — attach keys to a lease that expires; the keys are deleted automatically when the lease lapses. The basis for service discovery and leader election.
- **Multi-version concurrency control (MVCC)** — every change creates a new revision, so you can read historical state and watch from any point in history.
- **Atomic transactions** — compare-and-set across multiple keys in a single `Txn`.
- **Mini-tools built in** — distributed locks (`lock`), leader election (`elect`), and cross-cluster mirroring (`make-mirror`).
- **Role-based authentication** — users, roles, and per-key/per-prefix permissions, with `simple` token or `jwt` auth.

### Prerequisites

A 64-bit Linux, macOS, or Windows host. A single node is enough to try etcd; production clusters use an odd number of members (3 or 5).

### Run a single node

```bash
etcd
```

```text
{"level":"info","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2379"}
{"level":"info","msg":"ready to serve client requests"}
```

By default etcd listens for clients on `http://localhost:2379` and for peers on `http://localhost:2380`, and writes its state to `./default.etcd`.

### Write and read your first key

```bash
etcdctl put greeting "Hello, etcd"
```

```text
OK
```

```bash
etcdctl get greeting
```

```text
greeting
Hello, etcd
```

> [!TIP]
> Add `-w table` (or `-w json`) to most read commands for structured output: `etcdctl get greeting -w json`. The default is the human-readable `simple` format.

### What to try next

- [Install etcd properly](#2-installation) for your platform.
- Learn the [everyday key-value commands](#41-key-value-commands).
- Set up [watches](#43-watching-for-changes), [leases](#44-leases-ttl-keys), and [transactions](#45-transactions-txn).
- For cluster architecture, the full configuration flag reference, and security hardening, see the **etcd Administrator & Developer Guide**.

---

## 2. Installation

### System requirements

| Resource | Recommendation |
|---|---|
| OS | Linux (primary), macOS, Windows (64-bit) |
| Memory | 2–8 GB for typical clusters; more for large keyspaces |
| Disk | Fast SSD strongly recommended — etcd is sensitive to disk write latency (it `fsync`s every Raft log entry) |
| Network | Low-latency links between members; election timing assumes round-trips well under the election timeout |
| Ports | `2379` (client), `2380` (peer) by convention |

### Install methods

=== "Pre-built binary"
    Download the release archive for your platform from the etcd releases page, extract it, and place the three binaries on your `PATH`:

    ```bash
    tar xzvf etcd-*-linux-amd64.tar.gz
    cd etcd-*-linux-amd64
    sudo cp etcd etcdctl etcdutl /usr/local/bin/
    ```

=== "Docker"
    The official image runs the `etcd` server. Publish the client and peer ports:

    ```bash
    docker run -d --name etcd \
      -p 2379:2379 -p 2380:2380 \
      gcr.io/etcd-development/etcd:v3.7.0 \
      etcd --advertise-client-urls http://0.0.0.0:2379 \
           --listen-client-urls http://0.0.0.0:2379
    ```

=== "From source"
    etcd is written in Go (the source tree targets a recent Go toolchain). From a clone of the repository:

    ```bash
    ./scripts/build.sh
    ls ./bin
    # etcd  etcdctl  etcdutl
    ```

> [!IMPORTANT]
> By default etcd serves clients over **plain HTTP with no authentication**. This is fine for a laptop, but for anything reachable by other machines you must configure TLS and (optionally) authentication. The server prints `serving client traffic insecurely; this is strongly discouraged!` to remind you. See the Admin Guide for TLS setup.

### Post-install verification

```bash
etcd --version
```

```text
etcd Version: 3.7.0
Git SHA: 7d330a1
Go Version: go1.26
Go OS/Arch: linux/amd64
```

```bash
etcdctl version
```

```text
etcdctl version: 3.7.0
API version: 3.7
```

Confirm a running server answers:

```bash
etcdctl endpoint health
```

```text
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.095242ms
```

### Shell completion

Both `etcdctl` and `etcdutl` can generate completion scripts:

```bash
source <(etcdutl completion bash)
# persist it:
etcdutl completion bash | sudo tee /etc/bash_completion.d/etcdutl >/dev/null
```

Supported shells: `bash`, `zsh`, `fish`, `powershell`.

---

## 3. Configuration (for client users)

This section covers the handful of settings most users set on the **client** (`etcdctl`). The complete server flag reference lives in the Admin Guide.

### Pointing etcdctl at your cluster

Every `etcdctl` command accepts these global flags (inherited by all subcommands):

| Flag | Purpose | Default |
|---|---|---|
| `--endpoints` | Comma-separated list of cluster endpoints | `127.0.0.1:2379` |
| `--dial-timeout` | Dial timeout for the client connection | `2s` |
| `--command-timeout` | Timeout for short, single-shot commands | `5s` |
| `--keepalive-time` | Keepalive probe interval | `2s` |
| `--keepalive-timeout` | Keepalive probe timeout | `6s` |
| `-w`, `--write-out` | Output format: `simple`, `json`, `protobuf`, `fields`, `table` | `simple` |
| `--hex` | Print keys/values as hex | `false` |
| `--user` | Username or `user:password` for authentication | — |
| `--password` | Password (use with `--user`) | — |
| `--debug` | Enable verbose client + gRPC debug logging | `false` |

TLS flags:

| Flag | Purpose |
|---|---|
| `--cacert` | CA bundle to verify the server's certificate |
| `--cert` | Client TLS certificate file |
| `--key` | Client TLS key file |
| `--insecure-transport` | Disable transport security for client connections |
| `--insecure-skip-tls-verify` | Skip server certificate verification (testing only) |

Example against a 3-node TLS cluster:

```bash
etcdctl \
  --endpoints=https://10.0.0.10:2379,https://10.0.0.11:2379,https://10.0.0.12:2379 \
  --cacert=/etc/etcd/ca.pem --cert=/etc/etcd/client.pem --key=/etc/etcd/client-key.pem \
  endpoint health
```

### Environment variables

Any global flag can be set with an environment variable: prefix the flag name with `ETCDCTL_`, upper-case it, and replace dashes with underscores.

```bash
export ETCDCTL_ENDPOINTS=https://10.0.0.10:2379,https://10.0.0.11:2379
export ETCDCTL_DIAL_TIMEOUT=3s
export ETCDCTL_CACERT=/etc/etcd/ca.pem
export ETCDCTL_CERT=/etc/etcd/client.pem
export ETCDCTL_KEY=/etc/etcd/client-key.pem
etcdctl endpoint health   # no flags needed now
```

> [!NOTE]
> `ETCDCTL_` environment variables only apply to **global** flags. The special `ETCDCTL_API` variable is for etcdctl's internal use (it selects the v2 vs v3 API and defaults to v3).

> [!TIP]
> A few commands read their *arguments* from dedicated environment variables too — for example `watch` honors `ETCDCTL_WATCH_KEY` and `ETCDCTL_WATCH_RANGE_END`. These are covered in the relevant command sections below.

---

## 4. Core Usage

`etcdctl` is organized into command groups. The sections below follow what you actually want to do: store and read data, react to changes, manage TTLs, run transactions, administer the cluster, and secure it.

### 4.1 Key-value commands

#### `put` — write a key

```text
etcdctl put [options] <key> <value>
```

Assigns `<value>` to `<key>`, overwriting any existing value. RPC: `Put`.

| Option | Description |
|---|---|
| `--lease` | Lease ID (hex) to attach the key to |
| `--prev-kv` | Return the previous key/value before the write |
| `--ignore-value` | Update the key keeping its current value (e.g. to re-attach a lease) |
| `--ignore-lease` | Update the key keeping its current lease |

```bash
etcdctl put foo bar
```

```text
OK
```

Attach a lease, then inspect:

```bash
etcdctl put foo bar --lease=1234abcd
```

```text
OK
```

Return the prior value while overwriting:

```bash
etcdctl put foo baz --prev-kv
```

```text
OK
foo
bar
```

> [!NOTE]
> If you omit `<value>`, `put` reads the value from standard input. If a value begins with `-`, separate it with `--`: `etcdctl put -- mykey -dashed-value`. Multi-line or space-containing values must be quoted: `etcdctl put foo "bar 1 2 3"`.

#### `get` — read a key or range

```text
etcdctl get [options] <key> [range_end]
```

Reads one key, or the half-open range `[key, range_end)` if `range_end` is given. RPC: `Range`.

| Option | Description |
|---|---|
| `--prefix` | Get all keys matching the prefix |
| `--from-key` | Get all keys ≥ the given key (byte compare) |
| `--keys-only` | Return keys without values |
| `--limit` | Maximum number of results |
| `--rev` | Read at a specific historical revision |
| `--order` | `ASCEND` or `DESCEND` |
| `--sort-by` | `CREATE`, `KEY`, `MODIFY`, `VALUE`, or `VERSION` |
| `--consistency` | `l` (linearizable, default) or `s` (serializable) |
| `--print-value-only` | Print only values (with `-w simple`) |
| `--min-create-revision` / `--max-create-revision` | Restrict by create revision |
| `--min-mod-revision` / `--max-mod-revision` | Restrict by modify revision |

Read one key:

```bash
etcdctl get foo
```

```text
foo
bar
```

Read everything under a prefix:

```bash
etcdctl get --prefix app/
```

```text
app/config/db
postgres://...
app/config/port
8080
```

Read all keys from a starting point, keys only:

```bash
etcdctl get --from-key '' --keys-only
```

Read a key as it was at revision 7:

```bash
etcdctl get foo --rev=7
```

> [!TIP]
> Serializable reads (`--consistency=s`) skip the cross-cluster round trip and are faster, but may return slightly stale data because they are served from the local member without a quorum check. Use them when low latency matters more than reading the absolute latest value.

#### `del` — delete keys

```text
etcdctl del [options] <key> [range_end]
```

Deletes one key, or the range `[key, range_end)`. RPC: `DeleteRange`. Prints the number of keys removed.

| Option | Description |
|---|---|
| `--prefix` | Delete all keys matching the prefix |
| `--from-key` | Delete all keys ≥ the given key |
| `--prev-kv` | Return the deleted key/value pairs |

```bash
etcdctl del foo
```

```text
1
```

```bash
etcdctl del --prefix app/
```

```text
2
```

#### Hex output

If keys or values contain non-printable bytes, `simple` output can be ambiguous. Use `--hex` to hex-encode keys and values.

### 4.2 Compaction

```text
etcdctl compaction [options] <revision>
```

Because etcd keeps every past revision (MVCC), history grows without bound until you compact it. `compaction` discards all event history *prior to* the given revision to reclaim space in the backend. RPC: `Compact`.

| Option | Description |
|---|---|
| `--physical` | Wait for compaction to physically remove old revisions before returning |

```bash
etcdctl compaction 1234
```

```text
compacted revision 1234
```

> [!WARNING]
> After compaction, watches and `get --rev` requests for revisions **older than** the compacted revision fail with `mvcc: required revision has been compacted`. Compact only history you no longer need to read or replay. Most operators enable automatic compaction on the server (`--auto-compaction-mode` / `--auto-compaction-retention`) instead of running this by hand.

> [!NOTE]
> Compaction frees revisions inside the database, but it does **not** shrink the database file on disk. To return free space to the filesystem, run [`defrag`](#48-maintenance-commands).

### 4.3 Watching for changes

```text
etcdctl watch [options] [key | prefix] [range_end] [-- exec-command arg1 ...]
```

Streams events for a key or prefix until you stop it or it errors. RPC: `Watch`.

| Option | Description |
|---|---|
| `--prefix` | Watch all keys under the prefix |
| `--prev-kv` | Include the previous key/value with each event |
| `--rev` | Start watching from a past revision (replays history) |
| `--interactive`, `-i` | Interactive session; issue multiple `watch` requests on one stream |
| `--hex` | Hex-encode output |

Watch a single key (blocks, printing events as they happen):

```bash
etcdctl watch foo
```

```text
PUT
foo
bar
```

Watch a prefix and run a command on each event — the event metadata is passed via `ETCD_WATCH_*` environment variables:

```bash
etcdctl watch --prefix app/ -- sh -c 'echo changed: $ETCD_WATCH_KEY = $ETCD_WATCH_VALUE'
```

```text
PUT
app/port
8080
changed: app/port = 8080
```

Replay everything that happened since revision 5, then keep streaming:

```bash
etcdctl watch foo --rev=5
```

> [!TIP]
> You can drive `watch` entirely from the environment: set `ETCDCTL_WATCH_KEY` (and optionally `ETCDCTL_WATCH_RANGE_END`) and run `etcdctl watch`. This is handy in scripts and systemd units.

### 4.4 Leases (TTL keys)

A **lease** is a time-to-live object. Keys attached to a lease are deleted automatically when the lease expires. Refresh ("keep alive") a lease to keep its keys alive — the foundation of service registration and leader election.

| Command | Purpose | RPC |
|---|---|---|
| `lease grant <ttl>` | Create a lease with a TTL in seconds | `LeaseGrant` |
| `lease revoke <leaseID>` | Destroy a lease and delete its attached keys | `LeaseRevoke` |
| `lease timetolive <leaseID> [--keys]` | Show remaining TTL (and attached keys) | `LeaseTimeToLive` |
| `lease list` | List all active leases | `LeaseLeases` |
| `lease keep-alive <leaseID>` | Periodically refresh a lease so it never expires | `LeaseKeepAlive` |

Grant a 60-second lease and attach a key to it:

```bash
etcdctl lease grant 60
```

```text
lease 32695410dcc0ca06 granted with TTL(60s)
```

```bash
etcdctl put svc/web/1 "10.0.0.5:8080" --lease=32695410dcc0ca06
```

```text
OK
```

Inspect a lease and its keys:

```bash
etcdctl lease timetolive 32695410dcc0ca06 --keys
```

```text
lease 32695410dcc0ca06 granted with TTL(60s), remaining(48s), attached keys([svc/web/1])
```

Keep it alive (runs until interrupted):

```bash
etcdctl lease keep-alive 32695410dcc0ca06
```

```text
lease 32695410dcc0ca06 keepalived with TTL(60)
lease 32695410dcc0ca06 keepalived with TTL(60)
...
```

> [!NOTE]
> The server grants a TTL **greater than or equal to** what you request and selects the final value. The granted lease ID is a hexadecimal number — pass it verbatim to `--lease` and the other lease commands.

### 4.5 Transactions (`txn`)

```text
etcdctl txn [options]
```

A transaction reads a list of **comparisons**, a **success** block, and a **failure** block from standard input and applies them as a single atomic operation. If every comparison is true, the success block runs; otherwise the failure block runs. RPC: `Txn`.

| Option | Description |
|---|---|
| `-i`, `--interactive` | Prompt for compares, success, and failure interactively |
| `--hex` | Hex-encode output |

Comparisons can test a key's `create` revision, `mod` revision, `value`, `version`, or attached `lease`, using `<`, `=`, or `>`.

Interactive example (compare-and-set):

```bash
etcdctl txn -i
```

```text
compares:
mod("key1") > "0"

success requests (get, put, del):
put key1 "overwrote-key1"

failure requests (get, put, del):
put key1 "created-key1"

# SUCCESS

# OK
```

Non-interactive, via a here-string:

```bash
etcdctl txn <<<'mod("key1") > "0"

put key1 "overwrote-key1"

put key1 "created-key1"

'
```

> [!WARNING]
> Inside a `txn`, multi-line values must escape newlines as literal `\n`. Unlike `put`, the shell will *not* convert real newlines for you — a literal newline in a `txn` value causes a parse failure.

### 4.6 Cluster membership

| Command | Purpose | RPC |
|---|---|---|
| `member list` | List all members (ID, status, name, peer/client URLs) | `MemberList` |
| `member add <name> --peer-urls=...` | Add a new voting member | `MemberAdd` |
| `member update <memberID> --peer-urls=...` | Change a member's peer URLs | `MemberUpdate` |
| `member remove <memberID>` | Remove a member from consensus | `MemberRemove` |
| `move-leader <transferee-id>` | Transfer leadership to another member | `MoveLeader` |

List members as a table:

```bash
etcdctl -w table member list
```

```text
+------------------+---------+--------+------------------------+------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+------------------+---------+--------+------------------------+------------------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:12380 | http://127.0.0.1:2379  |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
+------------------+---------+--------+------------------------+------------------------+
```

Add a member. etcd prints the environment variables to start the new member with:

```bash
etcdctl member add infra4 --peer-urls=https://10.0.0.13:2380
```

```text
Member ced000fda4d05edf added to cluster 8c4281cc65c7b112

ETCD_NAME="infra4"
ETCD_INITIAL_CLUSTER="infra4=https://10.0.0.13:2380,infra1=http://10.0.0.30:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
```

> [!CAUTION]
> Adding or removing voting members changes the quorum size. Add members **one at a time** and wait for each to sync and report `started` before changing membership again. Removing too many members at once, or adding a member to a cluster that can't reach quorum, can make the cluster lose quorum and stop serving writes. Consider adding a **learner** (non-voting) member first and promoting it once it has caught up.

Transfer leadership away from the current leader (the request must be sent to the leader):

```bash
etcdctl --endpoints ${leader_ep} move-leader c89feb932daef420
```

```text
Leadership transferred from 45ddc0e800e20b93 to c89feb932daef420
```

### 4.7 Endpoint inspection

| Command | Purpose |
|---|---|
| `endpoint health` | Check whether endpoints can participate in consensus |
| `endpoint status` | Per-endpoint version, DB size, leader, raft term/index |
| `endpoint hashkv` | KV history hash (used to detect divergence/corruption) |

The `--cluster` flag fetches the full member list and runs the check against every member:

```bash
etcdctl endpoint --cluster health
```

```text
http://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.060091ms
http://127.0.0.1:22379 is healthy: successfully committed proposal: took = 903.138µs
http://127.0.0.1:32379 is healthy: successfully committed proposal: took = 1.113848ms
```

```bash
etcdctl -w table endpoint --cluster status
```

```text
+------------------------+------------------+---------------+---------+-----------+-----------+------------+
|        ENDPOINT        |        ID        |    VERSION    | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------------+------------------+---------------+---------+-----------+-----------+------------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |   3.7.0       |   25 kB |     false |         2 |          8 |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.7.0       |   25 kB |      true |         2 |          8 |
+------------------------+------------------+---------------+---------+-----------+-----------+------------+
```

### 4.8 Maintenance commands

#### Alarms

etcd raises an **alarm** (for example `NOSPACE` when the backend exceeds `--quota-backend-bytes`) to protect the cluster, putting it into a limited/read-only mode. Inspect and clear alarms:

```bash
etcdctl alarm list
```

```text
alarm:NOSPACE
```

```bash
etcdctl alarm disarm
```

```text
alarm:NOSPACE
```

> [!IMPORTANT]
> A `NOSPACE` alarm means the database hit its quota and etcd has stopped accepting writes. Recovery is a sequence: compact old revisions, run `defrag` to reclaim file space, then `alarm disarm`. Disarming *without* freeing space will not restore write capacity.

#### Defragment (online)

```text
etcdctl defrag [--cluster] [--endpoints=...]
```

Releases free space (left behind by deletes and compaction) in a **running** member's backend file back to the filesystem.

```bash
etcdctl defrag --cluster
```

```text
Finished defragmenting etcd member[http://127.0.0.1:2379]
Finished defragmenting etcd member[http://127.0.0.1:22379]
Finished defragmenting etcd member[http://127.0.0.1:32379]
```

> [!WARNING]
> Defragmenting a live member **blocks reads and writes on that member** while it rebuilds its database. Defrag one member at a time, never all at once, and avoid the leader during peak load. To defragment an offline data directory instead, use `etcdutl defrag` (see §5). The defrag request is **not** replicated — it applies only to the targeted endpoints, so list every member (or use `--cluster`).

#### Snapshots (backups)

```text
etcdctl snapshot save <filename>
```

Writes a point-in-time snapshot of the backend database to a file:

```bash
etcdctl snapshot save backup-$(date +%F).db
```

```text
Snapshot saved at backup-2026-06-04.db
```

> [!IMPORTANT]
> In v3.6 and later, `snapshot restore` and `snapshot status` were **removed from `etcdctl`** and moved to `etcdutl`. Use `etcdutl snapshot restore` / `etcdutl snapshot status` (see §5). `etcdctl snapshot save` remains, because it talks to a live server.

#### Downgrade

`downgrade` lets an administrator force a cluster's version down a minor release — an experimental v3.6+ feature, not recommended for production. It runs in stages:

```bash
etcdctl downgrade validate 3.6
```

```text
Downgrade validate success, cluster version 3.7
```

```bash
etcdctl downgrade enable 3.6
```

```text
Downgrade enable success, cluster version 3.7
```

After enabling, replace each member's binary with the older version one at a time, then confirm. `etcdctl downgrade cancel` aborts an in-progress downgrade.

> [!CAUTION]
> Downgrade is experimental. Validate first, downgrade members one at a time, and watch the leader log for `the cluster has been downgraded`. Take a snapshot before you begin.

### 4.9 Concurrency commands

#### `lock` — distributed mutex

```text
etcdctl lock [--ttl <seconds>] <lockname> [command arg1 ...]
```

Acquires a named distributed lock; holds it until `etcdctl` exits. If a command is given, it runs while the lock is held (with `ETCD_LOCK_KEY` and `ETCD_LOCK_REV` set in its environment).

```bash
etcdctl lock mylock echo "lock acquired"
```

```text
lock acquired
```

> [!WARNING]
> If `etcdctl lock` is killed abnormally and can't release the lock, the lock stays held until its lease expires — by default up to 60 seconds. Plan for that delay in critical sections.

#### `elect` — leader election

```text
etcdctl elect [--listen] <election-name> [proposal]
```

A candidate announces itself with a proposal value; whoever wins has their proposal published as the election's leader value. `--listen` observes without participating.

```bash
etcdctl elect myservice node-1
```

```text
myservice/1456952310051373265
node-1
```

### 4.10 Authentication commands

etcd supports role-based access control. The model: create users, create roles, grant key/prefix permissions to roles, grant roles to users, then enable auth. A `root` user with the `root` role is required before enabling.

| Command | Purpose |
|---|---|
| `auth enable` / `auth disable` | Turn authentication on/off |
| `user add <name>` | Create a user (prompts for password) |
| `user list` / `user get <name>` / `user delete <name>` | Manage users |
| `user passwd <name>` | Change a user's password |
| `user grant-role` / `user revoke-role` | Assign/remove roles |
| `role add <name>` / `role get` / `role list` / `role delete` | Manage roles |
| `role grant-permission [--prefix] <role> <read\|write\|readwrite> <key> [endkey]` | Grant key access |
| `role revoke-permission <role> <key> [endkey]` | Revoke key access |

Bootstrap and enable auth:

```bash
etcdctl user add root
# Password of root: ********
etcdctl role add root
etcdctl user grant-role root root
etcdctl auth enable
```

```text
Authentication Enabled
```

Create an app role limited to one prefix, and a user for it:

```bash
etcdctl --user=root:rootpw role add app
etcdctl --user=root:rootpw role grant-permission --prefix app readwrite app/
etcdctl --user=root:rootpw user add appuser
etcdctl --user=root:rootpw user grant-role appuser app
```

```text
Role app created
Role app updated
User appuser created
Role app is granted to user appuser
```

> [!CAUTION]
> Once `auth enable` succeeds, **every** request must authenticate, including administrative ones. Make sure the `root` user exists and you know its password before enabling — there is no back door. Lose the root password and you must recover from a snapshot.

### 4.11 Mirroring

```text
etcdctl make-mirror [options] <destination>
```

Continuously mirrors a key prefix from the current cluster to a destination cluster. Options include `--prefix`, `--dest-prefix`, `--no-dest-prefix`, the destination TLS flags (`--dest-cacert`, `--dest-cert`, `--dest-key`), `--dest-insecure-transport`, and `--max-txn-ops`. It prints a running count of keys transferred (updated about every 30 seconds).

```bash
etcdctl make-mirror --prefix app/ mirror.example.com:2379
```

```text
10
18
```

### 4.12 Health and performance checks

`etcdctl check perf` runs a 60-second write workload and reports whether the cluster meets throughput and latency targets. `etcdctl check datascale` estimates memory use for a workload.

| Load | Clients | Target writes/sec |
|---|---|---|
| `s` (small) | 50 | 10,000 |
| `m` (medium) | 200 | 100,000 |
| `l` (large) | 500 | 1,000,000 |
| `xl` (xLarge) | 1000 | 3,000,000 |

```bash
etcdctl check perf --load=s
```

```text
60 / 60 ... 100.00% 1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.087509s
PASS: Stddev is 0.011084s
PASS
```

Both checks accept `--prefix`, `--auto-compact`, and `--auto-defrag` to clean up the keys they create.

> [!NOTE]
> `check perf` passes or fails relative to the workload model you pick. A single laptop node will pass `s` and fail `l` — that's expected. Match the load to your hardware and cluster size.

### 4.13 Output formats

Every command that maps to an RPC respects `-w`/`--write-out`:

- **`simple`** (default) — human-readable, customized per command.
- **`json`** — the JSON encoding of the RPC response. Keys and values are base64-encoded because etcd stores raw bytes.
- **`protobuf`** — the raw protobuf-encoded response.
- **`fields`** — a coreutils-friendly `"Field" : value` line format.
- **`table`** — an ASCII table (used heavily by `member list` and `endpoint status`).

On failure, every command writes an error to standard error and returns a non-zero exit code regardless of format.

---

## 5. The `etcdutl` offline tool

`etcdutl` operates **directly on data-dir files and snapshots** — no running server. Use it for restores, offline defrag, schema migration, and integrity hashing.

| Command | Purpose |
|---|---|
| `etcdutl snapshot status <file>` | Print hash, revision, total keys, total size, version of a snapshot |
| `etcdutl snapshot restore <file> --data-dir <dir> [options]` | Rebuild a member data directory from a snapshot |
| `etcdutl defrag --data-dir <dir>` | Defragment an **offline** backend database |
| `etcdutl hashkv <file> [--rev N]` | Print the KV history hash of a database file |
| `etcdutl migrate --data-dir <dir> --target-version X.Y [--force]` | Migrate data-dir schema between etcd versions |
| `etcdutl version` | Print etcdutl and API version |
| `etcdutl completion <shell>` | Generate a shell completion script |

### Inspect a snapshot

```bash
etcdutl snapshot status backup-2026-06-04.db -w table
```

```text
+----------+----------+------------+------------+---------+
|   hash   | revision | total keys | total size | version |
+----------+----------+------------+------------+---------+
| 84e9a5bf |     1234 |        842 |      25 kB |   3.7.0 |
+----------+----------+------------+------------+---------+
```

### Restore a snapshot into a new data directory

```text
etcdutl snapshot restore <file> --data-dir <dir> [options]
```

| Option | Default | Purpose |
|---|---|---|
| `--data-dir` | — | Output data directory |
| `--wal-dir` | (uses `--data-dir`) | Separate WAL directory |
| `--name` | `default` | Member name |
| `--initial-cluster` | `default=http://localhost:2380` | Cluster layout for the restored member |
| `--initial-cluster-token` | `etcd-cluster` | Cluster token |
| `--initial-advertise-peer-urls` | `http://localhost:2380` | This member's peer URLs |
| `--skip-hash-check` | `false` | Skip integrity hash (required when restoring from a copied data dir) |
| `--initial-memory-map-size` | backend default | Initial mmap size of the database in bytes |
| `--bump-revision` | `0` | Increase the latest revision after restore |
| `--mark-compacted` | `false` | Mark the post-restore revision as a compaction point (required when `--bump-revision > 0`) |

```bash
etcdutl snapshot restore backup-2026-06-04.db \
  --name infra1 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --data-dir /var/lib/etcd-restored
```

> [!CAUTION]
> Restore each member of the new cluster from the **same** snapshot, giving each its own `--name`, `--data-dir`, and `--initial-advertise-peer-urls` but the **same** `--initial-cluster`. Then start all members with `--initial-cluster-state new`. Restoring into a non-empty `--data-dir`, or restoring members from different snapshots, will fail or produce a divergent cluster.

### Offline defragment

```bash
etcdutl defrag --data-dir /var/lib/etcd
```

```text
finished defragmenting the storage
```

### Migrate the data-dir schema

```bash
etcdutl migrate --data-dir /var/lib/etcd --target-version 3.6
```

The minimum supported `--target-version` is `3.5`. `--force` overrides the storage version even if normal migration fails (not recommended).

---

## 6. Common Tasks (recipes)

### Bootstrap a 3-node cluster (static)

> [!NOTE]
> This shows the shape of the command. The full flag reference and TLS hardening are in the Admin Guide.

```bash
# on infra1 (repeat with the right name/IP on infra2 and infra3)
etcd --name infra1 \
  --data-dir /var/lib/etcd \
  --listen-client-urls https://10.0.0.10:2379 \
  --advertise-client-urls https://10.0.0.10:2379 \
  --listen-peer-urls https://10.0.0.10:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-cluster-token my-etcd-token \
  --initial-cluster-state new
```

Verify:

```bash
etcdctl --endpoints=https://10.0.0.10:2379,https://10.0.0.11:2379,https://10.0.0.12:2379 endpoint health
```

### Take and verify a backup

```bash
etcdctl snapshot save /backups/etcd-$(date +%F).db
etcdutl snapshot status /backups/etcd-$(date +%F).db -w table
```

### Watch a config prefix and reload an app

```bash
etcdctl watch --prefix /myapp/config/ -- systemctl reload myapp
```

### Register a service with a TTL and keep it alive

```bash
LEASE=$(etcdctl lease grant 30 | awk '{print $2}')
etcdctl put /svc/web/$(hostname) "$(hostname -i):8080" --lease=$LEASE
etcdctl lease keep-alive $LEASE   # keep running for the lifetime of the process
```

### Atomic "create if not exists"

```bash
etcdctl txn <<<'create("/locks/job") = "0"

put /locks/job "owner-A"

get /locks/job

'
```

If the key doesn't exist yet (`create` revision is 0), the success branch claims it; otherwise the failure branch reads the current owner.

### Reclaim disk after heavy churn

```bash
REV=$(etcdctl endpoint status -w json | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
etcdctl compaction "$REV"
etcdctl defrag --cluster
```

### Rotate a member out and a new one in

```bash
# add the replacement (one at a time!)
etcdctl member add infra4 --peer-urls=https://10.0.0.13:2380
# start infra4 with the printed ETCD_* variables and --initial-cluster-state existing
etcdctl member list                       # confirm infra4 is started
etcdctl member remove <old-member-id>      # only after infra4 is healthy
```

---

## 7. Troubleshooting

Each entry lists the exact message, why it happens, how to fix it, and how to prevent it.

### Error: `etcdserver: request timed out`

**Cause:** The cluster could not commit the request within the timeout — usually slow disk (`fsync` latency), an overloaded leader, or a network partition affecting quorum.

**Solution:**
1. Check member health: `etcdctl endpoint --cluster health`.
2. Check disk and CPU on each member; etcd needs fast `fsync`.
3. Look for `etcdserver: leader changed` or `etcdserver: no leader` in the logs.

**Prevention:** Use SSDs, isolate etcd from noisy neighbors, and keep `--election-timeout` ≥ 5× `--heartbeat-interval` to tolerate latency.

### Error: `etcdserver: no leader`

**Cause:** The cluster has lost quorum (a majority of members are unreachable) and cannot elect a leader.

**Solution:**
1. Run `etcdctl endpoint --cluster status` against any reachable endpoint to see who is up.
2. Restart failed members; restore network connectivity.
3. If members were permanently lost and quorum cannot be regained, restore from a snapshot with `etcdutl snapshot restore`.

**Prevention:** Run an odd number of members (3 or 5) across failure domains so a single failure never breaks quorum.

### Error: `etcdserver: mvcc: database space exceeded` / alarm `NOSPACE`

**Cause:** The backend database reached `--quota-backend-bytes`; etcd entered read-only mode and raised a `NOSPACE` alarm.

**Solution:**
1. Compact old history: `etcdctl compaction <recent-revision>`.
2. Reclaim space: `etcdctl defrag --cluster`.
3. Clear the alarm: `etcdctl alarm disarm`.

**Prevention:** Enable auto-compaction (`--auto-compaction-mode`/`--auto-compaction-retention`) and monitor `etcd_mvcc_db_total_size_in_bytes`.

### Error: `mvcc: required revision has been compacted`

**Cause:** You requested a `get --rev` or `watch --rev` for a revision older than the last compaction.

**Solution:** Re-issue the request at a revision at or after the current compacted revision, or watch from the current revision (omit `--rev`).

**Prevention:** Don't compact history you still need to read or replay; track the compacted revision returned by `compaction`.

### Error: `etcdserver: request is too large`

**Cause:** The request exceeded `--max-request-bytes` (default ~1.5 MiB) or `--max-txn-ops` (default 128).

**Solution:** Split the work into smaller writes or a smaller transaction; raise the server limits only with care (large requests increase latency and memory use).

**Prevention:** Keep individual values and transactions small. etcd is a coordination store, not a blob store.

### Error: `etcdserver: too many requests`

**Cause:** The server is overloaded or applying a backlog (often during/after a defrag, snapshot, or leader change).

**Solution:** Back off and retry with jitter; reduce client concurrency.

**Prevention:** Rate-limit clients; avoid defragmenting under peak load.

### Error: `auth: authentication failed, invalid user ID or password`

**Cause:** Wrong username/password, or auth is enabled and the request didn't supply credentials.

**Solution:** Re-check `--user=user:password` (or `--user` + `--password`). Confirm the user exists: `etcdctl --user=root:... user list`.

**Prevention:** Store credentials in `ETCDCTL_*` environment variables or a secrets manager; verify with `etcdctl --user=... auth status`.

### Error: `auth: permission denied`

**Cause:** The authenticated user's roles don't grant the needed access to that key/range.

**Solution:** Grant the role the right permission, e.g. `role grant-permission --prefix myrole readwrite app/`.

**Prevention:** Design roles around key prefixes; use `user get <name> --detail` to audit effective permissions.

### Error: `etcdserver: can only promote a learner member which is in sync with the leader`

**Cause:** You tried to promote a learner that hasn't caught up to the leader's log yet.

**Solution:** Wait until the learner's raft index matches (check `endpoint status`), then retry the promotion.

**Prevention:** Always confirm a learner is in sync before promoting it. (Related community report: issue #20757 — promoting a member while auth is enabled.)

### Error: `etcdctl: no available endpoints` / `etcdclient: no available endpoints`

**Cause:** None of the configured `--endpoints` were reachable.

**Solution:** Verify the addresses and ports, network/firewall, and TLS settings; test with `etcdctl endpoint health`.

**Prevention:** List multiple endpoints so a single down member doesn't break the client.

### Error: `Error: context deadline exceeded`

**Cause:** A command exceeded `--command-timeout` or `--dial-timeout`.

**Solution:** Increase the timeout (`--command-timeout=15s`) or investigate why the cluster is slow.

**Prevention:** Set realistic timeouts for cross-region clusters.

### Error: `etcdserver: mvcc: required revision is a future revision`

**Cause:** You asked for a revision (e.g. in `etcdutl hashkv --rev`) higher than any that exists.

**Solution:** Use a revision ≤ the latest; omit `--rev` to use the latest.

**Prevention:** Read the current revision from `endpoint status` first.

### Error: `snapshot missing hash but --skip-hash-check=false`

**Cause:** Restoring a snapshot that was copied directly from a data directory (so it has no integrity hash).

**Solution:** Re-run `etcdutl snapshot restore` with `--skip-hash-check`.

**Prevention:** Prefer `etcdctl snapshot save` (which writes a hash) over copying the DB file.

### Error: `serving client traffic insecurely; this is strongly discouraged!`

**Cause:** Not an error — a warning that the server is running without client TLS.

**Solution:** Configure `--cert-file`/`--key-file` (and a CA for client-cert auth). See the Admin Guide.

**Prevention:** Always enable TLS for non-local clusters.

### Error: `etcdmain: error verifying flags, '...' is not a valid flag`

**Cause:** A typo'd or removed flag on the `etcd` command line.

**Solution:** Run `etcd --help` and check the flag name; some flags (e.g. `--logger=capnslog`) were removed in v3.5.

**Prevention:** Pin your launch flags and review them against the Admin Guide's reference when upgrading.

> [!TIP]
> Add `--debug` to any `etcdctl` command to dump the effective configuration and gRPC-level logs to stderr — invaluable when diagnosing TLS or connectivity problems.

---

## 8. FAQ

**Q: What ports does etcd use?**

By convention, `2379` for client (gRPC/HTTP) traffic and `2380` for peer (Raft) traffic. These come from the default `--listen-client-urls`/`--listen-peer-urls` and can be changed.

**Q: How many members should my cluster have?**

An odd number — 3 or 5. A 3-member cluster tolerates 1 failure; a 5-member cluster tolerates 2. Even-numbered clusters give no extra fault tolerance and just raise the quorum size.

**Q: What's the difference between `etcdctl` and `etcdutl`?**

`etcdctl` talks to a **running** cluster over the network (put/get, watch, membership, auth, online defrag, snapshot save). `etcdutl` works **offline** on data-dir files and snapshots (restore, offline defrag, migrate, hashkv). Snapshot restore/status moved from `etcdctl` to `etcdutl` in v3.6.

**Q: Why doesn't deleting keys free disk space?**

etcd uses MVCC: deletes and overwrites create new revisions but keep the old ones until you **compact**. Even after compaction, the backend file doesn't shrink — its freed pages go on an internal free list. Run `defrag` to return that space to the filesystem.

**Q: What is a lease and when should I use one?**

A lease is a TTL object; keys attached to it are deleted when it expires. Use leases for ephemeral data — service registrations, session keys, and the locks/elections behind `lock` and `elect`. Keep a lease alive with `lease keep-alive`.

**Q: How do I do a compare-and-swap?**

Use a `txn` with a comparison on the key's `mod`, `create`, `value`, or `version`. The success block runs only if the comparison holds. See §4.5.

**Q: What does `--consistency=s` (serializable) change?**

A serializable read is served locally without a quorum round trip — faster, but may return slightly stale data. The default is linearizable (`l`), which always reflects the latest committed write.

**Q: Can I get JSON output?**

Yes: `-w json`. Note that keys and values are base64-encoded in JSON because etcd stores raw bytes. `-w table` and `-w fields` are also available.

**Q: How do I back up and restore?**

Back up with `etcdctl snapshot save file.db`. Restore by running `etcdutl snapshot restore file.db --data-dir ...` on each new member (same snapshot, per-member name/peer URLs, shared `--initial-cluster`), then start with `--initial-cluster-state new`.

**Q: How do I enable authentication safely?**

Create the `root` user and `root` role, grant the role to the user, then `auth enable`. After that every request must authenticate — there's no bypass, so confirm the root credentials first.

**Q: What is a learner member?**

A non-voting member that replicates the log but doesn't count toward quorum. Add a learner, let it catch up, then promote it to a voting member. This makes adding members safer because it doesn't change quorum until promotion. The maximum number of learners is set by the server's `--max-learners`.

**Q: How do I move the leader off a node before maintenance?**

`etcdctl move-leader <transferee-id>`, sent to the current leader. Useful before restarting or defragmenting the leader.

**Q: My value starts with a dash and `put` treats it as a flag — what do I do?**

Use `--` to end flag parsing: `etcdctl put -- mykey -my-value`.

**Q: How do I watch from the beginning of history?**

`etcdctl watch <key> --rev=1` replays from the first revision (provided it hasn't been compacted away), then continues live.

**Q: Where is the full server configuration reference?**

In the **etcd Administrator & Developer Guide** — it documents every `etcd` flag, environment variable, TLS option, tuning parameter, and the cluster architecture.

---

_Generated June 2026 from etcd-io/etcd at 7d330a1. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._