etcd — User Guide

etcd is a distributed, reliable key-value store for the most critical data of a distributed system. It keeps a small amount of data consistent and highly available by replicating it across a cluster of machines using the Raft consensus algorithm. If a majority of the cluster is alive, etcd keeps serving — even through machine failures, including leader failures.

This guide is for the people who use etcd: application developers storing configuration and coordination data, operators running etcdctl against a live cluster, and anyone who needs to install etcd, store and watch keys, manage leases, take backups, or set up authentication. It covers the etcd server binary, the etcdctl client, and the etcdutl offline maintenance tool as they behave in the source at commit 7d330a1 (the v3.7 development line).

Note

etcd speaks a gRPC API (the v3 API). etcdctl uses the v3 API by default. The old v2 key-value API and its etcdctl v2 mode are deprecated and on a removal path; everything in this guide is v3.

1. Quick Start

etcd ships as three self-contained binaries:

• etcd — the server. Run one for local development, or three/five for a fault-tolerant cluster.
• etcdctl — the command-line client for talking to a running cluster (put/get keys, watch, leases, membership, auth, backups).
• etcdutl — an offline administration tool that operates directly on data-dir files and snapshots (defrag, restore, migrate, hash) without a running server.

Key features

• Strong consistency — linearizable reads and writes by default, backed by Raft consensus.
• Watch streams — subscribe to changes on a key or prefix and receive every event, optionally replayed from a past revision.
• Leases (TTL keys) — attach keys to a lease that expires; the keys are deleted automatically when the lease lapses. The basis for service discovery and leader election.
• Multi-version concurrency control (MVCC) — every change creates a new revision, so you can read historical state and watch from any point in history.
• Atomic transactions — compare-and-set across multiple keys in a single Txn.
• Mini-tools built in — distributed locks (lock), leader election (elect), and cross-cluster mirroring (make-mirror).
• Role-based authentication — users, roles, and per-key/per-prefix permissions, with simple token or jwt auth.

Prerequisites

A 64-bit Linux, macOS, or Windows host. A single node is enough to try etcd; production clusters use an odd number of members (3 or 5).

Run a single node

etcd

{"level":"info","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2379"}
{"level":"info","msg":"ready to serve client requests"}

By default etcd listens for clients on http://localhost:2379 and for peers on http://localhost:2380, and writes its state to ./default.etcd.

Write and read your first key

etcdctl put greeting "Hello, etcd"

OK

etcdctl get greeting

greeting
Hello, etcd

Tip

Add -w table (or -w json) to most read commands for structured output: etcdctl get greeting -w json. The default is the human-readable simple format.

What to try next

• Install etcd properly for your platform.
• Learn the everyday key-value commands.
• Set up watches, leases, and transactions.
• For cluster architecture, the full configuration flag reference, and security hardening, see the etcd Administrator & Developer Guide.

2. Installation

System requirements

Install methods

Pre-built binary

Download the release archive for your platform from the etcd releases page, extract it, and place the three binaries on your PATH:

tar xzvf etcd-*-linux-amd64.tar.gz
cd etcd-*-linux-amd64
sudo cp etcd etcdctl etcdutl /usr/local/bin/

Docker

The official image runs the etcd server. Publish the client and peer ports:

docker run -d --name etcd \
  -p 2379:2379 -p 2380:2380 \
  gcr.io/etcd-development/etcd:v3.7.0 \
  etcd --advertise-client-urls http://0.0.0.0:2379 \
       --listen-client-urls http://0.0.0.0:2379

From source

etcd is written in Go (the source tree targets a recent Go toolchain). From a clone of the repository:

./scripts/build.sh
ls ./bin
# etcd  etcdctl  etcdutl

Important

By default etcd serves clients over plain HTTP with no authentication. This is fine for a laptop, but for anything reachable by other machines you must configure TLS and (optionally) authentication. The server prints serving client traffic insecurely; this is strongly discouraged! to remind you. See the Admin Guide for TLS setup.

Post-install verification

etcd --version

etcd Version: 3.7.0
Git SHA: 7d330a1
Go Version: go1.26
Go OS/Arch: linux/amd64

etcdctl version

etcdctl version: 3.7.0
API version: 3.7

Confirm a running server answers:

etcdctl endpoint health

127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.095242ms

Shell completion

Both etcdctl and etcdutl can generate completion scripts:

source <(etcdutl completion bash)
# persist it:
etcdutl completion bash | sudo tee /etc/bash_completion.d/etcdutl >/dev/null

Supported shells: bash, zsh, fish, powershell.

3. Configuration (for client users)

This section covers the handful of settings most users set on the client (etcdctl). The complete server flag reference lives in the Admin Guide.

Pointing etcdctl at your cluster

Every etcdctl command accepts these global flags (inherited by all subcommands):

TLS flags:

Example against a 3-node TLS cluster:

etcdctl \
  --endpoints=https://10.0.0.10:2379,https://10.0.0.11:2379,https://10.0.0.12:2379 \
  --cacert=/etc/etcd/ca.pem --cert=/etc/etcd/client.pem --key=/etc/etcd/client-key.pem \
  endpoint health

Environment variables

Any global flag can be set with an environment variable: prefix the flag name with ETCDCTL_, upper-case it, and replace dashes with underscores.

export ETCDCTL_ENDPOINTS=https://10.0.0.10:2379,https://10.0.0.11:2379
export ETCDCTL_DIAL_TIMEOUT=3s
export ETCDCTL_CACERT=/etc/etcd/ca.pem
export ETCDCTL_CERT=/etc/etcd/client.pem
export ETCDCTL_KEY=/etc/etcd/client-key.pem
etcdctl endpoint health   # no flags needed now

Note

ETCDCTL_ environment variables only apply to global flags. The special ETCDCTL_API variable is for etcdctl's internal use (it selects the v2 vs v3 API and defaults to v3).

Tip

A few commands read their arguments from dedicated environment variables too — for example watch honors ETCDCTL_WATCH_KEY and ETCDCTL_WATCH_RANGE_END. These are covered in the relevant command sections below.

4. Core Usage

etcdctl is organized into command groups. The sections below follow what you actually want to do: store and read data, react to changes, manage TTLs, run transactions, administer the cluster, and secure it.

4.1 Key-value commands

put — write a key

etcdctl put [options] <key> <value>

Assigns <value> to <key>, overwriting any existing value. RPC: Put.

etcdctl put foo bar

OK

Attach a lease, then inspect:

etcdctl put foo bar --lease=1234abcd

OK

Return the prior value while overwriting:

etcdctl put foo baz --prev-kv

OK
foo
bar

Note

If you omit <value>, put reads the value from standard input. If a value begins with -, separate it with --: etcdctl put -- mykey -dashed-value. Multi-line or space-containing values must be quoted: etcdctl put foo "bar 1 2 3".

get — read a key or range

etcdctl get [options] <key> [range_end]

Reads one key, or the half-open range [key, range_end) if range_end is given. RPC: Range.

Read one key:

etcdctl get foo

foo
bar

Read everything under a prefix:

etcdctl get --prefix app/

app/config/db
postgres://...
app/config/port
8080

Read all keys from a starting point, keys only:

etcdctl get --from-key '' --keys-only

Read a key as it was at revision 7:

etcdctl get foo --rev=7

Tip

Serializable reads (--consistency=s) skip the cross-cluster round trip and are faster, but may return slightly stale data because they are served from the local member without a quorum check. Use them when low latency matters more than reading the absolute latest value.

del — delete keys

etcdctl del [options] <key> [range_end]

Deletes one key, or the range [key, range_end). RPC: DeleteRange. Prints the number of keys removed.

etcdctl del foo

1

etcdctl del --prefix app/

2

Hex output

If keys or values contain non-printable bytes, simple output can be ambiguous. Use --hex to hex-encode keys and values.

4.2 Compaction

etcdctl compaction [options] <revision>

Because etcd keeps every past revision (MVCC), history grows without bound until you compact it. compaction discards all event history prior to the given revision to reclaim space in the backend. RPC: Compact.

etcdctl compaction 1234

compacted revision 1234

Warning

After compaction, watches and get --rev requests for revisions older than the compacted revision fail with mvcc: required revision has been compacted. Compact only history you no longer need to read or replay. Most operators enable automatic compaction on the server (--auto-compaction-mode / --auto-compaction-retention) instead of running this by hand.

Note

Compaction frees revisions inside the database, but it does not shrink the database file on disk. To return free space to the filesystem, run defrag.

4.3 Watching for changes

etcdctl watch [options] [key | prefix] [range_end] [-- exec-command arg1 ...]

Streams events for a key or prefix until you stop it or it errors. RPC: Watch.

Watch a single key (blocks, printing events as they happen):

etcdctl watch foo

PUT
foo
bar

Watch a prefix and run a command on each event — the event metadata is passed via ETCD_WATCH_* environment variables:

etcdctl watch --prefix app/ -- sh -c 'echo changed: $ETCD_WATCH_KEY = $ETCD_WATCH_VALUE'

PUT
app/port
8080
changed: app/port = 8080

Replay everything that happened since revision 5, then keep streaming:

etcdctl watch foo --rev=5

Tip

You can drive watch entirely from the environment: set ETCDCTL_WATCH_KEY (and optionally ETCDCTL_WATCH_RANGE_END) and run etcdctl watch. This is handy in scripts and systemd units.

4.4 Leases (TTL keys)

A lease is a time-to-live object. Keys attached to a lease are deleted automatically when the lease expires. Refresh ("keep alive") a lease to keep its keys alive — the foundation of service registration and leader election.

Grant a 60-second lease and attach a key to it:

etcdctl lease grant 60

lease 32695410dcc0ca06 granted with TTL(60s)

etcdctl put svc/web/1 "10.0.0.5:8080" --lease=32695410dcc0ca06

OK

Inspect a lease and its keys:

etcdctl lease timetolive 32695410dcc0ca06 --keys

lease 32695410dcc0ca06 granted with TTL(60s), remaining(48s), attached keys([svc/web/1])

Keep it alive (runs until interrupted):

etcdctl lease keep-alive 32695410dcc0ca06

lease 32695410dcc0ca06 keepalived with TTL(60)
lease 32695410dcc0ca06 keepalived with TTL(60)
...

Note

The server grants a TTL greater than or equal to what you request and selects the final value. The granted lease ID is a hexadecimal number — pass it verbatim to --lease and the other lease commands.

4.5 Transactions (txn)

etcdctl txn [options]

A transaction reads a list of comparisons, a success block, and a failure block from standard input and applies them as a single atomic operation. If every comparison is true, the success block runs; otherwise the failure block runs. RPC: Txn.

Comparisons can test a key's create revision, mod revision, value, version, or attached lease, using <, =, or >.

Interactive example (compare-and-set):

etcdctl txn -i

compares:
mod("key1") > "0"

success requests (get, put, del):
put key1 "overwrote-key1"

failure requests (get, put, del):
put key1 "created-key1"

# SUCCESS

# OK

Non-interactive, via a here-string:

etcdctl txn <<<'mod("key1") > "0"

put key1 "overwrote-key1"

put key1 "created-key1"

'

Warning

Inside a txn, multi-line values must escape newlines as literal \n. Unlike put, the shell will not convert real newlines for you — a literal newline in a txn value causes a parse failure.

4.6 Cluster membership

List members as a table:

etcdctl -w table member list

+------------------+---------+--------+------------------------+------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+------------------+---------+--------+------------------------+------------------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:12380 | http://127.0.0.1:2379  |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
+------------------+---------+--------+------------------------+------------------------+

Add a member. etcd prints the environment variables to start the new member with:

etcdctl member add infra4 --peer-urls=https://10.0.0.13:2380

Member ced000fda4d05edf added to cluster 8c4281cc65c7b112

ETCD_NAME="infra4"
ETCD_INITIAL_CLUSTER="infra4=https://10.0.0.13:2380,infra1=http://10.0.0.30:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Caution

Adding or removing voting members changes the quorum size. Add members one at a time and wait for each to sync and report started before changing membership again. Removing too many members at once, or adding a member to a cluster that can't reach quorum, can make the cluster lose quorum and stop serving writes. Consider adding a learner (non-voting) member first and promoting it once it has caught up.

Transfer leadership away from the current leader (the request must be sent to the leader):

etcdctl --endpoints ${leader_ep} move-leader c89feb932daef420

Leadership transferred from 45ddc0e800e20b93 to c89feb932daef420

4.7 Endpoint inspection

The --cluster flag fetches the full member list and runs the check against every member:

etcdctl endpoint --cluster health

http://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.060091ms
http://127.0.0.1:22379 is healthy: successfully committed proposal: took = 903.138µs
http://127.0.0.1:32379 is healthy: successfully committed proposal: took = 1.113848ms

etcdctl -w table endpoint --cluster status

+------------------------+------------------+---------------+---------+-----------+-----------+------------+
|        ENDPOINT        |        ID        |    VERSION    | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------------+------------------+---------------+---------+-----------+-----------+------------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |   3.7.0       |   25 kB |     false |         2 |          8 |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.7.0       |   25 kB |      true |         2 |          8 |
+------------------------+------------------+---------------+---------+-----------+-----------+------------+

4.8 Maintenance commands

Alarms

etcd raises an alarm (for example NOSPACE when the backend exceeds --quota-backend-bytes) to protect the cluster, putting it into a limited/read-only mode. Inspect and clear alarms:

etcdctl alarm list

alarm:NOSPACE

etcdctl alarm disarm

alarm:NOSPACE

Important

A NOSPACE alarm means the database hit its quota and etcd has stopped accepting writes. Recovery is a sequence: compact old revisions, run defrag to reclaim file space, then alarm disarm. Disarming without freeing space will not restore write capacity.

Defragment (online)

etcdctl defrag [--cluster] [--endpoints=...]

Releases free space (left behind by deletes and compaction) in a running member's backend file back to the filesystem.

etcdctl defrag --cluster

Finished defragmenting etcd member[http://127.0.0.1:2379]
Finished defragmenting etcd member[http://127.0.0.1:22379]
Finished defragmenting etcd member[http://127.0.0.1:32379]

Warning

Defragmenting a live member blocks reads and writes on that member while it rebuilds its database. Defrag one member at a time, never all at once, and avoid the leader during peak load. To defragment an offline data directory instead, use etcdutl defrag (see §5). The defrag request is not replicated — it applies only to the targeted endpoints, so list every member (or use --cluster).

Snapshots (backups)

etcdctl snapshot save <filename>

Writes a point-in-time snapshot of the backend database to a file:

etcdctl snapshot save backup-$(date +%F).db

Snapshot saved at backup-2026-06-04.db

Important

In v3.6 and later, snapshot restore and snapshot status were removed from etcdctl and moved to etcdutl. Use etcdutl snapshot restore / etcdutl snapshot status (see §5). etcdctl snapshot save remains, because it talks to a live server.

Downgrade

downgrade lets an administrator force a cluster's version down a minor release — an experimental v3.6+ feature, not recommended for production. It runs in stages:

etcdctl downgrade validate 3.6

Downgrade validate success, cluster version 3.7

etcdctl downgrade enable 3.6

Downgrade enable success, cluster version 3.7

After enabling, replace each member's binary with the older version one at a time, then confirm. etcdctl downgrade cancel aborts an in-progress downgrade.

Caution

Downgrade is experimental. Validate first, downgrade members one at a time, and watch the leader log for the cluster has been downgraded. Take a snapshot before you begin.

4.9 Concurrency commands

lock — distributed mutex

etcdctl lock [--ttl <seconds>] <lockname> [command arg1 ...]

Acquires a named distributed lock; holds it until etcdctl exits. If a command is given, it runs while the lock is held (with ETCD_LOCK_KEY and ETCD_LOCK_REV set in its environment).

etcdctl lock mylock echo "lock acquired"

lock acquired

Warning

If etcdctl lock is killed abnormally and can't release the lock, the lock stays held until its lease expires — by default up to 60 seconds. Plan for that delay in critical sections.

elect — leader election

etcdctl elect [--listen] <election-name> [proposal]

A candidate announces itself with a proposal value; whoever wins has their proposal published as the election's leader value. --listen observes without participating.

etcdctl elect myservice node-1

myservice/1456952310051373265
node-1

4.10 Authentication commands

etcd supports role-based access control. The model: create users, create roles, grant key/prefix permissions to roles, grant roles to users, then enable auth. A root user with the root role is required before enabling.

Bootstrap and enable auth:

etcdctl user add root
# Password of root: ********
etcdctl role add root
etcdctl user grant-role root root
etcdctl auth enable

Authentication Enabled

Create an app role limited to one prefix, and a user for it:

etcdctl --user=root:rootpw role add app
etcdctl --user=root:rootpw role grant-permission --prefix app readwrite app/
etcdctl --user=root:rootpw user add appuser
etcdctl --user=root:rootpw user grant-role appuser app

Role app created
Role app updated
User appuser created
Role app is granted to user appuser

Caution

Once auth enable succeeds, every request must authenticate, including administrative ones. Make sure the root user exists and you know its password before enabling — there is no back door. Lose the root password and you must recover from a snapshot.

4.11 Mirroring

etcdctl make-mirror [options] <destination>

Continuously mirrors a key prefix from the current cluster to a destination cluster. Options include --prefix, --dest-prefix, --no-dest-prefix, the destination TLS flags (--dest-cacert, --dest-cert, --dest-key), --dest-insecure-transport, and --max-txn-ops. It prints a running count of keys transferred (updated about every 30 seconds).

etcdctl make-mirror --prefix app/ mirror.example.com:2379

10
18

4.12 Health and performance checks

etcdctl check perf runs a 60-second write workload and reports whether the cluster meets throughput and latency targets. etcdctl check datascale estimates memory use for a workload.

etcdctl check perf --load=s

60 / 60 ... 100.00% 1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.087509s
PASS: Stddev is 0.011084s
PASS

Both checks accept --prefix, --auto-compact, and --auto-defrag to clean up the keys they create.

Note

check perf passes or fails relative to the workload model you pick. A single laptop node will pass s and fail l — that's expected. Match the load to your hardware and cluster size.

4.13 Output formats

Every command that maps to an RPC respects -w/--write-out:

• simple (default) — human-readable, customized per command.
• json — the JSON encoding of the RPC response. Keys and values are base64-encoded because etcd stores raw bytes.
• protobuf — the raw protobuf-encoded response.
• fields — a coreutils-friendly "Field" : value line format.
• table — an ASCII table (used heavily by member list and endpoint status).

On failure, every command writes an error to standard error and returns a non-zero exit code regardless of format.

5. The etcdutl offline tool

etcdutl operates directly on data-dir files and snapshots — no running server. Use it for restores, offline defrag, schema migration, and integrity hashing.

Inspect a snapshot

etcdutl snapshot status backup-2026-06-04.db -w table

+----------+----------+------------+------------+---------+
|   hash   | revision | total keys | total size | version |
+----------+----------+------------+------------+---------+
| 84e9a5bf |     1234 |        842 |      25 kB |   3.7.0 |
+----------+----------+------------+------------+---------+

Restore a snapshot into a new data directory

etcdutl snapshot restore <file> --data-dir <dir> [options]

etcdutl snapshot restore backup-2026-06-04.db \
  --name infra1 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --data-dir /var/lib/etcd-restored

Caution

Restore each member of the new cluster from the same snapshot, giving each its own --name, --data-dir, and --initial-advertise-peer-urls but the same --initial-cluster. Then start all members with --initial-cluster-state new. Restoring into a non-empty --data-dir, or restoring members from different snapshots, will fail or produce a divergent cluster.

Offline defragment

etcdutl defrag --data-dir /var/lib/etcd

finished defragmenting the storage

Migrate the data-dir schema

etcdutl migrate --data-dir /var/lib/etcd --target-version 3.6

The minimum supported --target-version is 3.5. --force overrides the storage version even if normal migration fails (not recommended).

6. Common Tasks (recipes)

Bootstrap a 3-node cluster (static)

Note

This shows the shape of the command. The full flag reference and TLS hardening are in the Admin Guide.

# on infra1 (repeat with the right name/IP on infra2 and infra3)
etcd --name infra1 \
  --data-dir /var/lib/etcd \
  --listen-client-urls https://10.0.0.10:2379 \
  --advertise-client-urls https://10.0.0.10:2379 \
  --listen-peer-urls https://10.0.0.10:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-cluster-token my-etcd-token \
  --initial-cluster-state new

Verify:

etcdctl --endpoints=https://10.0.0.10:2379,https://10.0.0.11:2379,https://10.0.0.12:2379 endpoint health

Take and verify a backup

etcdctl snapshot save /backups/etcd-$(date +%F).db
etcdutl snapshot status /backups/etcd-$(date +%F).db -w table

Watch a config prefix and reload an app

etcdctl watch --prefix /myapp/config/ -- systemctl reload myapp

Register a service with a TTL and keep it alive

LEASE=$(etcdctl lease grant 30 | awk '{print $2}')
etcdctl put /svc/web/$(hostname) "$(hostname -i):8080" --lease=$LEASE
etcdctl lease keep-alive $LEASE   # keep running for the lifetime of the process

Atomic "create if not exists"

etcdctl txn <<<'create("/locks/job") = "0"

put /locks/job "owner-A"

get /locks/job

'

If the key doesn't exist yet (create revision is 0), the success branch claims it; otherwise the failure branch reads the current owner.

Reclaim disk after heavy churn

REV=$(etcdctl endpoint status -w json | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
etcdctl compaction "$REV"
etcdctl defrag --cluster

Rotate a member out and a new one in

# add the replacement (one at a time!)
etcdctl member add infra4 --peer-urls=https://10.0.0.13:2380
# start infra4 with the printed ETCD_* variables and --initial-cluster-state existing
etcdctl member list                       # confirm infra4 is started
etcdctl member remove <old-member-id>      # only after infra4 is healthy

7. Troubleshooting

Each entry lists the exact message, why it happens, how to fix it, and how to prevent it.

Error: etcdserver: request timed out

Cause: The cluster could not commit the request within the timeout — usually slow disk (fsync latency), an overloaded leader, or a network partition affecting quorum.

Solution:

1. Check member health: etcdctl endpoint --cluster health.
2. Check disk and CPU on each member; etcd needs fast fsync.
3. Look for etcdserver: leader changed or etcdserver: no leader in the logs.

Prevention: Use SSDs, isolate etcd from noisy neighbors, and keep --election-timeout ≥ 5× --heartbeat-interval to tolerate latency.

Error: etcdserver: no leader

Cause: The cluster has lost quorum (a majority of members are unreachable) and cannot elect a leader.

Solution:

1. Run etcdctl endpoint --cluster status against any reachable endpoint to see who is up.
2. Restart failed members; restore network connectivity.
3. If members were permanently lost and quorum cannot be regained, restore from a snapshot with etcdutl snapshot restore.

Prevention: Run an odd number of members (3 or 5) across failure domains so a single failure never breaks quorum.

Error: etcdserver: mvcc: database space exceeded / alarm NOSPACE

Cause: The backend database reached --quota-backend-bytes; etcd entered read-only mode and raised a NOSPACE alarm.

Solution:

1. Compact old history: etcdctl compaction <recent-revision>.
2. Reclaim space: etcdctl defrag --cluster.
3. Clear the alarm: etcdctl alarm disarm.

Prevention: Enable auto-compaction (--auto-compaction-mode/--auto-compaction-retention) and monitor etcd_mvcc_db_total_size_in_bytes.

Error: mvcc: required revision has been compacted

Cause: You requested a get --rev or watch --rev for a revision older than the last compaction.

Solution: Re-issue the request at a revision at or after the current compacted revision, or watch from the current revision (omit --rev).

Prevention: Don't compact history you still need to read or replay; track the compacted revision returned by compaction.

Error: etcdserver: request is too large

Cause: The request exceeded --max-request-bytes (default ~1.5 MiB) or --max-txn-ops (default 128).

Solution: Split the work into smaller writes or a smaller transaction; raise the server limits only with care (large requests increase latency and memory use).

Prevention: Keep individual values and transactions small. etcd is a coordination store, not a blob store.

Error: etcdserver: too many requests

Cause: The server is overloaded or applying a backlog (often during/after a defrag, snapshot, or leader change).

Solution: Back off and retry with jitter; reduce client concurrency.

Prevention: Rate-limit clients; avoid defragmenting under peak load.

Error: auth: authentication failed, invalid user ID or password

Cause: Wrong username/password, or auth is enabled and the request didn't supply credentials.

Solution: Re-check --user=user:password (or --user + --password). Confirm the user exists: etcdctl --user=root:... user list.

Prevention: Store credentials in ETCDCTL_* environment variables or a secrets manager; verify with etcdctl --user=... auth status.

Error: auth: permission denied

Cause: The authenticated user's roles don't grant the needed access to that key/range.

Solution: Grant the role the right permission, e.g. role grant-permission --prefix myrole readwrite app/.

Prevention: Design roles around key prefixes; use user get <name> --detail to audit effective permissions.

Error: etcdserver: can only promote a learner member which is in sync with the leader

Cause: You tried to promote a learner that hasn't caught up to the leader's log yet.

Solution: Wait until the learner's raft index matches (check endpoint status), then retry the promotion.

Prevention: Always confirm a learner is in sync before promoting it. (Related community report: issue #20757 — promoting a member while auth is enabled.)

Error: etcdctl: no available endpoints / etcdclient: no available endpoints

Cause: None of the configured --endpoints were reachable.

Solution: Verify the addresses and ports, network/firewall, and TLS settings; test with etcdctl endpoint health.

Prevention: List multiple endpoints so a single down member doesn't break the client.

Error: Error: context deadline exceeded

Cause: A command exceeded --command-timeout or --dial-timeout.

Solution: Increase the timeout (--command-timeout=15s) or investigate why the cluster is slow.

Prevention: Set realistic timeouts for cross-region clusters.

Error: etcdserver: mvcc: required revision is a future revision

Cause: You asked for a revision (e.g. in etcdutl hashkv --rev) higher than any that exists.

Solution: Use a revision ≤ the latest; omit --rev to use the latest.

Prevention: Read the current revision from endpoint status first.

Error: snapshot missing hash but --skip-hash-check=false

Cause: Restoring a snapshot that was copied directly from a data directory (so it has no integrity hash).

Solution: Re-run etcdutl snapshot restore with --skip-hash-check.

Prevention: Prefer etcdctl snapshot save (which writes a hash) over copying the DB file.

Error: serving client traffic insecurely; this is strongly discouraged!

Cause: Not an error — a warning that the server is running without client TLS.

Solution: Configure --cert-file/--key-file (and a CA for client-cert auth). See the Admin Guide.

Prevention: Always enable TLS for non-local clusters.

Error: etcdmain: error verifying flags, '...' is not a valid flag

Cause: A typo'd or removed flag on the etcd command line.

Solution: Run etcd --help and check the flag name; some flags (e.g. --logger=capnslog) were removed in v3.5.

Prevention: Pin your launch flags and review them against the Admin Guide's reference when upgrading.

Tip

Add --debug to any etcdctl command to dump the effective configuration and gRPC-level logs to stderr — invaluable when diagnosing TLS or connectivity problems.

8. FAQ

Q: What ports does etcd use?

By convention, 2379 for client (gRPC/HTTP) traffic and 2380 for peer (Raft) traffic. These come from the default --listen-client-urls/--listen-peer-urls and can be changed.

Q: How many members should my cluster have?

An odd number — 3 or 5. A 3-member cluster tolerates 1 failure; a 5-member cluster tolerates 2. Even-numbered clusters give no extra fault tolerance and just raise the quorum size.

Q: What's the difference between etcdctl and etcdutl?

etcdctl talks to a running cluster over the network (put/get, watch, membership, auth, online defrag, snapshot save). etcdutl works offline on data-dir files and snapshots (restore, offline defrag, migrate, hashkv). Snapshot restore/status moved from etcdctl to etcdutl in v3.6.

Q: Why doesn't deleting keys free disk space?

etcd uses MVCC: deletes and overwrites create new revisions but keep the old ones until you compact. Even after compaction, the backend file doesn't shrink — its freed pages go on an internal free list. Run defrag to return that space to the filesystem.

Q: What is a lease and when should I use one?

A lease is a TTL object; keys attached to it are deleted when it expires. Use leases for ephemeral data — service registrations, session keys, and the locks/elections behind lock and elect. Keep a lease alive with lease keep-alive.

Q: How do I do a compare-and-swap?

Use a txn with a comparison on the key's mod, create, value, or version. The success block runs only if the comparison holds. See §4.5.

Q: What does --consistency=s (serializable) change?

A serializable read is served locally without a quorum round trip — faster, but may return slightly stale data. The default is linearizable (l), which always reflects the latest committed write.

Q: Can I get JSON output?

Yes: -w json. Note that keys and values are base64-encoded in JSON because etcd stores raw bytes. -w table and -w fields are also available.

Q: How do I back up and restore?

Back up with etcdctl snapshot save file.db. Restore by running etcdutl snapshot restore file.db --data-dir ... on each new member (same snapshot, per-member name/peer URLs, shared --initial-cluster), then start with --initial-cluster-state new.

Q: How do I enable authentication safely?

Create the root user and root role, grant the role to the user, then auth enable. After that every request must authenticate — there's no bypass, so confirm the root credentials first.

Q: What is a learner member?

A non-voting member that replicates the log but doesn't count toward quorum. Add a learner, let it catch up, then promote it to a voting member. This makes adding members safer because it doesn't change quorum until promotion. The maximum number of learners is set by the server's --max-learners.

Q: How do I move the leader off a node before maintenance?

etcdctl move-leader <transferee-id>, sent to the current leader. Useful before restarting or defragmenting the leader.

Q: My value starts with a dash and put treats it as a flag — what do I do?

Use -- to end flag parsing: etcdctl put -- mykey -my-value.

Q: How do I watch from the beginning of history?

etcdctl watch <key> --rev=1 replays from the first revision (provided it hasn't been compacted away), then continues live.

Q: Where is the full server configuration reference?

In the etcd Administrator & Developer Guide — it documents every etcd flag, environment variable, TLS option, tuning parameter, and the cluster architecture.