# Elasticsearch Administrator & Developer Guide

This guide is the complete operational, architectural, and developer reference for **Elasticsearch 9.5.0**. It is written for engineers who deploy, operate, secure, tune, and extend Elasticsearch in production — from first cluster bring-up through large-scale multi-region deployments. It tracks source commit `7ccc603b` (2026-04-24) on the `main` branch; release branches present in this source tree are `9.4`, `9.3`, `9.2`, and `8.19`.

> [!NOTE]
> Every configuration setting, command, flag, and API path in this document has been derived from the Elasticsearch source tree at `7ccc603b`. No values are fabricated. Where behavior changed between major versions, the relevant version is noted.

---

## Table of Contents

1. [Architecture Overview](#1-architecture-overview)
2. [System Requirements](#2-system-requirements)
3. [Configuration Reference](#3-configuration-reference)
4. [Deployment & Scaling](#4-deployment--scaling)
5. [Operations](#5-operations)
6. [Performance](#6-performance)
7. [Security](#7-security)
8. [Developer Internals](#8-developer-internals)
9. [Integration Points](#9-integration-points)

---

## 1. Architecture Overview

### 1.1 What Elasticsearch Is

Elasticsearch is a distributed document store and search engine built on **Apache Lucene**. Documents are JSON objects stored in **indices**. Each index is horizontally sharded across cluster nodes for scale, and each shard is replicated for fault tolerance. All reads, writes, and administration go through a single HTTP API on port **9200** (TLS by default since 8.0).

Key characteristics:

- **Eventually consistent search, strongly consistent writes.** Writes succeed after being applied on the primary shard plus the configured replica count. Search reads from the last refreshed segment (default 1 second behind writes).
- **No external coordination service.** No ZooKeeper, no etcd, no Consul. Master election and cluster-state replication are handled by the in-tree coordinator (`server/src/main/java/org/elasticsearch/cluster/coordination/`), which uses a Raft-inspired protocol.
- **Security on by default.** Since 8.0, every fresh single-node and multi-node start auto-generates a CA, node certificates, and a random password for the `elastic` superuser.
- **Files are append-only Lucene segments.** Updates are copy-on-write; deletes are logical (tombstones) until a background merge rewrites the segment. Never back up a live cluster by copying shard directories.
- **Plugin-based extension model.** Every non-core capability — security, ML, SQL, ES|QL, watcher, CCR — is a plugin at the Java level. The boundary between `server/` and `x-pack/` is a packaging and licensing distinction, not a runtime one.

### 1.2 Source Tree Layout

The repository (`elastic/elasticsearch`) is a Gradle monorepo of **40,369 files** and **68.6 million characters**. Top-level areas:

| Directory | Description |
|-----------|-------------|
| `server/` | Core engine: indexing, search, cluster coordination, transport, REST dispatch, settings, aggregations, scripting runtime |
| `x-pack/` | Commercial features under the Elastic License: security, ML, SQL, ES\|QL, EQL, monitoring, watcher, CCR, searchable snapshots, LogsDB, profiling, Prometheus metrics, spatial, vector tile, identity provider, inference |
| `modules/` | Bundled open-source extensions always included in the distribution: analyzers, ingest processors, Painless, geo, reindex, runtime fields, repository-azure/gcs, parent-join |
| `plugins/` | Optional installable plugins: language analyzers (ICU, Kuromoji, SmartCN, Nori, Phonetic, Stempel), discovery-gce, mapper-murmur3, store-smb |
| `libs/` | Shared internal libraries used by server and plugins: core, ssl-config, native (JNI), entitlement, geo, h3, tdigest, lz4, grok, simdvec, log4j, plugin-api |
| `distribution/` | Packaging: Docker entrypoint, deb/rpm/zip/tar, server-cli, JVM tools |
| `rest-api-spec/` | 600 JSON endpoint specification files — machine-readable ground truth for all REST APIs |
| `build-tools/` / `build-tools-internal/` | Internal Gradle plugins (not shipped) |
| `benchmarks/` | JMH micro-benchmarks |
| `test/` | Test framework and fixtures |
| `qa/` | Integration test suites: evil-tests, smoke-tests, rolling-upgrade, full-cluster-restart, multi-version |

### 1.3 Process Model

Each Elasticsearch node is a single **JVM process**. The startup chain from `distribution/tools/server-cli/` is:

1. Locate JDK (bundled at `jdk/` unless `ES_JAVA_HOME` is set).
2. Merge JVM options from `config/jvm.options`, files under `config/jvm.options.d/`, and automatic ergonomics (heap sizing from container memory limits if not explicitly set).
3. Launch `org.elasticsearch.bootstrap.Elasticsearch`, which runs **bootstrap checks**, binds network ports, and starts subsystems.
4. If `telemetry.tracing.enabled: true`, attach the APM instrumentation agent.

Once running, a node hosts these concurrent subsystems in-process:

| Subsystem | Transport | Notes |
|-----------|-----------|-------|
| **HTTP layer** | Netty HTTP/1.1 + HTTP/2 on `9200` | Client REST traffic |
| **Transport layer** | Netty TCP on `9300` | Inter-node RPC: cluster-state, search fan-out, index operations |
| **Indexing engine** | One Lucene `IndexWriter` per primary shard | Translog-backed for durability |
| **Cluster coordinator** | In-process | Master election, state publication, membership |
| **Shard allocator** | In-process | Placement decisions, rebalancing |
| **Scheduler** | In-process | ILM, SLM, transforms, watcher, rollup |
| **Plugin hosts** | In-process | Security, ML, SQL, ES\|QL, fleet, etc. |

### 1.4 Node Roles

A node's responsibilities are declared by the `node.roles` list in `elasticsearch.yml`. A node may hold multiple roles.

| Role | Description | Production guidance |
|------|-------------|---------------------|
| `master` | Eligible to be elected cluster master | Require exactly 3 in production for HA quorum |
| `data` | Generic data tier (deprecated as sole role) | Prefer explicit tier roles below |
| `data_content` | Non-time-series data (catalogs, user data) | Standard default for regular indices |
| `data_hot` | Newest, most-written time-series data | Fastest NVMe disks; highest IOPS |
| `data_warm` | Recent, less-frequently-written data | Cheaper spinning/SATA SSD acceptable |
| `data_cold` | Rarely queried historical data | Often backed by searchable snapshots |
| `data_frozen` | Extremely cold, query-on-demand only | Searchable snapshots only; near-zero heap overhead |
| `ingest` | Runs ingest pipelines on incoming documents | Co-locate with data_hot unless heavy enrichment |
| `ml` | Machine learning jobs, anomaly detection, NLP inference | Requires extra heap; X-Pack only |
| `transform` | Continuous and pivot transforms | Usually co-located with data nodes in small clusters |
| `remote_cluster_client` | Can connect to and search remote clusters | Required on nodes that issue CCS queries |
| `voting_only` | Participates in master election but cannot be elected | Efficient quorum padding with minimal resources |

An empty `node.roles: []` declares a **coordinating-only node**: routes client traffic and reduces search results but holds no shards and is never master-eligible.

**Recommended layout for a production hot/warm cluster:**

```
3 × dedicated-master  node.roles: [master]
N × hot-data          node.roles: [data_hot, data_content, ingest, transform]
M × warm/cold-data    node.roles: [data_warm, data_cold]
2 × coordinating      node.roles: []                        (optional)
K × ML                node.roles: [ml, remote_cluster_client]  (optional)
```

### 1.5 Cluster State and Coordination

The **cluster state** is a single versioned document held in memory on every node and persisted to disk by the elected master. It contains:

- Cluster metadata (name, UUID)
- Index metadata (mappings, settings, aliases, data stream definitions)
- Routing table (which shard copy lives on which node, shard states)
- Node membership list
- Ingest pipeline definitions
- Index templates and component templates
- ILM policies, SLM policies, transform configs, ML configs
- Security role mappings

#### Master Election

Masters elect via the **coordinator** (`server/src/main/java/org/elasticsearch/cluster/coordination/`):

- **Terms** — monotonically increasing integers. Every new election bumps the term.
- **Voting configuration** — set of master-eligible nodes whose majority constitutes a quorum. Evolves with the cluster state as nodes join or leave.
- **Publication** — two-phase commit: send diff to followers → wait for acknowledgment from quorum → commit. A committed state is durable before the commit returns.
- **Follower check** — elected master pings all followers periodically. Unreachable followers are removed.
- **Leader check** — each follower pings the master. If the master is unreachable, followers trigger re-election.
- **Bootstrap** — `cluster.initial_master_nodes` is consumed only when nodes boot with no existing cluster state. Remove it from all configs after the cluster first forms.

> [!IMPORTANT]
> `cluster.initial_master_nodes` must be set **only once**, at initial cluster formation, and must name only the nodes physically present at that moment. After the cluster forms, remove this setting from every node's `elasticsearch.yml`. If left in place it can cause a split-brain when those named nodes are later replaced.

#### Shard Allocation

The allocator runs as a stack of `AllocationDecider` implementations followed by a balancer:

1. **Hard constraints** — `index.routing.allocation.require.*`, `include.*`, `exclude.*`, node role compatibility, version compatibility, data-tier filters.
2. **Disk watermarks** — stop placing new shards at `low` (85%), begin migrating existing shards at `high` (90%), block writes at `flood_stage` (95%).
3. **Zone awareness** — `cluster.routing.allocation.awareness.attributes: zone` spreads replicas across declared failure domains.
4. **Balance scoring** — `BalancedShardsAllocator` equalizes weighted shard count, write load, and disk usage across nodes.
5. **Desired balance** (9.x) — asynchronous desired-balance computation decouples re-planning from actual shard moves, improving responsiveness under load.

Diagnose any unassigned shard:

```http
GET /_cluster/allocation/explain
```

### 1.6 Storage: Indices, Shards, and Segments

An **index** is a logical namespace split into one or more **primary shards**. Each primary is a self-contained Lucene index. Writes always go to the primary first, then propagate to replicas. Reads can be served by any live copy.

#### Lucene Segment Lifecycle

1. Documents are written to an in-memory indexing buffer and simultaneously to the **translog** (write-ahead log on disk).
2. At **refresh** (`index.refresh_interval`, default `1s`), the buffer flushes to a new on-disk **segment** and becomes searchable.
3. Segments are **immutable**. Updates write a new version; the old version is tombstoned. Deletes mark tombstones.
4. Background **merges** consolidate small segments into larger ones, physically removing tombstoned documents.
5. At **flush**, Lucene commits and the translog is truncated.

#### Shard Count Trade-offs

| Factor | Fewer large shards | More smaller shards |
|--------|--------------------|---------------------|
| Heap usage per node | Lower | Higher |
| Intra-query parallelism | Lower | Higher |
| Small-query fan-out overhead | Lower | Higher |
| Recovery time | Slower per shard | Faster per shard |
| Rebalance granularity | Coarse | Fine |

**Target shard size: 10–50 GB** of post-merge on-disk data. Query-heavy catalogs: lower end. Bulk log archives: upper end.

**Maximum shards per node:** aim for **≤ 20 × heap_GB**. Hard ceiling ~1,000–2,000 shards per node before GC and state-management pressure becomes significant.

### 1.7 Module and Plugin Inventory

#### Core Server Packages (key areas)

| Package path | Contents |
|---|---|
| `server/src/main/java/org/elasticsearch/common/settings/` | 24 files, 405K chars: `Setting`, `Settings`, `AbstractScopedSettings`, `KeyStoreWrapper`, `ClusterSettings`, `IndexScopedSettings` |
| `server/src/main/java/org/elasticsearch/search/internal/` | 20 files, 167K chars: `ShardSearchRequest`, `SearchContext`, `ContextIndexSearcher` |
| `server/src/main/java/org/elasticsearch/search/aggregations/pipeline/` | 39 files, 74K chars: all pipeline aggregation implementations |
| `server/src/main/java/org/elasticsearch/injection/guice/internal/` | 25 files, 91K chars: internal Guice DI for plugin wiring |
| `server/src/main/java/org/elasticsearch/common/cli/` | `EnvironmentAwareCommand`, `KeyStoreAwareCommand` — base classes for all bin/ tools |
| `server/src/main/java/org/elasticsearch/plugins/internal/` | 12 files: internal plugin system APIs |

#### X-Pack Plugins (commercial, Elastic License)

| Plugin | Purpose |
|--------|---------|
| `x-pack/plugin/security/` | Authentication (native, LDAP, AD, PKI, SAML, OIDC, JWT, Kerberos), authorization (roles, FLS/DLS), TLS auto-config, audit logging, API keys, service accounts |
| `x-pack/plugin/ml/` | Anomaly detection, data frame analytics, trained models (PyTorch NLP), inference tasks |
| `x-pack/plugin/esql/` | ES\|QL query language engine — piped, columnar query processing |
| `x-pack/plugin/sql/` | SQL query layer with JDBC and ODBC drivers |
| `x-pack/plugin/eql/` | Event Query Language for sequence-based threat detection |
| `x-pack/plugin/inference/` | ML inference service: 13 service implementations, pluggable inference backends |
| `x-pack/plugin/transform/` | Pivot and continuous data transforms |
| `x-pack/plugin/watcher/` | Alerting/Watcher: scheduled watches, conditions, actions (email, Slack, PagerDuty, webhook, index) |
| `x-pack/plugin/ccr/` | Cross-cluster replication: leader/follower index streaming |
| `x-pack/plugin/rollup/` | Time-series rollup jobs (legacy; prefer downsampling for new deployments) |
| `x-pack/plugin/blob-cache/` | 5 files, 56K chars: blob cache for searchable snapshots |
| `x-pack/plugin/searchable-snapshots/` | Searchable snapshots: cold/frozen indices served from snapshot repositories |
| `x-pack/plugin/logsdb/` | LogsDB index mode — optimized mapping and codec for log workloads |
| `x-pack/plugin/ent-search/` | Enterprise Search features |
| `x-pack/plugin/profiling/` | Universal Profiling: continuous profiling data ingestion and flamegraph rendering |
| `x-pack/plugin/prometheus/` | Prometheus-compatible metrics endpoint |
| `x-pack/plugin/spatial/` | Spatial/GIS: shape queries, geo_shape, cartesian fields |
| `x-pack/plugin/vector-tile/` | Vector tile search for map rendering |
| `x-pack/plugin/identity-provider/` | SAML identity provider (Elasticsearch as IdP) |
| `x-pack/plugin/core/` | Shared X-Pack core: scheduler, ML job config, security core types |

#### Bundled Modules

| Module | Contents |
|--------|---------|
| `modules/ingest-common/` | 45 ingest processors: append, bytes, community_id, convert, csv, date, date_index_name, dissect, dot_expander, drop, enrich, fail, fingerprint, foreach, grok, gsub, html_strip, join, json, kv, lowercase, network_direction, pipeline, registered_domain, remove, rename, reroute, script, set, set_security_user, sort, split, trim, uppercase, uri_parts, urldecode, user_agent |
| `modules/ingest-geoip/` | GeoIP processor (MaxMind GeoLite2 and GeoIP2 databases) |
| `modules/lang-painless/` | Painless scripting language (7 API files; used in scripted fields, ingest, transforms, watcher) |
| `modules/analysis-common/` | Text analysis: tokenizers (standard, letter, whitespace, UAX URL email, path hierarchy, pattern), token filters (stop, stemmer, synonym, ASCII folding, n-gram, edge n-gram, shingle, etc.), char filters |
| `modules/rank-eval/` | Ranking evaluation API for offline relevance measurement |
| `modules/repository-azure/` | Azure Blob Storage snapshot repository |
| `modules/repository-gcs/` | Google Cloud Storage snapshot repository |
| `modules/legacy-geo/` | Legacy geo_shape support (pre-BKD shapes) |
| `modules/runtime-fields-common/` | Runtime fields evaluation engine |
| `modules/parent-join/` | Parent-child document joins via `join` field type |
| `modules/reindex/` | `_reindex`, `_update_by_query`, `_delete_by_query` with throttling and remote reindex |
| `modules/mapper-extras/` | Additional field types: `rank_feature`, `rank_features`, `scaled_float`, `token_count`, `annotated_text` |
| `modules/percolator/` | Percolator queries (store queries, match against documents) |
| `modules/data-streams/` | Data stream lifecycle management |
| `modules/apm/` | APM / Application Performance Monitoring integration |

#### Optional Plugins

| Plugin | Purpose |
|--------|---------|
| `plugins/analysis-icu/` | Unicode text analysis using ICU4J |
| `plugins/analysis-kuromoji/` | Japanese morphological analysis |
| `plugins/analysis-smartcn/` | Smart Chinese analyzer |
| `plugins/analysis-nori/` | Korean analysis |
| `plugins/analysis-phonetic/` | Phonetic token filters (Soundex, Metaphone, Caverphone, etc.) |
| `plugins/analysis-stempel/` | Polish analysis (Stempel stemmer) |
| `plugins/discovery-gce/` | Google Compute Engine instance discovery |
| `plugins/mapper-murmur3/` | Murmur3 hash field type |
| `plugins/store-smb/` | SMB/Windows share as data path (Windows only) |
| `plugins/examples/` | Plugin development examples: custom-settings, rest-handler, security-authorization-engine, painless-whitelist, rescore |

#### Shared Libraries (`libs/`)

| Library | Purpose |
|---------|---------|
| `libs/core/` | Core Java utilities: Strings, Numbers, DateMath |
| `libs/ssl-config/` | SSL/TLS configuration primitives |
| `libs/native/` | 10 files, 25K chars: JNI wrappers for OS-level operations (mlockall, prctl, syscall) |
| `libs/entitlement/` | 15 config files, 95K chars: runtime security capability system (replaces SecurityManager in 9.x) |
| `libs/geo/` | 10 files, 101K chars: geometry utilities (GeoUtils, SpatialUtils, GeometryParser) |
| `libs/h3/` | H3 hexagonal grid library for spatial indexing |
| `libs/tdigest/` | T-Digest quantile estimation for percentile aggregations |
| `libs/lz4/` | LZ4 fast compression (translog and network) |
| `libs/grok/` | Grok pattern matching (used by ingest-common grok processor) |
| `libs/plugin-api/` | 6 files: public Plugin settings API for third-party plugins |
| `libs/simdvec/` | SIMD-accelerated vector operations (for kNN dense vectors) |
| `libs/user-agent-api/` | User agent parsing utilities |
| `libs/log4j/` | Log4j2 wrapper and configuration |
| `libs/server-launcher-common/` | 2 files, 12K chars: shared launcher utilities |

### 1.8 Build System

Elasticsearch uses **Gradle** with internal build plugins under `build-tools-internal/`:

| Gradle plugin | Purpose |
|---------------|---------|
| `elasticsearch.java` | Standard Java compilation settings, formatting, checkstyle |
| `elasticsearch.build` | Top-level build lifecycle |
| `elasticsearch.publish` | Maven Central publishing via `nmcp` aggregation |
| `elasticsearch.docker-support` | Multi-arch Docker image builds |
| `elasticsearch.jdk-download` | Pins and downloads the bundled JDK |
| `elasticsearch.global-build-info` | Exposes version, git hash, build date |
| `elasticsearch.runtime-jdk-provision` | Provisions JDK for test clusters |
| `elasticsearch.fips` | FIPS 140-2 mode build |
| `elasticsearch.internal-testclusters` | In-process test clusters for integration tests |
| `elasticsearch.run` | Launches a local dev cluster with `./gradlew run` |
| `elasticsearch.run-ccs` | Launches a two-cluster CCS dev setup |
| `elasticsearch.release-tools` | Handles release version manipulation and changelogs |

---

## 2. System Requirements

### 2.1 Minimum Requirements (Development)

| Resource | Minimum |
|----------|---------|
| CPU architecture | x86_64 or aarch64 |
| CPU cores | 2 |
| RAM | 4 GB (2 GB heap + page cache) |
| Disk | 10 GB, local filesystem |
| JVM | Bundled OpenJDK (included in tarball/Docker image) |
| OS | Linux kernel ≥ 4.4, macOS 12+, Windows Server 2019+ |

### 2.2 Production Per-Node Baseline

| Resource | Recommended |
|----------|-------------|
| CPU | 8–32 cores (data nodes); 4–8 (master nodes) |
| RAM | 32–64 GB per data node; 8–16 GB per master node |
| JVM heap | 50% of RAM, **maximum ~30.5 GB** (compressed oops boundary) |
| Disk (hot) | NVMe SSD; 1.5–2× raw data size accounting for replicas |
| Disk (warm/cold) | SATA SSD acceptable; same sizing multiplier |
| Network | 10 Gb/s for > 1 TB/day ingest or large multi-node clusters |

### 2.3 Required Kernel Parameters

```bash
# /etc/sysctl.d/99-elasticsearch.conf
vm.max_map_count=262144
vm.swappiness=1
```

Apply without reboot: `sysctl -w vm.max_map_count=262144`

### 2.4 Required System Limits

In `/etc/security/limits.conf` or systemd unit override:

```
elasticsearch  -  nofile   65535
elasticsearch  -  nproc    4096
elasticsearch  -  memlock  unlimited
```

systemd:

```ini
[Service]
LimitNOFILE=65535
LimitNPROC=4096
LimitMEMLOCK=infinity
```

### 2.5 Heap Sizing

> [!WARNING]
> Never set JVM heap above **~30.5 GB**. Above this threshold the JVM switches from 4-byte compressed ordinary object pointers to 8-byte full pointers, effectively wasting 30–40% of heap. Add nodes rather than growing heap past this boundary. Set `-Xms` and `-Xmx` to the **same value** to prevent runtime resizing pauses.

```
# config/jvm.options.d/heap.options
-Xms30g
-Xmx30g
```

Or via environment: `ES_JAVA_OPTS="-Xms16g -Xmx16g"`

The remaining RAM (the ~50% not allocated to heap) is reserved for the OS **filesystem page cache** backing Lucene's memory-mapped segment files.

### 2.6 Swap

Disable completely:

```bash
swapoff -a
# Remove swap entries from /etc/fstab for persistence
```

Set `bootstrap.memory_lock: true` in `elasticsearch.yml`. The node will refuse to start if the lock fails — fix the ulimit, not the config.

### 2.7 Storage Recommendations

- **Local NVMe** for `data_hot`. Network-attached storage introduces latency spikes that impact GC and write durability.
- **XFS or ext4** with `noatime` mount option.
- Avoid BTRFS (copy-on-write semantics conflict with Lucene's update model).
- `path.data` directory must be owned by the Elasticsearch OS user with `0700` permissions.

### 2.8 Docker Constraints

- Set a container memory limit; heap ergonomics reads `cgroup` memory limits.
- `--ulimit memlock=-1:-1` and `--ulimit nofile=65535:65535`
- Set `vm.max_map_count=262144` on the **host** — cannot be set inside the container.
- Use a named volume or `hostPath` PVC for `path.data`.

---

## 3. Configuration Reference

Elasticsearch has two configuration surfaces:

1. **`elasticsearch.yml`** — static, per-node, read at startup.
2. **Cluster settings** — dynamic, cluster-wide, applied live via `PUT /_cluster/settings`. Use `persistent` (survives full restart) rather than `transient` (lost on full cluster restart; deprecated).

### 3.1 Node Identity

```yaml
cluster.name: prod-logs
node.name: es-hot-01
```

### 3.2 Node Roles

```yaml
node.roles:
  - data_hot
  - data_content
  - ingest
  - transform
```

Valid role values: `master`, `data`, `data_hot`, `data_warm`, `data_cold`, `data_frozen`, `data_content`, `ingest`, `ml`, `transform`, `remote_cluster_client`, `voting_only`. Empty list = coordinating-only.

### 3.3 Custom Node Attributes

```yaml
node.attr.zone:     us-east-1a
node.attr.rack:     rack-7
node.attr.storage:  nvme
```

Referenced by allocation filters and awareness. Visible in `_cat/nodeattrs` and `GET /_nodes`.

### 3.4 Paths

```yaml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo:
  - /mnt/snapshots
```

`path.data` may be a YAML list to stripe across multiple disks.

> [!NOTE]
> When `path.data` contains multiple paths, Elasticsearch stripes new shards across them in round-robin. It does **not** mirror — if one path fails, the shards on that path are lost. Rely on shard replicas across nodes for redundancy, not multiple local paths.

### 3.5 Networking

```yaml
network.host: _site_
http.port: 9200
transport.port: 9300
http.cors.enabled: false
http.compression: true
http.max_content_length: 100mb
```

`network.host` special values:

| Value | Meaning |
|-------|---------|
| `_local_` | Loopback only |
| `_site_` | All site-local addresses |
| `_global_` | All globally-routable addresses |
| `_[interface]_` | Named network interface (e.g., `_eth0_`) |
| `_ec2:privateIpv4_` | EC2 private IPv4 via metadata service |
| `_ec2:publicIpv4_` | EC2 public IPv4 |
| `_gce:hostname_` | GCE instance hostname |

To publish a different address than bound (NAT / Docker):

```yaml
network.host: _local_
network.publish_host: 10.0.1.42
```

### 3.6 Discovery and Cluster Formation

```yaml
discovery.type: multi-node
discovery.seed_hosts:
  - "es-master-01.internal"
  - "es-master-02.internal:9300"
cluster.initial_master_nodes:
  - "es-master-01"
  - "es-master-02"
  - "es-master-03"
```

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `discovery.type` | static | `multi-node` | `single-node` skips election for dev. |
| `discovery.seed_hosts` | static | `["127.0.0.1","::1"]` | Initial contact points. |
| `discovery.seed_providers` | static | `settings` | Alternative: `file`, `ec2`, `gce`, `azure-classic`. |
| `cluster.initial_master_nodes` | static | empty | Bootstrap-only. Remove after first cluster formation. |
| `cluster.fault_detection.leader_check.interval` | dynamic | `1s` | How often followers check the leader. |
| `cluster.fault_detection.leader_check.timeout` | dynamic | `10s` | Leader check timeout. |
| `cluster.fault_detection.follower_check.interval` | dynamic | `1s` | How often master checks followers. |
| `cluster.fault_detection.follower_check.timeout` | dynamic | `10s` | Follower check timeout. |
| `cluster.publish.timeout` | dynamic | `30s` | Timeout for cluster-state publication. |
| `cluster.election.back_off_time` | dynamic | `100ms` | Back-off after failed election. |
| `cluster.election.max_timeout` | dynamic | `10s` | Maximum election timeout. |

### 3.7 Memory and Bootstrap

```yaml
bootstrap.memory_lock: true
bootstrap.system_call_filter: true
```

Both are verified at startup; failure aborts the node. Set `bootstrap.system_call_filter: false` only on kernels that do not support seccomp.

### 3.8 Allocation and Rebalancing (Cluster Dynamic Settings)

| Setting | Default | Description |
|---------|---------|-------------|
| `cluster.routing.allocation.enable` | `all` | `all`, `primaries`, `new_primaries`, `none` |
| `cluster.routing.allocation.disk.watermark.low` | `85%` | Stop placing new shards above this disk usage |
| `cluster.routing.allocation.disk.watermark.high` | `90%` | Begin migrating shards away above this |
| `cluster.routing.allocation.disk.watermark.flood_stage` | `95%` | Block all index writes above this |
| `cluster.routing.allocation.awareness.attributes` | — | Attribute name(s) for shard spread |
| `cluster.routing.allocation.awareness.force.<attr>.values` | — | Explicit list of attribute values |
| `cluster.routing.allocation.balance.shard` | `0.45f` | Weight for shard count in balancer |
| `cluster.routing.allocation.balance.write_load` | `10.0` | Weight for write load in balancer (9.x) |
| `cluster.routing.allocation.balance.disk_usage` | `2e-11` | Weight for disk usage in balancer (9.x) |
| `cluster.routing.allocation.node_concurrent_recoveries` | `2` | Concurrent recoveries per node |
| `cluster.routing.allocation.node_initial_primaries_recoveries` | `4` | Concurrent primary recoveries on restart |
| `cluster.routing.rebalance.enable` | `all` | Enable/restrict automatic rebalancing |
| `indices.recovery.max_bytes_per_sec` | `40mb` | Recovery bandwidth throttle per node |

### 3.9 Index-Level Settings

Applied in templates or at index creation time. Dynamic settings can be updated via `PUT /{index}/_settings`:

| Setting | Default | Description |
|---------|---------|-------------|
| `index.number_of_shards` | `1` | Primary shard count. Immutable after creation. |
| `index.number_of_replicas` | `1` | Replica count. Changeable live. |
| `index.refresh_interval` | `1s` | How often to make new writes searchable. `-1` = manual only. |
| `index.translog.durability` | `request` | `request` (fsync per write) or `async` (batched fsyncs). |
| `index.translog.flush_threshold_size` | `512mb` | Translog size that triggers a Lucene commit. |
| `index.translog.sync_interval` | `5s` | Fsync interval for `async` durability. |
| `index.codec` | `default` | Lucene codec: `default`, `best_compression` (zstd). |
| `index.mapping.total_fields.limit` | `1000` | Maximum mapped fields. Increase for wide JSON. |
| `index.mapping.depth.limit` | `20` | Maximum nested object depth. |
| `index.max_result_window` | `10000` | Maximum `from + size`. Increase cautiously. |
| `index.max_terms_count` | `65536` | Maximum terms in a terms query. |
| `index.routing.allocation.require.*` | — | Hard allocation requirement. |
| `index.routing.allocation.include.*` | — | Soft inclusion preference. |
| `index.routing.allocation.exclude.*` | — | Hard exclusion. |
| `index.routing.allocation.include._tier_preference` | — | Tier preference list (e.g., `data_hot,data_warm`). |
| `index.lifecycle.name` | — | ILM policy to apply. |
| `index.blocks.write` | `false` | Block write operations. Required before `_shrink`/`_split`. |
| `index.blocks.read_only_allow_delete` | `false` | Set automatically by flood-stage watermark. |
| `index.search.slowlog.threshold.query.warn` | `-1` | Log queries slower than this at WARN. |
| `index.search.slowlog.threshold.fetch.warn` | `-1` | Log fetch phases slower than this at WARN. |
| `index.indexing.slowlog.threshold.index.warn` | `-1` | Log indexing operations slower than this at WARN. |

### 3.10 Thread Pools

Thread pool sizes are auto-tuned from CPU count. Override only with profiling data:

| Pool | Type | Default size | Queue size | Used for |
|------|------|-------------|------------|----------|
| `write` | fixed | `min(cpu, 32)` | 10,000 | Index, bulk, update, delete |
| `search` | fixed_auto_queue | `max(cpu*3/2+1, 5)` | 1,000 | Search, aggregation queries |
| `search_coordination` | fixed_auto_queue | `max(cpu/2, 1)` | 1,000 | CCS coordination |
| `get` | fixed | `cpu` | 1,000 | Get by ID |
| `analyze` | fixed | `1` | 16 | Analyze API |
| `snapshot` | scaling | `min(5, cpu/2)` | — | Snapshot operations |
| `snapshot_meta` | scaling | `min(50, cpu*3)` | — | Snapshot metadata |
| `management` | scaling | `1–5` | — | Cluster management tasks |
| `flush` | scaling | `1–min(5, cpu/2)` | — | Translog flush and fsync |
| `force_merge` | fixed | `1` | — | Force-merge operations |
| `refresh` | scaling | `1–min(10, cpu/2)` | — | Index refresh |

### 3.11 Slow Logs

```http
PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn":   "10s",
  "index.search.slowlog.threshold.query.info":   "5s",
  "index.search.slowlog.threshold.query.debug":  "2s",
  "index.search.slowlog.threshold.query.trace":  "500ms",
  "index.search.slowlog.threshold.fetch.warn":   "1s",
  "index.indexing.slowlog.threshold.index.warn": "10s",
  "index.indexing.slowlog.level":                "info"
}
```

Logs emit as structured JSON to `logs/<cluster>_index_search_slowlog.json` and `logs/<cluster>_index_indexing_slowlog.json`.

### 3.12 Runtime Log Level Changes

```http
PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.discovery": "DEBUG"
  }
}
```

### 3.13 JVM Options

Put overrides in `config/jvm.options.d/` (never edit `config/jvm.options` directly):

```
# config/jvm.options.d/custom.options
-Xms30g
-Xmx30g
-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=30
-XX:G1ReservePercent=25
-XX:+AlwaysPreTouch
-Djava.io.tmpdir=/var/lib/elasticsearch/tmp
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
```

### 3.14 Keystore (Secure Settings)

```bash
bin/elasticsearch-keystore create [-p]
bin/elasticsearch-keystore add <setting>
bin/elasticsearch-keystore add-file <setting> <file>
bin/elasticsearch-keystore list
bin/elasticsearch-keystore remove <setting>
bin/elasticsearch-keystore passwd
bin/elasticsearch-keystore upgrade
```

Reload without restart:

```http
POST /_nodes/reload_secure_settings
{ "secure_settings_password": "optional-keystore-password" }
```

Common secure settings:

| Setting key | Purpose |
|-------------|---------|
| `xpack.security.transport.ssl.keystore.secure_password` | Transport TLS keystore password |
| `xpack.security.http.ssl.keystore.secure_password` | HTTP TLS keystore password |
| `s3.client.<name>.access_key` | S3 client access key |
| `s3.client.<name>.secret_key` | S3 client secret key |
| `azure.client.<name>.account` | Azure storage account name |
| `azure.client.<name>.key` | Azure storage account key |
| `gcs.client.<name>.credentials_file` | GCS service account JSON path |
| `xpack.security.authc.realms.ldap.<name>.secure_bind_password` | LDAP bind password |

### 3.15 Docker Environment Variable Convention

From `EnvironmentAwareCommand.java`:

- `ES_SETTING_` prefix variables are translated: remove prefix, replace `_` with `.`, unescape `..` back to `_`, lowercase.
- `ES_SETTING_CLUSTER__NAME=prod-logs` → `cluster.name: prod-logs`
- `ES_SETTING_NODE__NAME=es-01` → `node.name: es-01`
- Lowercase dot-separated variables also work directly in the Docker entrypoint.

### 3.16 Plugin Configuration File

```yaml
# config/elasticsearch-plugins.yml
plugins:
  - id: analysis-icu
  - id: custom-plugin
    location: https://example.com/my-plugin-9.5.0.zip
  - id: maven-plugin
    location: org.example:my-plugin:1.0.0
  - id: proxied-plugin
    location: https://example.com/plugin.zip
    proxy: https://proxy.internal:3128
proxy: https://proxy.internal:3128
```

Manual management:

```bash
bin/elasticsearch-plugin install analysis-icu
bin/elasticsearch-plugin install https://example.com/plugin.zip
bin/elasticsearch-plugin remove analysis-icu
bin/elasticsearch-plugin list
```

---

## 4. Deployment & Scaling

### 4.1 Distribution Packages

Elasticsearch ships in three packaging families from `distribution/`:

| Package | Description |
|---------|-------------|
| Docker image | `docker.elastic.co/elasticsearch/elasticsearch:9.5.0` — bundled JDK, CA/cert bootstrap, auto-config on first start. Preferred for containerized platforms. |
| `.deb` / `.rpm` | systemd units, `/etc/elasticsearch/` config, `/var/lib/elasticsearch/` data, standard Linux service lifecycle. |
| `.tar.gz` / `.zip` | Self-contained directory. Good for developers and air-gapped environments. |

### 4.2 Single-Node Development

```yaml
# elasticsearch.yml
cluster.name: dev-local
node.name: dev-node-1
discovery.type: single-node
xpack.security.enabled: false
```

Start: `bin/elasticsearch` (foreground) or `bin/elasticsearch -d -p /var/run/elasticsearch.pid` (daemon).

### 4.3 Multi-Node Cluster Bootstrap

**Step 1** — Prepare `elasticsearch.yml` on each node:

```yaml
cluster.name: prod-logs
node.name: es-master-01          # unique per node
node.roles: [master]
network.host: 10.0.1.11          # unique per node
discovery.seed_hosts:
  - "10.0.1.11"
  - "10.0.1.12"
  - "10.0.1.13"
cluster.initial_master_nodes:
  - "es-master-01"
  - "es-master-02"
  - "es-master-03"
bootstrap.memory_lock: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/transport.p12
```

**Step 2** — Generate certificates (on one node, copy to all):

```bash
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12 --pass
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --dns es-master-01,es-master-02,es-master-03 \
  --out bundle.zip
```

**Step 3** — Start all nodes, verify cluster formation:

```bash
curl -sk -u elastic:$PASSWORD https://es-master-01:9200/_cluster/health?pretty
```

**Step 4** — Remove `cluster.initial_master_nodes` from all configs and do a rolling restart.

### 4.4 Docker Compose (Three-Node + Kibana)

The reference compose topology from the source tree:

```yaml
version: "3.8"
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip &&
        unzip config/certs/ca.zip -d config/certs &&
        bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip \
          --in config/certs/instances.yml \
          --ca-cert config/certs/ca/ca.crt \
          --ca-key config/certs/ca/ca.key &&
        unzip config/certs/certs.zip -d config/certs
      '

  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200

  kibana:
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    ports:
      - ${KIBANA_PORT}:5601
    volumes:
      - certs:/usr/share/kibana/config/certs

volumes:
  certs:
  esdata01:
  esdata02:
  esdata03:
```

### 4.5 Kubernetes (ECK)

The officially supported operator is **Elastic Cloud on Kubernetes (ECK)**. It manages:

- Cluster lifecycle (create, scale, upgrade)
- Rolling upgrades with health gate checks
- TLS certificate generation and rotation
- Storage volume management
- Secret management for passwords and API keys

Minimum ECK manifest for a 3-node cluster:

```yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: prod-logs
spec:
  version: 9.5.0
  nodeSets:
  - name: masters
    count: 3
    config:
      node.roles: ["master", "voting_only"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: standard
  - name: hot-data
    count: 3
    config:
      node.roles: ["data_hot", "data_content", "ingest"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Gi
        storageClassName: fast-nvme
```

For bare StatefulSets without ECK, you must manually handle cert rotation, Pod Disruption Budgets, anti-affinity rules, and node naming stability.

### 4.6 Data Tiers

Elasticsearch supports five data tiers. Each tier is a `node.roles` value; indices target a tier via `index.routing.allocation.include._tier_preference`.

| Tier | Role | Storage | Use |
|------|------|---------|-----|
| Content | `data_content` | Fast SSD | Non-time-series: catalogs, user data |
| Hot | `data_hot` | NVMe | Current time-series writes |
| Warm | `data_warm` | SATA SSD | Recent, less-frequently queried |
| Cold | `data_cold` | Spinning or object storage | Rarely queried historical data |
| Frozen | `data_frozen` | Object storage only | Query-on-demand; searchable snapshots |

**Tier-aware index template:**

```http
PUT /_index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-policy",
      "index.routing.allocation.include._tier_preference": "data_hot"
    }
  }
}
```

### 4.7 ILM — Index Lifecycle Management

ILM automates the hot → warm → cold → frozen → delete progression:

```http
PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "set_priority": { "priority": 50 },
          "readonly": {},
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "14d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "found-snapshots" }
        }
      },
      "frozen": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "found-snapshots" }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}
```

ILM management commands:

```http
GET /_ilm/explain/{index}          # Current phase, action, step
POST /_ilm/retry/{index}           # Retry after a failed step
POST /_ilm/move/{index}            # Manually advance to a specific step
POST /_ilm/stop                    # Pause ILM globally
POST /_ilm/start                   # Resume ILM
GET /_ilm/status                   # ILM running/stopped state
```

### 4.8 Data Stream Lifecycle (Simpler Alternative to ILM)

For pure retention + downsampling without the full ILM state machine:

```http
PUT /_data_stream/logs-app-prod/_lifecycle
{
  "data_retention": "90d",
  "downsampling": [
    { "after": "7d",  "fixed_interval": "1h" },
    { "after": "30d", "fixed_interval": "1d" }
  ]
}
```

### 4.9 Snapshots and Restore

Snapshots are the **only** supported backup mechanism. Do not copy shard directories off a live cluster.

#### Register a Repository

S3:

```http
PUT /_snapshot/my-s3
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-snapshots",
    "base_path": "prod",
    "client": "default",
    "server_side_encryption": true
  }
}
```

S3 credentials in the keystore:

```bash
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
curl -u elastic:$P https://localhost:9200/_nodes/reload_secure_settings -X POST
```

Azure:

```http
PUT /_snapshot/my-azure
{
  "type": "azure",
  "settings": {
    "container": "my-es-snapshots",
    "base_path": "prod"
  }
}
```

GCS:

```http
PUT /_snapshot/my-gcs
{
  "type": "gcs",
  "settings": {
    "bucket": "my-es-snapshots",
    "base_path": "prod",
    "client": "default"
  }
}
```

Shared filesystem (NFS — prefer object storage):

```http
PUT /_snapshot/my-fs
{
  "type": "fs",
  "settings": {
    "location": "/mnt/snapshots/prod",
    "compress": true
  }
}
```

Supported repository types: `fs`, `url` (read-only), `s3`, `azure`, `gcs`, `hdfs` (plugin), `source` (source-only, compressed archival).

#### SLM — Automated Snapshot Policies

```http
PUT /_slm/policy/daily-snapshots
{
  "schedule":   "0 30 1 * * ?",
  "name":       "<daily-snap-{now/d}>",
  "repository": "my-s3",
  "config": {
    "indices":              ["*"],
    "include_global_state": true,
    "ignore_unavailable":   true
  },
  "retention": {
    "expire_after": "30d",
    "min_count":    5,
    "max_count":    50
  }
}
```

#### Restore

```http
POST /_snapshot/my-s3/daily-snap-2026-04-22/_restore
{
  "indices":              "logs-app-*",
  "rename_pattern":       "(.+)",
  "rename_replacement":   "restored-$1",
  "include_global_state": false
}
```

#### Searchable Snapshots

Cold/frozen indices can be **searchable snapshots** — data lives only in the repository and is read on demand. ILM integrates this via the `searchable_snapshot` action. No replicas are stored locally, so capacity per node can be enormous. The `x-pack/plugin/blob-cache/` (56K chars) provides the local disk cache layer for repeated access.

### 4.10 Cross-Cluster Search (CCS)

Register remote clusters:

```http
PUT /_cluster/settings
{
  "persistent": {
    "cluster.remote.eu-prod.mode":          "proxy",
    "cluster.remote.eu-prod.proxy_address": "eu-prod-proxy.example.com:9400",
    "cluster.remote.us-logs.mode":          "sniff",
    "cluster.remote.us-logs.seeds":         ["us-logs-1.example.com:9300"]
  }
}
```

- **Sniff mode** — discovers all nodes in the remote cluster and opens direct connections. Requires transport port access.
- **Proxy mode** — multiplexes all traffic through a single proxy address. Required for cross-VPC/cross-k8s traffic without exposing the transport port.

Search across clusters:

```http
GET /eu-prod:logs-*,us-logs:logs-*,logs-*/_search
{
  "query": { "match": { "message": "error" } }
}
```

Use `?ccs_minimize_roundtrips=true` (the default) — fetches only the final top-N docs from remote clusters, reducing round trips significantly.

### 4.11 Cross-Cluster Replication (CCR)

CCR streams writes from a leader index to a follower in another cluster in near-real-time.

Use cases:
- **Disaster recovery** — follower in a secondary region; promote on primary failure.
- **Read scaling** — local followers in app-adjacent regions serve local read traffic.
- **Aggregation** — centralize indices from many leader clusters into a single analysis cluster.

Create a follower:

```http
PUT /logs-app-follower/_ccr/follow
{
  "remote_cluster":    "eu-prod",
  "leader_index":      "logs-app-000001",
  "settings":          { "index.number_of_replicas": 0 }
}
```

Follower indices are **read-only** on the follower side. To promote (e.g., on DR failover):

```http
POST /logs-app-follower/_ccr/pause_follow
POST /logs-app-follower/_ccr/unfollow
```

---

## 5. Operations

### 5.1 Health Monitoring

```http
GET /_cluster/health
GET /_cluster/health?level=indices
GET /_health_report                           # richer multi-dimensional health (9.x)
GET /_cluster/state?filter_path=metadata.indices.*.state
```

Health status:

| Status | Meaning |
|--------|---------|
| `green` | All primary and replica shards allocated |
| `yellow` | All primaries allocated; at least one replica unassigned |
| `red` | At least one primary unassigned — reads/writes to affected indices fail |

Key diagnostic endpoints:

| Endpoint | Purpose |
|----------|---------|
| `GET /_cluster/allocation/explain` | Explains why a shard is unassigned |
| `GET /_cat/nodes?v` | Node list with load, heap, roles |
| `GET /_cat/indices?v&s=store.size:desc` | All indices sorted by size |
| `GET /_cat/shards?v&h=index,shard,prirep,state,node` | Per-shard state |
| `GET /_cat/allocation?v` | Disk usage and shard count per node |
| `GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected` | Thread pool saturation |
| `GET /_cat/recovery?v&active_only=true` | In-progress shard recovery |
| `GET /_cat/pending_tasks?v` | Queued cluster-state tasks |
| `GET /_nodes/stats` | Full per-node stats (JVM, threadpools, indices, OS, process, transport, HTTP) |
| `GET /_nodes/hot_threads?threads=5` | Top CPU consumers across all nodes |

> [!TIP]
> Use `GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected` as your first diagnostic when users report search or indexing slowness. A rising `rejected` count on the `search` or `write` pool is the earliest measurable signal of capacity pressure — it appears before latency visibly degrades.

### 5.2 Rolling Restart (Zero Downtime)

1. Disable replica allocation to prevent unnecessary shard movement:

   ```http
   PUT /_cluster/settings
   { "persistent": { "cluster.routing.allocation.enable": "primaries" } }
   ```

2. Stop indexing if possible, then flush:

   ```http
   POST /_flush
   ```

3. Stop the target node, upgrade/patch it, start it back.

4. Wait until the node rejoins:

   ```http
   GET /_cat/nodes?v
   ```

5. Re-enable allocation:

   ```http
   PUT /_cluster/settings
   { "persistent": { "cluster.routing.allocation.enable": null } }
   ```

6. Wait for green health:

   ```http
   GET /_cluster/health?wait_for_status=green&timeout=5m
   ```

7. Move to the next node.

> [!NOTE]
> Always upgrade/restart **data nodes first, master-eligible nodes last** to preserve quorum throughout the window. After re-enabling allocation, peer recovery will replay translog entries and reallocate replicas — throttle recovery bandwidth with `indices.recovery.max_bytes_per_sec` if the cluster serves production traffic during the window.

### 5.3 Decommissioning a Node

Route all shards off the node before stopping it:

```http
PUT /_cluster/settings
{ "persistent": { "cluster.routing.allocation.exclude._name": "es-warm-09" } }
```

Monitor until 0 shards remain:

```http
GET /_cat/allocation?v
```

For a master-eligible node, also add to voting-config exclusions before stopping:

```http
POST /_cluster/voting_config_exclusions?node_names=es-master-02&timeout=1m
```

After decommissioning, clear the exclusion:

```http
DELETE /_cluster/voting_config_exclusions
```

### 5.4 Graceful Shutdown API (9.x)

The shutdown API allows controlled node removal with proper wait logic:

```http
PUT /_nodes/<node_id>/shutdown
{
  "type":             "remove",     # or "restart", "replace"
  "reason":           "AZ drain 2026-04-23",
  "target_node_name": "es-warm-99"  # for "replace" type
}
```

Monitor shutdown progress:

```http
GET /_nodes/<node_id>/shutdown
```

### 5.5 Shard Shrink, Split, and Reindex

**Shrink** (reduce shard count — target must divide evenly):

```http
# First: mark source read-only and route to a single node
PUT /source-index/_settings
{
  "index.blocks.write": true,
  "index.routing.allocation.require._name": "es-hot-01"
}
# Then: shrink
POST /source-index/_shrink/target-index
{ "settings": { "index.number_of_shards": 1 } }
```

**Split** (increase shard count — target must be a multiple of source):

```http
PUT /source-index/_settings
{ "index.blocks.write": true }
POST /source-index/_split/target-index
{ "settings": { "index.number_of_shards": 4 } }
```

**Reindex** (arbitrary shard count change or cross-cluster copy):

```http
POST /_reindex
{
  "source": { "index": "old-index" },
  "dest":   { "index": "new-index" }
}
```

### 5.6 Recovering a Lost Master Quorum (Last Resort)

If you permanently lose a majority of master-eligible nodes with their data:

```bash
# On the surviving node with the most recent cluster state:
bin/elasticsearch-node unsafe-bootstrap --confirm
```

> [!CAUTION]
> `unsafe-bootstrap` forms a new cluster from a single node. Any cluster-state changes that were applied on the lost masters but not yet acknowledged by this node are **permanently lost**. This includes new indices, mapping changes, security configs, and ILM policies added since the last successful publication. Always snapshot first. Restore from snapshots after bootstrapping if necessary.

### 5.7 Fixing Flood-Stage Write Block

When the flood-stage watermark is hit, all indices become read-only:

1. **Free disk first** (delete old indices, add nodes, expand volumes).
2. Clear the write block:

   ```http
   PUT /_all/_settings
   { "index.blocks.read_only_allow_delete": null }
   ```

Running the clear before freeing disk will cause the block to re-apply immediately.

### 5.8 Force Merge

```http
POST /my-index/_forcemerge?max_num_segments=1
```

> [!CAUTION]
> Only run `_forcemerge` on **read-only indices** (ILM warm/cold, closed index, write-blocked). Force-merging active write indices destroys the merge policy geometry, can cause large GC spikes, and the I/O can saturate the node. `max_num_segments=1` is appropriate only for truly static data.

### 5.9 APM and Telemetry

```yaml
telemetry.agent.server_url: https://apm-server.internal:8200
telemetry.secret_token: secret-token-here        # or telemetry.api_key
telemetry.tracing.enabled: true
telemetry.metrics.enabled: true
```

APM tracing emits spans for search, indexing, and cluster-state operations. Useful for tracing slow queries from the ES side rather than the client side.

### 5.10 Prometheus Metrics

The `x-pack/plugin/prometheus/` plugin exposes a Prometheus-compatible scrape endpoint. After installing or enabling the plugin, metrics are served at `GET /_prometheus/metrics`. This covers node stats, index stats, JVM, thread pools, circuit breakers, and more in Prometheus exposition format.

### 5.11 Monitoring Stack Options

| Approach | How |
|----------|-----|
| Elastic Stack monitoring | Metricbeat / Elastic Agent collects `_nodes/stats` + `_cluster/stats` into a dedicated monitoring cluster; view in Kibana Stack Monitoring |
| Prometheus + Grafana | Scrape `/_prometheus/metrics` or use a community exporter |
| Watcher alerts | Schedule watches against cluster health endpoints for self-contained alerting |

### 5.12 Upgrade Procedures

#### Supported Upgrade Paths

- **Minor upgrades** (e.g., 9.4 → 9.5): rolling, online, no reindex required.
- **Major upgrades** (e.g., 8.x → 9.x): rolling upgrade from the **last minor of the previous major** (e.g., 8.19 → 9.0 directly). Earlier minors require upgrading to the last minor first.
- **Index compatibility**: indices created in version N are compatible with N and N+1 major only. Indices older than that must be reindexed or retired before a major upgrade.

Check readiness:

```http
GET /_migration/deprecations
```

#### Rolling Upgrade Procedure (Minor)

1. Read release notes and resolve critical deprecations (`GET /_migration/deprecations`).
2. Disable shard allocation for replicas: `cluster.routing.allocation.enable: primaries`.
3. For each node: stop → upgrade package/image → start → wait for rejoin.
4. Re-enable allocation after each node, wait for `green`.
5. Upgrade master-eligible nodes **last**.
6. Verify: `GET /` returns the new version on all nodes.

#### Major Upgrade Notes

- Run mixed major versions for as short a window as possible. The cluster tolerates **current major and current major - 1** simultaneously during the rolling window.
- Test against the **Upgrade Assistant** in Kibana (reads `_migration/deprecations` and guides through remediation) well before the maintenance window.
- Snapshot the cluster before starting.

---

## 6. Performance

### 6.1 Indexing Optimization

**Bulk requests:**

- Target **5–15 MB per bulk body** and hundreds to low thousands of documents per request.
- Monitor `rejected` on the `write` thread pool; back off with exponential retry on `429`.
- Use async bulk clients (the Elasticsearch Java client, Python async client) to pipeline requests.

**Refresh interval:**

```http
PUT /my-index/_settings
{ "index.refresh_interval": "30s" }
```

Set to `30s` or `-1` during initial bulk loads; return to `1s` (or the default) afterward.

**Translog durability:**

```http
PUT /my-index/_settings
{ "index.translog.durability": "async" }
```

`async` batches fsyncs to disk (interval controlled by `index.translog.sync_interval: 5s`). Throughput can improve 2× at the cost of up to `sync_interval` of data on a hard crash. Acceptable for log ingestion; not for financial transactions.

**Replicas during bulk load:**

```http
PUT /my-index/_settings
{ "index.number_of_replicas": 0 }
# ... bulk index ...
PUT /my-index/_settings
{ "index.number_of_replicas": 1 }
```

**Auto-generated IDs:**

Let Elasticsearch assign `_id` when possible — avoids a "does this document exist?" lookup on the shard, saving one read per document.

**Segment merges:**

Do not over-force-merge. One `_forcemerge?max_num_segments=1` at the end of a bulk load on a static dataset is fine. Continuous or repeated force-merging is pathological.

**Codec:**

```http
PUT /_index_template/logs-compressed
{
  "template": {
    "settings": { "index.codec": "best_compression" }
  }
}
```

`best_compression` (zstd) reduces storage 10–30% at a modest CPU cost. Best for warm/cold indices where search speed matters less.

### 6.2 Search Optimization

**Filters vs. queries:**

Structure-matched constraints (term, range, exists) in `filter` context — they are cached and skip relevance scoring:

```json
{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "status": "active" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "must": [
        { "match": { "message": "error" } }
      ]
    }
  }
}
```

**Routing:**

If your access pattern is always `tenant=X`, route documents by tenant:

```http
POST /my-index/_doc?routing=tenant-acme
{ ... }

GET /my-index/_search?routing=tenant-acme
{ ... }
```

Queries with the matching routing only hit the one shard that holds that tenant's data.

**Shard request cache:**

On by default. Effective for `size: 0` (aggregation-only) queries against immutable data. Invalidated on every refresh — raise `index.refresh_interval` to extend cache lifetime for static data.

**Field data and `keyword` fields:**

Avoid sorting or aggregating on `text` fields — fielddata is disabled by default and loads the entire column into heap. Use a `keyword` sub-field:

```json
{
  "mappings": {
    "properties": {
      "message": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" } }
      }
    }
  }
}
```

**Pagination patterns:**

| Method | Best for |
|--------|---------|
| `from` + `size` | First few pages of small result sets (≤ `max_result_window = 10000`) |
| `search_after` + PIT | Deep pagination, large result sets, stable cursor across refreshes |
| Scroll (legacy) | Deprecated; use `search_after` + PIT for new code |

**Point-in-time (PIT):**

```http
POST /my-index/_pit?keep_alive=5m
# Returns: { "id": "PIT_ID" }

GET /_search
{
  "pit": { "id": "PIT_ID", "keep_alive": "5m" },
  "sort": [{ "@timestamp": "desc" }, { "_shard_doc": "asc" }],
  "search_after": [<last_sort_values>]
}
```

**`preference=_local`:**

For latency-sensitive queries from coordinating nodes that also hold shards, use `?preference=_local` to avoid a network hop.

### 6.3 Memory and GC Tuning

- Heap: 50% of RAM, max ~30.5 GB. Never exceed this.
- G1GC (`-XX:+UseG1GC`) is the default. `-XX:InitiatingHeapOccupancyPercent=30` triggers GC sooner to avoid long pauses.
- `-XX:+AlwaysPreTouch` pre-faults heap pages at startup, reducing GC latency spikes in early operation.
- Keep the filesystem page cache large. Never run Elasticsearch on a node that also hosts a page-cache-heavy service (database, HDFS, etc.).
- Disable swap: `swapoff -a` + `vm.swappiness=1` + `bootstrap.memory_lock: true`.
- Monitor `GET /_nodes/stats/jvm` for `heap_used_percent`. Sustained > 75% is a warning; > 85% needs action (add nodes or reduce load).

### 6.4 Circuit Breakers

Circuit breakers prevent OOM by tracking memory used by operations. Key breakers:

| Breaker | Default limit | Purpose |
|---------|--------------|---------|
| `parent` | 95% heap | Overall limit; trips before heap exhaustion |
| `request` | 60% heap | Limits memory for a single request's data structures |
| `fielddata` | 40% heap | Limits fielddata (heap-resident column cache) |
| `in_flight_requests` | 100% heap | Limits in-flight HTTP request bodies |
| `accounting` | 100% heap | Tracks long-lived structures |
| `eql_sequence` | 50% of request breaker | EQL sequence computation limit |
| `model_inference` | 50% of request breaker | ML inference memory limit |

Check breaker state:

```http
GET /_nodes/stats/breaker
```

A `circuit_breaking_exception` means a single request exceeded the breaker limit. Fix by reducing request size, using `filter` rather than `query`, paginating with `search_after`, or raising the specific breaker limit carefully.

### 6.5 Dense Vector (kNN) Performance

- Use `dense_vector` field type with `dims` matching your embedding model output.
- For large corpora, use `index: true` and HNSW indexing (`m` and `ef_construction` parameters). Default `m=16`, `ef_construction=100`.
- Query with `knn` inside `_search`:

  ```json
  {
    "knn": {
      "field":          "embedding",
      "query_vector":   [0.1, 0.2, ...],
      "k":              10,
      "num_candidates": 100
    }
  }
  ```

- `num_candidates` controls recall vs. speed trade-off per shard. Higher = more accurate, more CPU.
- SIMD acceleration is provided by `libs/simdvec/` on x86_64 (AVX2/AVX-512) and aarch64 (NEON). Requires a CPU that supports these; falls back to scalar otherwise.

### 6.6 Benchmarking

Use **Rally** (`github.com/elastic/rally`) — the official benchmarking harness. Built-in tracks include `nyc_taxis`, `http_logs`, `geopoint`, `nested`, `eql`, `dense_vectors`. Build custom tracks from a snapshot of your data and a representative query log.

For micro-benchmarks of core algorithms, `benchmarks/` contains JMH benchmarks used by the core team.

---

## 7. Security

### 7.1 Security Model Overview

Security is **on by default** from 8.0 onward. A freshly started single node auto-configures:

- TLS on both transport (9300) and HTTP (9200) layers.
- A CA and per-node certificate.
- A random password for the `elastic` superuser, printed to the terminal.
- An enrollment token for Kibana and joining nodes.

> [!WARNING]
> Never store TLS passwords, API secrets, or LDAP bind passwords in `elasticsearch.yml`. These would be readable by any process with filesystem access. Always use `bin/elasticsearch-keystore add` for sensitive values. The keystore is encrypted at rest and only decrypted in memory by the running node.

### 7.2 Security Feature Toggles

| Setting | Default | Description |
|---------|---------|-------------|
| `xpack.security.enabled` | `true` | Master switch for authentication, authorization, and audit |
| `xpack.security.autoconfiguration.enabled` | `true` | Auto-generates CA, certs, and passwords on first start |
| `xpack.security.enrollment.enabled` | `false` (set `true` by autoconfig) | Allow generating enrollment tokens |
| `xpack.security.fips_mode.enabled` | `false` | FIPS 140-2 compliance mode |
| `xpack.security.fips_mode.required_providers` | — | E.g., `["BCFIPS", "BCJSSE"]` |
| `xpack.security.hide_settings` | — | Wildcard patterns of settings to redact from node info |

### 7.3 TLS Configuration

#### Transport TLS (mandatory for multi-node)

```yaml
xpack.security.transport.ssl.enabled:            true
xpack.security.transport.ssl.verification_mode:  certificate
xpack.security.transport.ssl.keystore.path:      certs/transport.p12
xpack.security.transport.ssl.truststore.path:    certs/transport.p12
```

Passwords in the keystore:

```bash
bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
```

#### HTTP TLS (client-facing)

```yaml
xpack.security.http.ssl.enabled:            true
xpack.security.http.ssl.keystore.path:      certs/http.p12
xpack.security.http.ssl.keystore.type:      PKCS12
```

HTTP SSL settings reference:

| Setting | Description |
|---------|-------------|
| `xpack.security.http.ssl.keystore.path` | Path to `.jks` or `.p12`. Mutually exclusive with `ssl.key`. |
| `xpack.security.http.ssl.keystore.type` | `jks` or `PKCS12` (defaults from extension `.p12`, `.pfx`, `.pkcs12`). |
| `xpack.security.http.ssl.keystore.secure_password` | Keystore password (in the ES keystore). |
| `xpack.security.http.ssl.keystore.secure_key_password` | Key-entry password (in the ES keystore). |
| `xpack.security.http.ssl.truststore.path` | Trusted-CA store path. |
| `xpack.security.http.ssl.truststore.type` | Truststore format. |
| `xpack.security.http.ssl.truststore.secure_password` | Truststore password (in the ES keystore). |
| `xpack.security.http.ssl.certificate_authorities` | PEM CA cert list (alternative to truststore). |
| `xpack.security.http.ssl.verification_mode` | `none`, `certificate`, `full` (default: `full`). |
| `xpack.security.http.ssl.cipher_suites` | Explicit cipher suite list (default: JDK defaults). |
| `xpack.security.http.ssl.supported_protocols` | E.g., `["TLSv1.3", "TLSv1.2"]`. |

### 7.4 Certificate Management

```bash
# Generate a CA
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12 --pass

# Generate a node certificate from the CA
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --dns node-1.example.com,node-2.example.com \
  --ip 10.0.1.11,10.0.1.12 \
  --out bundle.zip

# Interactive wizard for HTTP TLS
bin/elasticsearch-certutil http

# Generate a CSR for external CA signing
bin/elasticsearch-certutil csr \
  --name node-1 \
  --dns node-1.example.com \
  --out csr.zip
```

For Kubernetes/ECK, certificate management is fully automated by the operator.

### 7.5 Authentication Realms

Realms are authentication providers stacked in priority order under `xpack.security.authc.realms`:

| Realm type | Purpose | Notes |
|------------|---------|-------|
| `native` | Built-in user database | `elastic`, `kibana_system`, `logstash_system`, etc. |
| `file` | Local flat-file users | Managed with `elasticsearch-users`. Good for break-glass accounts. |
| `ldap` | Corporate LDAP or Active Directory | Supports user DN template and group-to-role mapping. |
| `active_directory` | AD-specific wrapper | Domain, Global Catalog, DnS domain settings. |
| `pki` | Client certificate authentication | Distinguished name mapped to role. |
| `saml` | SAML 2.0 service provider | Used for Kibana SSO via an IdP (Okta, Azure AD, Ping, etc.). |
| `oidc` | OpenID Connect | Used for Kibana SSO. |
| `kerberos` | Kerberos/SPNEGO | Enterprise SSO. |
| `jwt` | Signed JWT bearer tokens | Service-to-service auth; supports RSA and HMAC signing. |

Example realm stack:

```yaml
xpack.security.authc.realms:
  native:
    native1:
      order: 0
  ldap:
    corporate_ldap:
      order: 1
      url: "ldaps://ldap.example.com:636"
      bind_dn: "cn=elastic-bind,dc=example,dc=com"
      user_search:
        base_dn: "dc=example,dc=com"
      group_search:
        base_dn: "dc=example,dc=com"
      files:
        role_mapping: "config/role_mapping.yml"
  pki:
    pki1:
      order: 2
      certificate_authorities: ["certs/ca.crt"]
```

### 7.6 Authorization — Roles

Roles are sets of cluster privileges, index privileges, application privileges, and optional FLS/DLS.

```http
PUT /_security/role/logs_reader
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names":      ["logs-*"],
      "privileges": ["read", "view_index_metadata"],
      "field_security": {
        "grant":  ["@timestamp", "service", "message", "level"],
        "except": []
      },
      "query": "{\"term\":{\"tenant\":\"acme\"}}"
    }
  ]
}
```

Cluster privileges (common):

| Privilege | Access |
|-----------|--------|
| `all` | Full cluster access |
| `monitor` | `GET /_cluster/health`, `_nodes`, `_cat/*`, `_tasks` |
| `manage` | All monitor + manage cluster-level resources |
| `manage_security` | Full security API access |
| `manage_ilm` | ILM policy management |
| `manage_slm` | SLM policy management |
| `manage_ml` | ML job management |
| `manage_transform` | Transform management |
| `manage_watcher` | Watcher management |
| `manage_ccr` | CCR management |

Index privileges (common):

| Privilege | Access |
|-----------|--------|
| `all` | Full index access |
| `read` | Search, get, explain, field caps |
| `write` | Index, bulk, update, delete |
| `create_doc` | Index only (no update/delete) |
| `index` | Index + update |
| `delete` | Delete documents |
| `create` | Create index |
| `delete_index` | Delete index |
| `manage` | All index management operations |
| `view_index_metadata` | Index info, mappings, settings, aliases |

### 7.7 Users

```http
# Create
POST /_security/user/alice
{
  "password":  "changeme",
  "roles":     ["logs_reader"],
  "full_name": "Alice Smith",
  "email":     "[email removed]"
}

# Change password
PUT /_security/user/alice/_password
{ "password": "newpassword" }

# Disable
PUT /_security/user/alice/_disable

# Delete
DELETE /_security/user/alice
```

### 7.8 API Keys

API keys are preferred for application credentials. They are scoped, expirable, and can be invalidated without rotating passwords.

```http
# Create an API key
POST /_security/api_key
{
  "name":       "my-app-key",
  "expiration": "30d",
  "role_descriptors": {
    "my-app-role": {
      "indices": [{ "names": ["logs-*"], "privileges": ["read"] }]
    }
  }
}
# Response includes "id" and "api_key" (base64-encoded joined as id:api_key)

# Use in requests:
# Authorization: ApiKey <base64(id:api_key)>

# Invalidate
DELETE /_security/api_key
{ "id": "KEY_ID" }

# Query all active keys
GET /_security/api_key?username=alice
```

### 7.9 Service Accounts

Service accounts are non-user security principals for infrastructure services (Fleet, Kibana, Logstash). They use file-backed or index-backed tokens, not passwords.

```bash
bin/elasticsearch-service-tokens create elastic/fleet-server my-fleet-token
bin/elasticsearch-service-tokens list elastic/fleet-server
bin/elasticsearch-service-tokens delete elastic/fleet-server my-fleet-token
```

### 7.10 Audit Logging

```yaml
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include:
  - access_denied
  - authentication_failed
  - run_as_denied
  - tampered_request
  - connection_denied
  - anonymous_access_denied
xpack.security.audit.logfile.events.exclude: []
```

Audit logs emit as structured JSON to `logs/<cluster>_audit.json`. Fields include `@timestamp`, `node.id`, `event.type`, `event.action`, `user.name`, `request.id`, `url.path`, `network.client.ip`.

### 7.11 Field-Level and Document-Level Security (FLS/DLS)

**FLS** — restrict which fields a role can read:

```json
"field_security": {
  "grant":  ["@timestamp", "service", "level"],
  "except": []
}
```

**DLS** — restrict which documents a role can read (via a query):

```json
"query": "{\"term\":{\"tenant\":\"acme\"}}"
```

FLS/DLS are evaluated per-segment at query time. They add overhead proportional to the complexity of the DLS query. Avoid DLS on frequently-updated large indices.

### 7.12 FIPS Mode

```yaml
xpack.security.fips_mode.enabled: true
xpack.security.fips_mode.required_providers:
  - "BCFIPS"
  - "BCJSSE"
```

Requires:
- A FIPS 140-2 validated JVM (Bouncy Castle FIPS or an NSS-backed JDK).
- All keystores must be BCFKS format (not PKCS12 or JKS).
- Passwords must be stored in the ES keystore; the keystore itself must be BCFKS.
- Build with `elasticsearch.fips` Gradle plugin for distribution-level FIPS compliance.

### 7.13 Network Isolation

> [!CAUTION]
> **Transport port (9300) is internal-only.** Firewall it from the internet and from application networks. Only Elasticsearch nodes and monitoring sidecars should reach it. Never expose a cleartext port 9200 to the public internet. Always terminate TLS at Elasticsearch itself or at a re-encrypting load balancer — not a TLS-terminating proxy that communicates plaintext to ES.

Recommended network segmentation:

```
Internet
  → Reverse proxy (TLS termination, optional WAF)
  → Elasticsearch HTTP port 9200 (TLS, internal network only)

Application servers
  → Elasticsearch HTTP port 9200 (TLS, internal network only)

Elasticsearch nodes
  ↔ Transport port 9300 (TLS, cluster-internal network only, firewall from all other networks)
```

---

## 8. Developer Internals

### 8.1 Request Lifecycle — Write

1. HTTP request arrives on a coordinating node's Netty HTTP pipeline.
2. The REST handler decodes the request and dispatches to `TransportBulkAction` (or `TransportIndexAction` for single-doc writes).
3. The coordinating node resolves the target index/alias, resolves aliases, and computes the shard per document using `_routing` (defaults to `_id`).
4. Per shard, the write is forwarded to the primary shard's node over the transport layer.
5. The primary applies the write to its Lucene `IndexWriter` and the translog.
6. The primary fan-outs to replica shards, waits for `wait_for_active_shards` acknowledgments.
7. The response returns to the coordinating node and then to the client.
8. At the next refresh (periodic or explicit), the new segment becomes searchable.
9. At flush, Lucene commits and the translog is truncated.

### 8.2 Request Lifecycle — Search

1. HTTP request arrives on the coordinating node.
2. Query is parsed and validated against the index mapping.
3. The coordinating node builds a per-shard execution plan.
4. **Query phase:** request fans out to one copy (primary or replica) of each target shard. Each shard executes the query locally, scores documents, and returns the top-N doc IDs + scores.
5. **Reduce phase:** the coordinating node merges shard results, computing the global top-N.
6. **Fetch phase:** the coordinating node requests the full `_source` (+ highlighted fields, inner hits) for the final top-N documents from their respective shards.
7. The merged, fetched response is returned to the client.

Aggregations plug into steps 4 (shard-local partial reduce) and 5 (global reduce). ES|QL uses a different columnar engine that evaluates on the coordinator.

### 8.3 Coordinator — Quorum Details

Implemented in `server/src/main/java/org/elasticsearch/cluster/coordination/`:

| Class | Role |
|-------|------|
| `Coordinator` | Main coordination loop; runs election, publication |
| `ElectionScheduler` | Times election attempts with configurable back-off |
| `PreVoteCollector` | Runs a pre-vote round before a full election |
| `Publication` | Manages the two-phase commit for a single cluster-state update |
| `ClusterFormationFailureHelper` | Diagnoses and logs cluster-formation problems |
| `FollowersChecker` | Master-side follower-health checking |
| `LeaderChecker` | Follower-side leader-health checking |
| `PeerFinder` | Discovers other master-eligible nodes at startup and on reconnect |

### 8.4 Allocator Internals

`server/src/main/java/org/elasticsearch/cluster/routing/allocation/`:

| Component | Description |
|-----------|-------------|
| `AllocationDecider` | Stack of deciders; each returns YES/NO/THROTTLE for a shard→node assignment |
| `BalancedShardsAllocator` | Balancer that equalizes weighted metrics |
| `DesiredBalanceComputer` | 9.x: async desired-balance solver decoupled from actual moves |
| `ShardsAllocator` | Interface implemented by `BalancedShardsAllocator` |
| `ExistingShardsAllocator` | Handles recovery of shards from existing copies on restart |
| `DiskThresholdDecider` | Disk-watermark enforcement |
| `AwarenessAllocationDecider` | Zone/rack awareness enforcement |
| `FilterAllocationDecider` | `require.*`, `include.*`, `exclude.*` enforcement |
| `ReplicaAfterPrimaryActiveAllocationDecider` | Prevents replica allocation before primary is active |

### 8.5 Lucene Integration

- Each shard holds its own `IndexWriter` (`org.apache.lucene.index.IndexWriter`), configured from `IndexSettings`.
- Segments are mmap'd via `org.apache.lucene.store.MMapDirectory` — they are OS page-cached, not JVM-heap-cached. This is why keeping heap at only 50% of RAM is critical.
- Custom codecs in `server/src/main/java/org/elasticsearch/index/codec/`:
  - BKD-tree numerics (`doc_values` for numeric fields).
  - Bitset compression for sorted set doc values.
  - Keyword ordinals (fast cardinality aggregations).
  - `best_compression` (zstd via Lucene90StoredFieldsFormat).
- The `index.codec` setting maps to a registered Lucene codec factory.

### 8.6 Plugin System

Plugins extend the node via `Plugin` subclasses registered with the server. All plugins are loaded from the `plugins/` directory at startup.

Key extension point interfaces (from `server/src/main/java/org/elasticsearch/plugins/`):

| Interface | What it adds |
|-----------|--------------|
| `ActionPlugin` | REST endpoint handlers (`RestHandler`) and transport actions |
| `AnalysisPlugin` | Custom tokenizers, token filters, char filters, analyzers |
| `IngestPlugin` | Custom ingest processors |
| `MapperPlugin` | Custom field types (`Mapper`, `MetadataFieldMapper`) |
| `SearchPlugin` | Custom queries, suggesters, rescorers, aggregations, fetch sub-phases |
| `DiscoveryPlugin` | Custom seed providers and network transports |
| `RepositoryPlugin` | Custom snapshot repositories |
| `ScriptPlugin` | Custom Painless contexts or script engines |
| `ClusterPlugin` | Custom shard allocation deciders and collectors |
| `PersistentTaskPlugin` | Background tasks managed by the cluster state |
| `IndexStorePlugin` | Custom shard directory implementations |

X-Pack features are all `Plugin` subclasses loaded exactly like third-party plugins. The `x-pack/plugin/core/` module provides shared infrastructure for all X-Pack plugins.

### 8.7 Entitlement System (9.x)

`libs/entitlement/` introduces a runtime capability system that replaces the legacy Java `SecurityManager` (removed in JDK 18+). Plugins declare what they are permitted to do in an `entitlement-policy.yaml` file:

```yaml
# Example entitlement policy
entitlements:
  - type: network_socket
    description: "Allow TCP connections for custom transport"
  - type: file_read
    path: /etc/custom-plugin-config
  - type: file_write
    path: /var/run/custom-plugin
```

The entitlement agent instruments JDK methods at startup and enforces policies at runtime. Internal tooling ships with the build:

- `public-callers-finder` — finds third-party calls to JDK methods that need entitlement.
- `securitymanager-scanner` — finds code still using the old `SecurityManager` API.

### 8.8 Settings System Architecture

From `server/src/main/java/org/elasticsearch/common/settings/`:

- **`Setting<T>`** — a typed, validated setting with metadata (property scope, default value, validator, updater).
- **`Setting.Property`** — `NodeScope` (per node), `IndexScope` (per index), `Dynamic` (updatable without restart), `Deprecated`, `Filterable` (hidden from node info), `Consistent` (must be same across cluster).
- **`AbstractScopedSettings`** — registry for all `Setting` instances. Validates incoming config, applies transactional updates.
- **`ClusterSettings`** — `NodeScope` registry (cluster-dynamic and static node settings).
- **`IndexScopedSettings`** — `IndexScope` registry.
- **Affix settings** — wildcard settings like `node.attr.*` or `s3.client.<name>.endpoint`. Matched at registration and at runtime.
- **Setting key pattern** — `[-\w]+([.][-\w]+)*` (from `AbstractScopedSettings`).
- **Archived settings** — settings that are no longer recognized are prefixed with `archived.` to prevent startup failure while flagging for review.

### 8.9 REST API Architecture

The REST API surface is specified by **600 JSON files** in `rest-api-spec/src/main/resources/rest-api-spec/api/`. Each file describes:

- Endpoint name and handler class
- HTTP methods and path variants
- Path parameters and query parameters (type, required, default)
- Request body schema reference
- Documentation URL

These specs are consumed by:
- The test harness (`rest-api-spec` YAML REST tests)
- Client code generators (Java, Python, JavaScript, Go, Ruby, .NET, PHP, Rust)
- The Elasticsearch documentation build

REST handlers implement `RestHandler` and are registered via `ActionPlugin.getRestHandlers()`. Transport actions implement `TransportAction<Request, Response>` and are the business logic layer invoked by REST handlers (and directly by inter-node transport calls).

### 8.10 Ingest Pipeline Architecture

Ingest pipelines run on `ingest`-capable nodes before documents are indexed. Pipeline execution in `modules/ingest-common/`:

1. The write request specifies `?pipeline=my-pipeline` or a default pipeline is set on the index.
2. The coordinating node selects an ingest node and forwards the request.
3. The ingest node runs the pipeline: each processor receives a `IngestDocument` and transforms it in place.
4. On success, the transformed document is forwarded to the shard primary.
5. On `OnFailure`, the failure handler pipeline runs.

Processor types available (45 built-in from `modules/ingest-common/`):

**Data transformation:** `convert`, `gsub`, `trim`, `uppercase`, `lowercase`, `split`, `join`, `bytes`, `urldecode`, `html_strip`, `uri_parts`, `base64_encode`, `base64_decode`

**Extraction:** `grok`, `dissect`, `csv`, `kv`, `json`, `registered_domain`, `user_agent`, `network_direction`

**Enrichment:** `enrich` (lookup from enrich policy), `geoip` (from `modules/ingest-geoip/`), `ip_location`

**Routing:** `reroute`, `pipeline`, `drop`

**Field manipulation:** `set`, `append`, `remove`, `rename`, `dot_expander`, `date`, `date_index_name`, `fingerprint`, `community_id`

**Control flow:** `fail`, `foreach`, `script`, `set_security_user`, `sort`

### 8.11 Painless Scripting

Painless (`modules/lang-painless/`) is Elasticsearch's built-in scripting language. It compiles to JVM bytecode via ASM. Key characteristics:

- **Sandboxed**: only whitelisted methods and classes are accessible. The whitelist is defined in `modules/lang-painless/src/main/resources/org/elasticsearch/painless/spi/*.txt` files.
- **Statically typed**: type inference with explicit type annotation optional.
- **Optimized for ES**: built-in access to document fields, scoring context, aggregation state.
- **Contexts**: `filter`, `score`, `update`, `ingest`, `field`, `watcher_condition`, `agg_combine`, `agg_reduce`, `transform`.

Example Painless script (scripted field):

```json
{
  "script_fields": {
    "full_name": {
      "script": {
        "source": "doc['first_name'].value + ' ' + doc['last_name'].value"
      }
    }
  }
}
```

Stored scripts:

```http
PUT /_scripts/my-script
{
  "script": {
    "lang": "painless",
    "source": "ctx._source.views += params.increment"
  }
}
```

### 8.12 Aggregation Framework

Aggregations are organized in three tiers:

| Tier | Type | Examples |
|------|------|---------|
| **Bucket** | Group documents into buckets | `terms`, `date_histogram`, `range`, `filters`, `geohash_grid`, `nested`, `parent`, `sampler`, `composite` |
| **Metric** | Compute a value over a bucket | `avg`, `min`, `max`, `sum`, `cardinality`, `percentiles`, `stats`, `extended_stats`, `geo_centroid`, `weighted_avg`, `top_hits`, `scripted_metric` |
| **Pipeline** | Post-process other aggregations | `derivative`, `moving_avg`, `cumulative_sum`, `bucket_sort`, `bucket_selector`, `serial_diff`, `avg_bucket`, `max_bucket`, `min_bucket`, `sum_bucket`, `percentiles_bucket`, `stats_bucket`, `extended_stats_bucket` (39 files, 74K chars in `search/aggregations/pipeline/`) |

Pipeline aggregations (`server/src/main/java/org/elasticsearch/search/aggregations/pipeline/`) are evaluated at the coordinating node after all shards have returned their partial results, enabling multi-pass analysis (e.g., moving averages over a date histogram).

---

## 9. Integration Points

### 9.1 REST API Namespaces

The full 600-endpoint surface spans these namespaces:

| Namespace | Endpoints |
|-----------|----------|
| Document | Index, get, update, delete, bulk, mget, reindex, update_by_query, delete_by_query, termvectors |
| Search | search, msearch, async_search, scroll, pit, knn, search_mvt, terms_enum, count, field_caps, rank_eval, explain, _query (ES\|QL), sql |
| Indices | CRUD, open/close, clone, shrink, split, rollover, downsample, flush, forcemerge, refresh, stats, segments, recovery, settings, mapping, aliases, blocks, templates, resolve, validate, analyze |
| Data Streams | CRUD, promote, modify, lifecycle |
| Cluster | health, health_report, state, settings, stats, pending_tasks, reroute, allocation/explain, voting_config_exclusions, remote/info |
| Nodes | info, stats, hot_threads, usage, reload_secure_settings, shutdown |
| Cat | All `_cat/*` endpoints (aliases, allocation, count, fielddata, health, indices, master, nodeattrs, nodes, pending_tasks, plugins, recovery, repositories, segments, shards, snapshots, tasks, templates, thread_pool, transforms, ml/*, component_templates, circuit_breakers) |
| Snapshot | CRUD repos, CRUD snapshots, restore, verify, status, analyze |
| SLM | CRUD policies, execute, status, stats |
| ILM | CRUD policies, explain, move, retry, start, stop, status |
| Ingest | CRUD pipelines, simulate, geoip databases, ip_location databases |
| Security | user, role, role_mapping, api_key, oauth2/token, saml/*, oidc/*, privilege, service, cross_cluster/api_key, query/*, enroll/*, profile, settings |
| ML | anomaly detection jobs, datafeeds, data frame analytics, trained models, inference, calendars, filters, categories, results, model snapshots, memory, start/stop operations, info, preview |
| Transform | CRUD transforms, start, stop, reset, preview, stats, upgrade |
| Watcher | CRUD watches, execute, activate, deactivate, ack, stats, start, stop, delete history |
| CCR | follow, unfollow, pause, resume, forget, stats, info, auto_follow patterns |
| Rollup | CRUD jobs, start, stop, caps, search, stats |
| Enrich | CRUD policies, execute, stats |
| EQL | search, get, get_status, delete |
| ES\|QL | `_query` (synchronous and async) |
| Connector | CRUD connectors, sync_jobs, check_in, last_sync, cancel_sync |
| Inference | CRUD inference endpoints, `_inference/{task_type}/{id}` |
| Query Rules | CRUD rulesets, test |
| Synonyms | CRUD synonym sets and rules |
| Fleet | global_checkpoints, search, msearch |
| Searchable Snapshots | mount, stats, cache_stats |
| Profiling | flamegraph, stacktraces, topn_functions, status |
| Shutdown | CRUD shutdown markers per node |
| Licensing | get, update, trial, start_basic, start_trial, start_basic, post_start_basic, post_start_trial, delete |
| Features | get, reset |
| Migration | deprecations, system_features |
| Monitoring | bulk |
| Autoscaling | CRUD policies, decisions |
| Streams | CRUD streams |
| Search Application | CRUD search applications, render query, list |
| Dangling Indices | list, import, delete |
| Text Structure | find_structure, find_field_structure, find_message_structure, test_grok_pattern |
| Graph | explore |
| Security (cross-cluster API keys) | create, update, get, query, invalidate |

### 9.2 Client Libraries

Official Elasticsearch clients use the `rest-api-spec` JSON files for code generation and test coverage:

| Language | Package |
|----------|---------|
| Java | `co.elastic.clients:elasticsearch-java` |
| Python | `elasticsearch-py` |
| JavaScript/TypeScript | `@elastic/elasticsearch` |
| Go | `go-elasticsearch` |
| Ruby | `elasticsearch-ruby` |
| .NET | `NEST` / `Elasticsearch.Net` |
| PHP | `elasticsearch-php` |
| Rust | `elasticsearch-rs` |

All official clients implement:
- Automatic node sniffing or static node list.
- Round-robin or custom node selection.
- Retry on node failures.
- Sniff-on-failure.
- Compression.
- Connection pooling.

### 9.3 Kibana Integration

Kibana connects to Elasticsearch as the `kibana_system` built-in user. Its required index patterns:

| Index / Pattern | Purpose |
|----------------|---------|
| `.kibana*` | Kibana saved objects (dashboards, visualizations, index patterns) |
| `.kibana_task_manager*` | Kibana task manager |
| `.kibana_alerting*` | Kibana alerting state |
| `.reporting*` | Kibana reporting |
| `.fleet*` | Fleet (Elastic Agent) enrollment and policy data |
| `.siem-signals*` | Security SIEM detection engine alerts |
| `.lists*` | Security exception lists |
| `metrics-*`, `logs-*`, `traces-*` | Elastic Agent data streams |

The `kibana_system` user must have `manage_security` on the `.kibana*` indices for setup. In production, Kibana should connect to Elasticsearch over TLS using the cluster CA.

### 9.4 Logstash Integration

Logstash uses the `logstash_system` built-in user for monitoring and a separate API key (or `logstash_writer` role) for output writes. The Elasticsearch output plugin supports:

- Bulk writes to data streams or legacy index patterns.
- Retry with exponential backoff on `429`.
- ILM for automatic index rollover.
- Pipeline (ingest pipeline) specification per output.
- TLS verification against the cluster CA.

### 9.5 Elastic Agent and Fleet

Fleet is the central management plane for Elastic Agents. The Fleet server connects to Elasticsearch via a service account token:

```bash
bin/elasticsearch-service-tokens create elastic/fleet-server my-fleet-token
```

Agents enroll with Fleet Server and send data to Elasticsearch data streams (`logs-*`, `metrics-*`, `traces-*`) using API keys created by Fleet on enrollment. Data streams follow the ECS (Elastic Common Schema) naming convention.

### 9.6 Ingest Enrichment Pipelines

Enrich policies allow looking up data from an Elasticsearch index during ingest:

```http
PUT /_enrich/policy/ip-to-geo
{
  "geo_match": {
    "indices":        "ip-ranges",
    "match_field":    "cidr",
    "enrich_fields":  ["country", "org", "asn"]
  }
}
POST /_enrich/policy/ip-to-geo/_execute
```

Then use the `enrich` processor in a pipeline:

```http
PUT /_ingest/pipeline/enrich-geo
{
  "processors": [
    {
      "enrich": {
        "policy_name":     "ip-to-geo",
        "field":           "src_ip",
        "target_field":    "geo",
        "ignore_missing":  true
      }
    }
  ]
}
```

### 9.7 Watcher (Alerting)

Watcher runs scheduled watches that query Elasticsearch, evaluate conditions, and trigger actions:

```http
PUT /_watcher/watch/low-disk-alert
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
  "input": {
    "http": {
      "request": {
        "host":   "localhost",
        "port":   9200,
        "path":   "/_cat/allocation?v&format=json",
        "scheme": "https",
        "auth": { "basic": { "username": "elastic", "password": "{{ctx.metadata.es_password}}" } }
      }
    }
  },
  "condition": {
    "script": {
      "source": "return ctx.payload.any(n -> Double.parseDouble(n.get('disk.percent', '0')) > 80)"
    }
  },
  "actions": {
    "slack_alert": {
      "slack": {
        "message": {
          "to":   ["#ops-alerts"],
          "text": "Disk usage exceeds 80% on at least one node"
        }
      }
    }
  }
}
```

### 9.8 Transforms

Transforms create summary indices from source data. Two modes:

**Pivot transform** (batch or continuous):

```http
PUT /_transform/hourly-agg
{
  "source":  { "index": ["logs-app-*"] },
  "dest":    { "index": "logs-app-hourly" },
  "sync":    { "time": { "field": "@timestamp", "delay": "60s" } },
  "pivot": {
    "group_by": {
      "hour":    { "date_histogram": { "field": "@timestamp", "calendar_interval": "1h" } },
      "service": { "terms": { "field": "service.name" } }
    },
    "aggregations": {
      "event_count": { "value_count": { "field": "_id" } },
      "avg_duration": { "avg": { "field": "duration_ms" } },
      "p99_duration": { "percentiles": { "field": "duration_ms", "percents": [99] } }
    }
  }
}
```

**Latest transform** (takes the latest doc per entity):

```http
PUT /_transform/latest-status
{
  "source": { "index": ["events-*"] },
  "dest":   { "index": "entity-latest-status" },
  "sync":   { "time": { "field": "@timestamp" } },
  "latest": {
    "unique_key": ["entity_id"],
    "sort":       "@timestamp"
  }
}
```

---

## Troubleshooting Reference

> [!NOTE]
> A systematic approach to Elasticsearch problems: (1) check `GET /_cluster/health`, (2) run `GET /_cluster/allocation/explain` for unassigned shards, (3) check `GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected` for saturation, (4) look at `GET /_nodes/hot_threads?threads=5` for hot CPUs, (5) check logs in `path.logs` for ERROR/WARN entries.

### Cluster Health

| Symptom | First command | Common cause |
|---------|--------------|--------------|
| Yellow cluster | `GET /_cluster/allocation/explain` | Replica unassigned due to disk watermark, node down, or awareness constraint |
| Red cluster | `GET /_cluster/allocation/explain` | Primary shard lost; node permanently gone |
| All shards unassigned after restart | `GET /_cluster/state` | `cluster.initial_master_nodes` misconfigured; stale voting config |

> [!CAUTION]
> A red cluster means at least one primary shard is unassigned. Reads and writes to the affected index will return errors. If the primary was on a permanently lost node, recover it with `POST /_cluster/reroute` using `allocate_stale_primary` — but this **acknowledges potential data loss** for writes that were on the lost node between the last flush and failure. Always snapshot before reroute commands on production clusters.

### Node Won't Start — Bootstrap Checks

| Check failure | Fix |
|---------------|-----|
| `vm.max_map_count too low` | `sysctl -w vm.max_map_count=262144` |
| `max file descriptors too low` | `ulimit -n 65535` / `LimitNOFILE=65535` in systemd |
| `memory locking failed` | `ulimit -l unlimited` / `LimitMEMLOCK=infinity` |
| `heap not equal Xms/Xmx` | Set both to the same value |
| `TLS required for multi-node` | Set `xpack.security.transport.ssl.enabled: true` |

### Rejected Execution Exceptions

```http
GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected
```

A rising `rejected` count on `write` = bulk queue full. On `search` = search queue full.

> [!WARNING]
> Raising `queue_size` only delays the problem — it increases memory pressure and hides the real capacity issue. Prefer client-side retry with exponential backoff and horizontal scaling over larger queue sizes.

Fix: add nodes, slow the ingest rate, or use async bulk clients that honor back-pressure.

### Circuit Breaker Exceptions

`circuit_breaking_exception` in the response means a request exceeded a memory breaker. The error message names the specific breaker.

| Breaker tripped | Common cause | Fix |
|----------------|-------------|-----|
| `parent` | Overall heap pressure | Add nodes; reduce `size` in queries |
| `request` | Single large query/aggregation | Use `filter` context; use `search_after`; reduce `size` |
| `fielddata` | Sorting/aggregating on a `text` field | Use `keyword` sub-field; disable fielddata |
| `in_flight_requests` | Very large request bodies | Reduce bulk size; compress requests |

### Slow Queries

1. Enable slow log (see section 3.11).
2. Check `GET /_nodes/hot_threads?threads=5` for hot search threads.
3. `GET /_tasks?actions=*search*&detailed=true` for in-flight slow searches.
4. Use the Profile API:

   ```http
   POST /my-index/_search
   { "profile": true, "query": { ... } }
   ```

   The `profile` response shows time breakdown per query clause and aggregation phase.

### Disk Full — Flood Stage

When the flood-stage watermark is hit, all indices become `read_only_allow_delete`:

1. Free disk (delete old indices, expand volumes, add nodes).
2. Clear the block:

   ```http
   PUT /_all/_settings
   { "index.blocks.read_only_allow_delete": null }
   ```

### Recovery Too Slow

```http
GET /_cat/recovery?v&active_only=true
```

Raise recovery bandwidth:

```http
PUT /_cluster/settings
{ "persistent": { "indices.recovery.max_bytes_per_sec": "200mb" } }
```

Default is `40mb` — conservative. Raise temporarily during deliberate rebalancing.

### Shard Corruption

A red shard that cannot recover may be corrupted. Last resort:

```bash
bin/elasticsearch-shard remove-corrupted-data
```

This deletes the corrupted segment and forces recovery from a replica. If no replica exists, data is lost.

### Split Brain Recovery

If you have two independent clusters with the same `cluster.name` after a partition (extremely rare with ≥ 3 masters), the path to recovery is:

1. Stop all nodes.
2. Identify which side has the most recent data (compare `GET /_cluster/state?filter_path=metadata.cluster_uuid,metadata.version`).
3. On the "winner" nodes, proceed normally.
4. On the "loser" nodes, use `bin/elasticsearch-node detach-cluster` then add them back to the winner cluster as fresh joiners (they will receive cluster state from the master).

### 25+ Common Issues and Resolutions

| # | Issue | Diagnosis command | Resolution |
|---|-------|------------------|------------|
| 1 | Red cluster | `GET /_cluster/allocation/explain` | Restore from snapshot or `allocate_stale_primary` |
| 2 | Yellow cluster (single-node) | `GET /_cluster/health` | Set `number_of_replicas: 0` on single-node clusters |
| 3 | Bootstrap check failure | Node logs at startup | Fix specific OS/JVM parameter (see table above) |
| 4 | `master_not_discovered_exception` | Node logs | Check `discovery.seed_hosts`, network reachability, firewall on port 9300 |
| 5 | Write rejected | `_cat/thread_pool` | Slow ingest, add nodes, increase `write` queue size (last resort) |
| 6 | Search rejected | `_cat/thread_pool` | Slow queries, add nodes, increase `search` queue size (last resort) |
| 7 | Circuit breaker on search | Error message | Reduce `size`, use `filter` context, paginate with `search_after` |
| 8 | Circuit breaker on fielddata | Error message | Use `keyword` sub-field instead of aggregating on `text` |
| 9 | Slow searches | `_nodes/hot_threads`, slow log | Profile, add `filter` context, use routing, check shard count |
| 10 | Disk watermark reached | `_cat/allocation` | Delete data, add nodes, adjust watermark, use ILM/searchable snapshots |
| 11 | Flood stage — write blocked | `_cat/indices` (all read_only) | Free disk, then clear `read_only_allow_delete: null` |
| 12 | Recovery too slow | `_cat/recovery` | Raise `indices.recovery.max_bytes_per_sec` |
| 13 | Shard too large | `_cat/shards` | Use rollover, reduce `max_primary_shard_size` in ILM |
| 14 | Too many shards | `_cat/nodes` (heap warning) | Shrink, reindex, reduce shard count in templates |
| 15 | Mapping explosion | Mapping size grows unbounded | Set `index.mapping.total_fields.limit`, use `dynamic: strict` |
| 16 | Old indices can't be upgraded | `_migration/deprecations` | Reindex to current version before major upgrade |
| 17 | TLS cert expired | TLS handshake errors in logs | Rotate certs with `elasticsearch-certutil`, update keystore |
| 18 | `elastic` password lost | Cannot login | `bin/elasticsearch-reset-password -u elastic` |
| 19 | Keystore password lost | Node won't start | No recovery; regenerate keystore from secrets management |
| 20 | Node stuck in `INITIALIZING` | `_cat/shards` | Check source shard; may be waiting for throttle to release |
| 21 | `split_brain` guard (quorum loss) | Cluster read-only | Restore quorum (add master nodes) or `unsafe-bootstrap` (data-loss risk) |
| 22 | Ingest pipeline failure | `GET /_ingest/pipeline/{name}/_simulate` | Check processor logic; add `on_failure` handler |
| 23 | ILM policy stuck | `GET /_ilm/explain/{index}` | Check step, retry with `POST /_ilm/retry/{index}` |
| 24 | Snapshot repository verification failure | `POST /_snapshot/{repo}/_verify` | Check bucket/container permissions, network access, credentials |
| 25 | CCS remote cluster unreachable | `GET /_remote/info` | Check proxy/seeds address, firewall on 9300/9400, TLS cert validity |
| 26 | ML job anomaly detection failing | `GET /_ml/anomaly_detectors/{job_id}/_stats` | Check node roles for `ml`, check heap availability, check datafeed |
| 27 | Transform failing | `GET /_transform/{id}/_stats` | Check `reason` field; inspect source index mapping changes |

---

## Quick Reference

### Cluster Health One-Liners

```bash
curl -sk -u elastic:$P https://localhost:9200/_cluster/health?pretty
curl -sk -u elastic:$P https://localhost:9200/_cat/nodes?v
curl -sk -u elastic:$P https://localhost:9200/_cat/indices?v&s=store.size:desc
curl -sk -u elastic:$P https://localhost:9200/_cluster/allocation/explain?pretty
curl -sk -u elastic:$P https://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,node
curl -sk -u elastic:$P https://localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected
curl -sk -u elastic:$P https://localhost:9200/_cat/recovery?v&active_only=true
curl -sk -u elastic:$P https://localhost:9200/_nodes/hot_threads?threads=5
curl -sk -u elastic:$P https://localhost:9200/_health_report?pretty
```

### Essential Environment Variables

| Variable | Meaning |
|----------|---------|
| `ES_JAVA_HOME` | JDK to use. Defaults to bundled `jdk/`. |
| `ES_JAVA_OPTS` | Extra JVM flags. Merged with `config/jvm.options`. |
| `ES_PATH_CONF` | Config directory. deb/rpm default: `/etc/elasticsearch`. |
| `ES_TMPDIR` | JVM temp directory. Default: `$ES_HOME/tmp`. |
| `ES_HOME` | Install root. Used by scripts. |
| `ELASTIC_PASSWORD` | Used by Docker entrypoint to set initial `elastic` password. |

### Key Ports

| Port | Role | Access |
|------|------|--------|
| `9200` | HTTP REST API (clients) | Apps and operations — TLS-terminated |
| `9300` | Transport (inter-node) | Cluster-internal only — firewall from apps and internet |
| `5601` | Kibana | Behind SSO / corporate VPN |
| `8200` | APM Server (optional) | Apps → APM |

### CLI Tools Summary

| Tool | Primary use |
|------|------------|
| `bin/elasticsearch` | Start the node |
| `bin/elasticsearch-keystore` | Manage secure settings |
| `bin/elasticsearch-users` | Manage file-realm users |
| `bin/elasticsearch-reset-password` | Reset built-in user passwords |
| `bin/elasticsearch-create-enrollment-token` | Generate enrollment tokens |
| `bin/elasticsearch-certutil` | Generate CA and node TLS certificates |
| `bin/elasticsearch-node` | Offline node maintenance (repurpose, unsafe-bootstrap, detach-cluster) |
| `bin/elasticsearch-shard` | Offline shard repair (remove-corrupted-data) |
| `bin/elasticsearch-plugin` | Install/remove plugins |
| `bin/elasticsearch-service-tokens` | Manage service account tokens |
| `bin/elasticsearch-sql-cli` | Interactive SQL shell |
| `bin/elasticsearch-croneval` | Evaluate Cron expressions (Watcher) |
| `bin/elasticsearch-syskeygen` | Generate Watcher system key |

### License Tiers

| Tier | Key features |
|------|-------------|
| **Basic (free)** | Core engine, security (auth/authz/TLS/audit), ILM, SLM, data streams, kNN, ES\|QL, SQL, monitoring, most Kibana features |
| **Gold** | Alerting, advanced monitoring, LDAP/AD realm |
| **Platinum** | Machine learning anomaly detection, CCR, SAML/OIDC/Kerberos, searchable snapshots (frozen tier), FIPS, Graph, custom roles |
| **Enterprise** | All Platinum features + universal profiling, additional enterprise integrations |

### Version Reference

| Branch | Version | Status |
|--------|---------|--------|
| `main` | 9.5.0 | Current development |
| `9.4` | 9.4.x | Maintenance |
| `9.3` | 9.3.x | Maintenance |
| `9.2` | 9.2.x | Maintenance |
| `8.19` | 8.19.x | Legacy LTS |

---

_Generated May 2026 from elastic/elasticsearch at 7ccc603. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._