Elasticsearch Administrator & Developer Guide

This guide is the complete operational, architectural, and developer reference for Elasticsearch 9.5.0. It is written for engineers who deploy, operate, secure, tune, and extend Elasticsearch in production — from first cluster bring-up through large-scale multi-region deployments. It tracks source commit 7ccc603b (2026-04-24) on the main branch; release branches present in this source tree are 9.4, 9.3, 9.2, and 8.19.

Note

Every configuration setting, command, flag, and API path in this document has been derived from the Elasticsearch source tree at 7ccc603b. No values are fabricated. Where behavior changed between major versions, the relevant version is noted.

Table of Contents

1. Architecture Overview
2. System Requirements
3. Configuration Reference
4. Deployment & Scaling
5. Operations
6. Performance
7. Security
8. Developer Internals
9. Integration Points

1. Architecture Overview

1.1 What Elasticsearch Is

Elasticsearch is a distributed document store and search engine built on Apache Lucene. Documents are JSON objects stored in indices. Each index is horizontally sharded across cluster nodes for scale, and each shard is replicated for fault tolerance. All reads, writes, and administration go through a single HTTP API on port 9200 (TLS by default since 8.0).

Key characteristics:

• Eventually consistent search, strongly consistent writes. Writes succeed after being applied on the primary shard plus the configured replica count. Search reads from the last refreshed segment (default 1 second behind writes).
• No external coordination service. No ZooKeeper, no etcd, no Consul. Master election and cluster-state replication are handled by the in-tree coordinator (server/src/main/java/org/elasticsearch/cluster/coordination/), which uses a Raft-inspired protocol.
• Security on by default. Since 8.0, every fresh single-node and multi-node start auto-generates a CA, node certificates, and a random password for the elastic superuser.
• Files are append-only Lucene segments. Updates are copy-on-write; deletes are logical (tombstones) until a background merge rewrites the segment. Never back up a live cluster by copying shard directories.
• Plugin-based extension model. Every non-core capability — security, ML, SQL, ES|QL, watcher, CCR — is a plugin at the Java level. The boundary between server/ and x-pack/ is a packaging and licensing distinction, not a runtime one.

1.2 Source Tree Layout

The repository (elastic/elasticsearch) is a Gradle monorepo of 40,369 files and 68.6 million characters. Top-level areas:

1.3 Process Model

Each Elasticsearch node is a single JVM process. The startup chain from distribution/tools/server-cli/ is:

1. Locate JDK (bundled at jdk/ unless ES_JAVA_HOME is set).
2. Merge JVM options from config/jvm.options, files under config/jvm.options.d/, and automatic ergonomics (heap sizing from container memory limits if not explicitly set).
3. Launch org.elasticsearch.bootstrap.Elasticsearch, which runs bootstrap checks, binds network ports, and starts subsystems.
4. If telemetry.tracing.enabled: true, attach the APM instrumentation agent.

Once running, a node hosts these concurrent subsystems in-process:

1.4 Node Roles

A node's responsibilities are declared by the node.roles list in elasticsearch.yml. A node may hold multiple roles.

An empty node.roles: [] declares a coordinating-only node: routes client traffic and reduces search results but holds no shards and is never master-eligible.

Recommended layout for a production hot/warm cluster:

3 × dedicated-master  node.roles: [master]
N × hot-data          node.roles: [data_hot, data_content, ingest, transform]
M × warm/cold-data    node.roles: [data_warm, data_cold]
2 × coordinating      node.roles: []                        (optional)
K × ML                node.roles: [ml, remote_cluster_client]  (optional)

1.5 Cluster State and Coordination

The cluster state is a single versioned document held in memory on every node and persisted to disk by the elected master. It contains:

• Cluster metadata (name, UUID)
• Index metadata (mappings, settings, aliases, data stream definitions)
• Routing table (which shard copy lives on which node, shard states)
• Node membership list
• Ingest pipeline definitions
• Index templates and component templates
• ILM policies, SLM policies, transform configs, ML configs
• Security role mappings

Master Election

Masters elect via the coordinator (server/src/main/java/org/elasticsearch/cluster/coordination/):

• Terms — monotonically increasing integers. Every new election bumps the term.
• Voting configuration — set of master-eligible nodes whose majority constitutes a quorum. Evolves with the cluster state as nodes join or leave.
• Publication — two-phase commit: send diff to followers → wait for acknowledgment from quorum → commit. A committed state is durable before the commit returns.
• Follower check — elected master pings all followers periodically. Unreachable followers are removed.
• Leader check — each follower pings the master. If the master is unreachable, followers trigger re-election.
• Bootstrap — cluster.initial_master_nodes is consumed only when nodes boot with no existing cluster state. Remove it from all configs after the cluster first forms.

Important

cluster.initial_master_nodes must be set only once, at initial cluster formation, and must name only the nodes physically present at that moment. After the cluster forms, remove this setting from every node's elasticsearch.yml. If left in place it can cause a split-brain when those named nodes are later replaced.

Shard Allocation

The allocator runs as a stack of AllocationDecider implementations followed by a balancer:

1. Hard constraints — index.routing.allocation.require.*, include.*, exclude.*, node role compatibility, version compatibility, data-tier filters.
2. Disk watermarks — stop placing new shards at low (85%), begin migrating existing shards at high (90%), block writes at flood_stage (95%).
3. Zone awareness — cluster.routing.allocation.awareness.attributes: zone spreads replicas across declared failure domains.
4. Balance scoring — BalancedShardsAllocator equalizes weighted shard count, write load, and disk usage across nodes.
5. Desired balance (9.x) — asynchronous desired-balance computation decouples re-planning from actual shard moves, improving responsiveness under load.

Diagnose any unassigned shard:

GET /_cluster/allocation/explain

1.6 Storage: Indices, Shards, and Segments

An index is a logical namespace split into one or more primary shards. Each primary is a self-contained Lucene index. Writes always go to the primary first, then propagate to replicas. Reads can be served by any live copy.

Lucene Segment Lifecycle

1. Documents are written to an in-memory indexing buffer and simultaneously to the translog (write-ahead log on disk).
2. At refresh (index.refresh_interval, default 1s), the buffer flushes to a new on-disk segment and becomes searchable.
3. Segments are immutable. Updates write a new version; the old version is tombstoned. Deletes mark tombstones.
4. Background merges consolidate small segments into larger ones, physically removing tombstoned documents.
5. At flush, Lucene commits and the translog is truncated.

Shard Count Trade-offs

Target shard size: 10–50 GB of post-merge on-disk data. Query-heavy catalogs: lower end. Bulk log archives: upper end.

Maximum shards per node: aim for ≤ 20 × heap_GB. Hard ceiling ~1,000–2,000 shards per node before GC and state-management pressure becomes significant.

1.7 Module and Plugin Inventory

Core Server Packages (key areas)

X-Pack Plugins (commercial, Elastic License)

Bundled Modules

Optional Plugins

Shared Libraries (libs/)

1.8 Build System

Elasticsearch uses Gradle with internal build plugins under build-tools-internal/:

2. System Requirements

2.1 Minimum Requirements (Development)

2.2 Production Per-Node Baseline

2.3 Required Kernel Parameters

# /etc/sysctl.d/99-elasticsearch.conf
vm.max_map_count=262144
vm.swappiness=1

Apply without reboot: sysctl -w vm.max_map_count=262144

2.4 Required System Limits

In /etc/security/limits.conf or systemd unit override:

elasticsearch  -  nofile   65535
elasticsearch  -  nproc    4096
elasticsearch  -  memlock  unlimited

systemd:

[Service]
LimitNOFILE=65535
LimitNPROC=4096
LimitMEMLOCK=infinity

2.5 Heap Sizing

Warning

Never set JVM heap above ~30.5 GB. Above this threshold the JVM switches from 4-byte compressed ordinary object pointers to 8-byte full pointers, effectively wasting 30–40% of heap. Add nodes rather than growing heap past this boundary. Set -Xms and -Xmx to the same value to prevent runtime resizing pauses.

# config/jvm.options.d/heap.options
-Xms30g
-Xmx30g

Or via environment: ES_JAVA_OPTS="-Xms16g -Xmx16g"

The remaining RAM (the ~50% not allocated to heap) is reserved for the OS filesystem page cache backing Lucene's memory-mapped segment files.

2.6 Swap

Disable completely:

swapoff -a
# Remove swap entries from /etc/fstab for persistence

Set bootstrap.memory_lock: true in elasticsearch.yml. The node will refuse to start if the lock fails — fix the ulimit, not the config.

2.7 Storage Recommendations

• Local NVMe for data_hot. Network-attached storage introduces latency spikes that impact GC and write durability.
• XFS or ext4 with noatime mount option.
• Avoid BTRFS (copy-on-write semantics conflict with Lucene's update model).
• path.data directory must be owned by the Elasticsearch OS user with 0700 permissions.

2.8 Docker Constraints

• Set a container memory limit; heap ergonomics reads cgroup memory limits.
• --ulimit memlock=-1:-1 and --ulimit nofile=65535:65535
• Set vm.max_map_count=262144 on the host — cannot be set inside the container.
• Use a named volume or hostPath PVC for path.data.

3. Configuration Reference

Elasticsearch has two configuration surfaces:

1. elasticsearch.yml — static, per-node, read at startup.
2. Cluster settings — dynamic, cluster-wide, applied live via PUT /_cluster/settings. Use persistent (survives full restart) rather than transient (lost on full cluster restart; deprecated).

3.1 Node Identity

cluster.name: prod-logs
node.name: es-hot-01

3.2 Node Roles

node.roles:
  - data_hot
  - data_content
  - ingest
  - transform

Valid role values: master, data, data_hot, data_warm, data_cold, data_frozen, data_content, ingest, ml, transform, remote_cluster_client, voting_only. Empty list = coordinating-only.

3.3 Custom Node Attributes

node.attr.zone:     us-east-1a
node.attr.rack:     rack-7
node.attr.storage:  nvme

Referenced by allocation filters and awareness. Visible in _cat/nodeattrs and GET /_nodes.

3.4 Paths

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo:
  - /mnt/snapshots

path.data may be a YAML list to stripe across multiple disks.

Note

When path.data contains multiple paths, Elasticsearch stripes new shards across them in round-robin. It does not mirror — if one path fails, the shards on that path are lost. Rely on shard replicas across nodes for redundancy, not multiple local paths.

3.5 Networking

network.host: _site_
http.port: 9200
transport.port: 9300
http.cors.enabled: false
http.compression: true
http.max_content_length: 100mb

network.host special values:

To publish a different address than bound (NAT / Docker):

network.host: _local_
network.publish_host: 10.0.1.42

3.6 Discovery and Cluster Formation

discovery.type: multi-node
discovery.seed_hosts:
  - "es-master-01.internal"
  - "es-master-02.internal:9300"
cluster.initial_master_nodes:
  - "es-master-01"
  - "es-master-02"
  - "es-master-03"

3.7 Memory and Bootstrap

bootstrap.memory_lock: true
bootstrap.system_call_filter: true

Both are verified at startup; failure aborts the node. Set bootstrap.system_call_filter: false only on kernels that do not support seccomp.

3.8 Allocation and Rebalancing (Cluster Dynamic Settings)

3.9 Index-Level Settings

Applied in templates or at index creation time. Dynamic settings can be updated via PUT /{index}/_settings:

3.10 Thread Pools

Thread pool sizes are auto-tuned from CPU count. Override only with profiling data:

3.11 Slow Logs

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn":   "10s",
  "index.search.slowlog.threshold.query.info":   "5s",
  "index.search.slowlog.threshold.query.debug":  "2s",
  "index.search.slowlog.threshold.query.trace":  "500ms",
  "index.search.slowlog.threshold.fetch.warn":   "1s",
  "index.indexing.slowlog.threshold.index.warn": "10s",
  "index.indexing.slowlog.level":                "info"
}

Logs emit as structured JSON to logs/<cluster>_index_search_slowlog.json and logs/<cluster>_index_indexing_slowlog.json.

3.12 Runtime Log Level Changes

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.discovery": "DEBUG"
  }
}

3.13 JVM Options

Put overrides in config/jvm.options.d/ (never edit config/jvm.options directly):

# config/jvm.options.d/custom.options
-Xms30g
-Xmx30g
-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=30
-XX:G1ReservePercent=25
-XX:+AlwaysPreTouch
-Djava.io.tmpdir=/var/lib/elasticsearch/tmp
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10

3.14 Keystore (Secure Settings)

bin/elasticsearch-keystore create [-p]
bin/elasticsearch-keystore add <setting>
bin/elasticsearch-keystore add-file <setting> <file>
bin/elasticsearch-keystore list
bin/elasticsearch-keystore remove <setting>
bin/elasticsearch-keystore passwd
bin/elasticsearch-keystore upgrade

Reload without restart:

POST /_nodes/reload_secure_settings
{ "secure_settings_password": "optional-keystore-password" }

Common secure settings:

3.15 Docker Environment Variable Convention

From EnvironmentAwareCommand.java:

• ES_SETTING_ prefix variables are translated: remove prefix, replace _ with ., unescape .. back to _, lowercase.
• ES_SETTING_CLUSTER__NAME=prod-logs → cluster.name: prod-logs
• ES_SETTING_NODE__NAME=es-01 → node.name: es-01
• Lowercase dot-separated variables also work directly in the Docker entrypoint.

3.16 Plugin Configuration File

# config/elasticsearch-plugins.yml
plugins:
  - id: analysis-icu
  - id: custom-plugin
    location: https://example.com/my-plugin-9.5.0.zip
  - id: maven-plugin
    location: org.example:my-plugin:1.0.0
  - id: proxied-plugin
    location: https://example.com/plugin.zip
    proxy: https://proxy.internal:3128
proxy: https://proxy.internal:3128

Manual management:

bin/elasticsearch-plugin install analysis-icu
bin/elasticsearch-plugin install https://example.com/plugin.zip
bin/elasticsearch-plugin remove analysis-icu
bin/elasticsearch-plugin list

4. Deployment & Scaling

4.1 Distribution Packages

Elasticsearch ships in three packaging families from distribution/:

4.2 Single-Node Development

# elasticsearch.yml
cluster.name: dev-local
node.name: dev-node-1
discovery.type: single-node
xpack.security.enabled: false

Start: bin/elasticsearch (foreground) or bin/elasticsearch -d -p /var/run/elasticsearch.pid (daemon).

4.3 Multi-Node Cluster Bootstrap

Step 1 — Prepare elasticsearch.yml on each node:

cluster.name: prod-logs
node.name: es-master-01          # unique per node
node.roles: [master]
network.host: 10.0.1.11          # unique per node
discovery.seed_hosts:
  - "10.0.1.11"
  - "10.0.1.12"
  - "10.0.1.13"
cluster.initial_master_nodes:
  - "es-master-01"
  - "es-master-02"
  - "es-master-03"
bootstrap.memory_lock: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/transport.p12

Step 2 — Generate certificates (on one node, copy to all):

bin/elasticsearch-certutil ca --out elastic-stack-ca.p12 --pass
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --dns es-master-01,es-master-02,es-master-03 \
  --out bundle.zip

Step 3 — Start all nodes, verify cluster formation:

curl -sk -u elastic:$PASSWORD https://es-master-01:9200/_cluster/health?pretty

Step 4 — Remove cluster.initial_master_nodes from all configs and do a rolling restart.

4.4 Docker Compose (Three-Node + Kibana)

The reference compose topology from the source tree:

version: "3.8"
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip &&
        unzip config/certs/ca.zip -d config/certs &&
        bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip \
          --in config/certs/instances.yml \
          --ca-cert config/certs/ca/ca.crt \
          --ca-key config/certs/ca/ca.key &&
        unzip config/certs/certs.zip -d config/certs
      '

  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200

  kibana:
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    ports:
      - ${KIBANA_PORT}:5601
    volumes:
      - certs:/usr/share/kibana/config/certs

volumes:
  certs:
  esdata01:
  esdata02:
  esdata03:

4.5 Kubernetes (ECK)

The officially supported operator is Elastic Cloud on Kubernetes (ECK). It manages:

• Cluster lifecycle (create, scale, upgrade)
• Rolling upgrades with health gate checks
• TLS certificate generation and rotation
• Storage volume management
• Secret management for passwords and API keys

Minimum ECK manifest for a 3-node cluster:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: prod-logs
spec:
  version: 9.5.0
  nodeSets:
  - name: masters
    count: 3
    config:
      node.roles: ["master", "voting_only"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: standard
  - name: hot-data
    count: 3
    config:
      node.roles: ["data_hot", "data_content", "ingest"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Gi
        storageClassName: fast-nvme

For bare StatefulSets without ECK, you must manually handle cert rotation, Pod Disruption Budgets, anti-affinity rules, and node naming stability.

4.6 Data Tiers

Elasticsearch supports five data tiers. Each tier is a node.roles value; indices target a tier via index.routing.allocation.include._tier_preference.

Tier-aware index template:

PUT /_index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-policy",
      "index.routing.allocation.include._tier_preference": "data_hot"
    }
  }
}

4.7 ILM — Index Lifecycle Management

ILM automates the hot → warm → cold → frozen → delete progression:

PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "set_priority": { "priority": 50 },
          "readonly": {},
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "14d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "found-snapshots" }
        }
      },
      "frozen": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "found-snapshots" }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

ILM management commands:

GET /_ilm/explain/{index}          # Current phase, action, step
POST /_ilm/retry/{index}           # Retry after a failed step
POST /_ilm/move/{index}            # Manually advance to a specific step
POST /_ilm/stop                    # Pause ILM globally
POST /_ilm/start                   # Resume ILM
GET /_ilm/status                   # ILM running/stopped state

4.8 Data Stream Lifecycle (Simpler Alternative to ILM)

For pure retention + downsampling without the full ILM state machine:

PUT /_data_stream/logs-app-prod/_lifecycle
{
  "data_retention": "90d",
  "downsampling": [
    { "after": "7d",  "fixed_interval": "1h" },
    { "after": "30d", "fixed_interval": "1d" }
  ]
}

4.9 Snapshots and Restore

Snapshots are the only supported backup mechanism. Do not copy shard directories off a live cluster.

Register a Repository

S3:

PUT /_snapshot/my-s3
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-snapshots",
    "base_path": "prod",
    "client": "default",
    "server_side_encryption": true
  }
}

S3 credentials in the keystore:

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
curl -u elastic:$P https://localhost:9200/_nodes/reload_secure_settings -X POST

Azure:

PUT /_snapshot/my-azure
{
  "type": "azure",
  "settings": {
    "container": "my-es-snapshots",
    "base_path": "prod"
  }
}

GCS:

PUT /_snapshot/my-gcs
{
  "type": "gcs",
  "settings": {
    "bucket": "my-es-snapshots",
    "base_path": "prod",
    "client": "default"
  }
}

Shared filesystem (NFS — prefer object storage):

PUT /_snapshot/my-fs
{
  "type": "fs",
  "settings": {
    "location": "/mnt/snapshots/prod",
    "compress": true
  }
}

Supported repository types: fs, url (read-only), s3, azure, gcs, hdfs (plugin), source (source-only, compressed archival).

SLM — Automated Snapshot Policies

PUT /_slm/policy/daily-snapshots
{
  "schedule":   "0 30 1 * * ?",
  "name":       "<daily-snap-{now/d}>",
  "repository": "my-s3",
  "config": {
    "indices":              ["*"],
    "include_global_state": true,
    "ignore_unavailable":   true
  },
  "retention": {
    "expire_after": "30d",
    "min_count":    5,
    "max_count":    50
  }
}

Restore

POST /_snapshot/my-s3/daily-snap-2026-04-22/_restore
{
  "indices":              "logs-app-*",
  "rename_pattern":       "(.+)",
  "rename_replacement":   "restored-$1",
  "include_global_state": false
}

Searchable Snapshots

Cold/frozen indices can be searchable snapshots — data lives only in the repository and is read on demand. ILM integrates this via the searchable_snapshot action. No replicas are stored locally, so capacity per node can be enormous. The x-pack/plugin/blob-cache/ (56K chars) provides the local disk cache layer for repeated access.

4.10 Cross-Cluster Search (CCS)

Register remote clusters:

PUT /_cluster/settings
{
  "persistent": {
    "cluster.remote.eu-prod.mode":          "proxy",
    "cluster.remote.eu-prod.proxy_address": "eu-prod-proxy.example.com:9400",
    "cluster.remote.us-logs.mode":          "sniff",
    "cluster.remote.us-logs.seeds":         ["us-logs-1.example.com:9300"]
  }
}

• Sniff mode — discovers all nodes in the remote cluster and opens direct connections. Requires transport port access.
• Proxy mode — multiplexes all traffic through a single proxy address. Required for cross-VPC/cross-k8s traffic without exposing the transport port.

Search across clusters:

GET /eu-prod:logs-*,us-logs:logs-*,logs-*/_search
{
  "query": { "match": { "message": "error" } }
}

Use ?ccs_minimize_roundtrips=true (the default) — fetches only the final top-N docs from remote clusters, reducing round trips significantly.

4.11 Cross-Cluster Replication (CCR)

CCR streams writes from a leader index to a follower in another cluster in near-real-time.

Use cases:

• Disaster recovery — follower in a secondary region; promote on primary failure.
• Read scaling — local followers in app-adjacent regions serve local read traffic.
• Aggregation — centralize indices from many leader clusters into a single analysis cluster.

Create a follower:

PUT /logs-app-follower/_ccr/follow
{
  "remote_cluster":    "eu-prod",
  "leader_index":      "logs-app-000001",
  "settings":          { "index.number_of_replicas": 0 }
}

Follower indices are read-only on the follower side. To promote (e.g., on DR failover):

POST /logs-app-follower/_ccr/pause_follow
POST /logs-app-follower/_ccr/unfollow

5. Operations

5.1 Health Monitoring

GET /_cluster/health
GET /_cluster/health?level=indices
GET /_health_report                           # richer multi-dimensional health (9.x)
GET /_cluster/state?filter_path=metadata.indices.*.state

Health status:

Key diagnostic endpoints:

Tip

Use GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected as your first diagnostic when users report search or indexing slowness. A rising rejected count on the search or write pool is the earliest measurable signal of capacity pressure — it appears before latency visibly degrades.

5.2 Rolling Restart (Zero Downtime)

1. Disable replica allocation to prevent unnecessary shard movement:

   PUT /_cluster/settings
   { "persistent": { "cluster.routing.allocation.enable": "primaries" } }
   

2. Stop indexing if possible, then flush:

   POST /_flush
   

3. Stop the target node, upgrade/patch it, start it back.

4. Wait until the node rejoins:

   GET /_cat/nodes?v
   

5. Re-enable allocation:

   PUT /_cluster/settings
   { "persistent": { "cluster.routing.allocation.enable": null } }
   

6. Wait for green health:

   GET /_cluster/health?wait_for_status=green&timeout=5m
   

7. Move to the next node.

Note

Always upgrade/restart data nodes first, master-eligible nodes last to preserve quorum throughout the window. After re-enabling allocation, peer recovery will replay translog entries and reallocate replicas — throttle recovery bandwidth with indices.recovery.max_bytes_per_sec if the cluster serves production traffic during the window.

5.3 Decommissioning a Node

Route all shards off the node before stopping it:

PUT /_cluster/settings
{ "persistent": { "cluster.routing.allocation.exclude._name": "es-warm-09" } }

Monitor until 0 shards remain:

GET /_cat/allocation?v

For a master-eligible node, also add to voting-config exclusions before stopping:

POST /_cluster/voting_config_exclusions?node_names=es-master-02&timeout=1m

After decommissioning, clear the exclusion:

DELETE /_cluster/voting_config_exclusions

5.4 Graceful Shutdown API (9.x)

The shutdown API allows controlled node removal with proper wait logic:

PUT /_nodes/<node_id>/shutdown
{
  "type":             "remove",     # or "restart", "replace"
  "reason":           "AZ drain 2026-04-23",
  "target_node_name": "es-warm-99"  # for "replace" type
}

Monitor shutdown progress:

GET /_nodes/<node_id>/shutdown

5.5 Shard Shrink, Split, and Reindex

Shrink (reduce shard count — target must divide evenly):

# First: mark source read-only and route to a single node
PUT /source-index/_settings
{
  "index.blocks.write": true,
  "index.routing.allocation.require._name": "es-hot-01"
}
# Then: shrink
POST /source-index/_shrink/target-index
{ "settings": { "index.number_of_shards": 1 } }

Split (increase shard count — target must be a multiple of source):

PUT /source-index/_settings
{ "index.blocks.write": true }
POST /source-index/_split/target-index
{ "settings": { "index.number_of_shards": 4 } }

Reindex (arbitrary shard count change or cross-cluster copy):

POST /_reindex
{
  "source": { "index": "old-index" },
  "dest":   { "index": "new-index" }
}

5.6 Recovering a Lost Master Quorum (Last Resort)

If you permanently lose a majority of master-eligible nodes with their data:

# On the surviving node with the most recent cluster state:
bin/elasticsearch-node unsafe-bootstrap --confirm

Caution

unsafe-bootstrap forms a new cluster from a single node. Any cluster-state changes that were applied on the lost masters but not yet acknowledged by this node are permanently lost. This includes new indices, mapping changes, security configs, and ILM policies added since the last successful publication. Always snapshot first. Restore from snapshots after bootstrapping if necessary.

5.7 Fixing Flood-Stage Write Block

When the flood-stage watermark is hit, all indices become read-only:

1. Free disk first (delete old indices, add nodes, expand volumes).

2. Clear the write block:

   PUT /_all/_settings
   { "index.blocks.read_only_allow_delete": null }
   

Running the clear before freeing disk will cause the block to re-apply immediately.

5.8 Force Merge

POST /my-index/_forcemerge?max_num_segments=1

Caution

Only run _forcemerge on read-only indices (ILM warm/cold, closed index, write-blocked). Force-merging active write indices destroys the merge policy geometry, can cause large GC spikes, and the I/O can saturate the node. max_num_segments=1 is appropriate only for truly static data.

5.9 APM and Telemetry

telemetry.agent.server_url: https://apm-server.internal:8200
telemetry.secret_token: secret-token-here        # or telemetry.api_key
telemetry.tracing.enabled: true
telemetry.metrics.enabled: true

APM tracing emits spans for search, indexing, and cluster-state operations. Useful for tracing slow queries from the ES side rather than the client side.

5.10 Prometheus Metrics

The x-pack/plugin/prometheus/ plugin exposes a Prometheus-compatible scrape endpoint. After installing or enabling the plugin, metrics are served at GET /_prometheus/metrics. This covers node stats, index stats, JVM, thread pools, circuit breakers, and more in Prometheus exposition format.

5.11 Monitoring Stack Options

5.12 Upgrade Procedures

Supported Upgrade Paths

• Minor upgrades (e.g., 9.4 → 9.5): rolling, online, no reindex required.
• Major upgrades (e.g., 8.x → 9.x): rolling upgrade from the last minor of the previous major (e.g., 8.19 → 9.0 directly). Earlier minors require upgrading to the last minor first.
• Index compatibility: indices created in version N are compatible with N and N+1 major only. Indices older than that must be reindexed or retired before a major upgrade.

Check readiness:

GET /_migration/deprecations

Rolling Upgrade Procedure (Minor)

1. Read release notes and resolve critical deprecations (GET /_migration/deprecations).
2. Disable shard allocation for replicas: cluster.routing.allocation.enable: primaries.
3. For each node: stop → upgrade package/image → start → wait for rejoin.
4. Re-enable allocation after each node, wait for green.
5. Upgrade master-eligible nodes last.
6. Verify: GET / returns the new version on all nodes.

Major Upgrade Notes

• Run mixed major versions for as short a window as possible. The cluster tolerates current major and current major - 1 simultaneously during the rolling window.
• Test against the Upgrade Assistant in Kibana (reads _migration/deprecations and guides through remediation) well before the maintenance window.
• Snapshot the cluster before starting.

6. Performance

6.1 Indexing Optimization

Bulk requests:

• Target 5–15 MB per bulk body and hundreds to low thousands of documents per request.
• Monitor rejected on the write thread pool; back off with exponential retry on 429.
• Use async bulk clients (the Elasticsearch Java client, Python async client) to pipeline requests.

Refresh interval:

PUT /my-index/_settings
{ "index.refresh_interval": "30s" }

Set to 30s or -1 during initial bulk loads; return to 1s (or the default) afterward.

Translog durability:

PUT /my-index/_settings
{ "index.translog.durability": "async" }

async batches fsyncs to disk (interval controlled by index.translog.sync_interval: 5s). Throughput can improve 2× at the cost of up to sync_interval of data on a hard crash. Acceptable for log ingestion; not for financial transactions.

Replicas during bulk load:

PUT /my-index/_settings
{ "index.number_of_replicas": 0 }
# ... bulk index ...
PUT /my-index/_settings
{ "index.number_of_replicas": 1 }

Auto-generated IDs:

Let Elasticsearch assign _id when possible — avoids a "does this document exist?" lookup on the shard, saving one read per document.

Segment merges:

Do not over-force-merge. One _forcemerge?max_num_segments=1 at the end of a bulk load on a static dataset is fine. Continuous or repeated force-merging is pathological.

Codec:

PUT /_index_template/logs-compressed
{
  "template": {
    "settings": { "index.codec": "best_compression" }
  }
}

best_compression (zstd) reduces storage 10–30% at a modest CPU cost. Best for warm/cold indices where search speed matters less.

6.2 Search Optimization

Filters vs. queries:

Structure-matched constraints (term, range, exists) in filter context — they are cached and skip relevance scoring:

{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "status": "active" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "must": [
        { "match": { "message": "error" } }
      ]
    }
  }
}

Routing:

If your access pattern is always tenant=X, route documents by tenant:

POST /my-index/_doc?routing=tenant-acme
{ ... }

GET /my-index/_search?routing=tenant-acme
{ ... }

Queries with the matching routing only hit the one shard that holds that tenant's data.

Shard request cache:

On by default. Effective for size: 0 (aggregation-only) queries against immutable data. Invalidated on every refresh — raise index.refresh_interval to extend cache lifetime for static data.

Field data and keyword fields:

Avoid sorting or aggregating on text fields — fielddata is disabled by default and loads the entire column into heap. Use a keyword sub-field:

{
  "mappings": {
    "properties": {
      "message": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" } }
      }
    }
  }
}

Pagination patterns:

Point-in-time (PIT):

POST /my-index/_pit?keep_alive=5m
# Returns: { "id": "PIT_ID" }

GET /_search
{
  "pit": { "id": "PIT_ID", "keep_alive": "5m" },
  "sort": [{ "@timestamp": "desc" }, { "_shard_doc": "asc" }],
  "search_after": [<last_sort_values>]
}

preference=_local:

For latency-sensitive queries from coordinating nodes that also hold shards, use ?preference=_local to avoid a network hop.

6.3 Memory and GC Tuning

• Heap: 50% of RAM, max ~30.5 GB. Never exceed this.
• G1GC (-XX:+UseG1GC) is the default. -XX:InitiatingHeapOccupancyPercent=30 triggers GC sooner to avoid long pauses.
• -XX:+AlwaysPreTouch pre-faults heap pages at startup, reducing GC latency spikes in early operation.
• Keep the filesystem page cache large. Never run Elasticsearch on a node that also hosts a page-cache-heavy service (database, HDFS, etc.).
• Disable swap: swapoff -a + vm.swappiness=1 + bootstrap.memory_lock: true.
• Monitor GET /_nodes/stats/jvm for heap_used_percent. Sustained > 75% is a warning; > 85% needs action (add nodes or reduce load).

6.4 Circuit Breakers

Circuit breakers prevent OOM by tracking memory used by operations. Key breakers:

Check breaker state:

GET /_nodes/stats/breaker

A circuit_breaking_exception means a single request exceeded the breaker limit. Fix by reducing request size, using filter rather than query, paginating with search_after, or raising the specific breaker limit carefully.

6.5 Dense Vector (kNN) Performance

• Use dense_vector field type with dims matching your embedding model output.

• For large corpora, use index: true and HNSW indexing (m and ef_construction parameters). Default m=16, ef_construction=100.

• Query with knn inside _search:

  {
    "knn": {
      "field":          "embedding",
      "query_vector":   [0.1, 0.2, ...],
      "k":              10,
      "num_candidates": 100
    }
  }
  

• num_candidates controls recall vs. speed trade-off per shard. Higher = more accurate, more CPU.

• SIMD acceleration is provided by libs/simdvec/ on x86_64 (AVX2/AVX-512) and aarch64 (NEON). Requires a CPU that supports these; falls back to scalar otherwise.

6.6 Benchmarking

Use Rally (github.com/elastic/rally) — the official benchmarking harness. Built-in tracks include nyc_taxis, http_logs, geopoint, nested, eql, dense_vectors. Build custom tracks from a snapshot of your data and a representative query log.

For micro-benchmarks of core algorithms, benchmarks/ contains JMH benchmarks used by the core team.

7. Security

7.1 Security Model Overview

Security is on by default from 8.0 onward. A freshly started single node auto-configures:

• TLS on both transport (9300) and HTTP (9200) layers.
• A CA and per-node certificate.
• A random password for the elastic superuser, printed to the terminal.
• An enrollment token for Kibana and joining nodes.

Warning

Never store TLS passwords, API secrets, or LDAP bind passwords in elasticsearch.yml. These would be readable by any process with filesystem access. Always use bin/elasticsearch-keystore add for sensitive values. The keystore is encrypted at rest and only decrypted in memory by the running node.

7.2 Security Feature Toggles

7.3 TLS Configuration

Transport TLS (mandatory for multi-node)

xpack.security.transport.ssl.enabled:            true
xpack.security.transport.ssl.verification_mode:  certificate
xpack.security.transport.ssl.keystore.path:      certs/transport.p12
xpack.security.transport.ssl.truststore.path:    certs/transport.p12

Passwords in the keystore:

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

HTTP TLS (client-facing)

xpack.security.http.ssl.enabled:            true
xpack.security.http.ssl.keystore.path:      certs/http.p12
xpack.security.http.ssl.keystore.type:      PKCS12

HTTP SSL settings reference:

7.4 Certificate Management

# Generate a CA
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12 --pass

# Generate a node certificate from the CA
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --dns node-1.example.com,node-2.example.com \
  --ip 10.0.1.11,10.0.1.12 \
  --out bundle.zip

# Interactive wizard for HTTP TLS
bin/elasticsearch-certutil http

# Generate a CSR for external CA signing
bin/elasticsearch-certutil csr \
  --name node-1 \
  --dns node-1.example.com \
  --out csr.zip

For Kubernetes/ECK, certificate management is fully automated by the operator.

7.5 Authentication Realms

Realms are authentication providers stacked in priority order under xpack.security.authc.realms:

Example realm stack:

xpack.security.authc.realms:
  native:
    native1:
      order: 0
  ldap:
    corporate_ldap:
      order: 1
      url: "ldaps://ldap.example.com:636"
      bind_dn: "cn=elastic-bind,dc=example,dc=com"
      user_search:
        base_dn: "dc=example,dc=com"
      group_search:
        base_dn: "dc=example,dc=com"
      files:
        role_mapping: "config/role_mapping.yml"
  pki:
    pki1:
      order: 2
      certificate_authorities: ["certs/ca.crt"]

7.6 Authorization — Roles

Roles are sets of cluster privileges, index privileges, application privileges, and optional FLS/DLS.

PUT /_security/role/logs_reader
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names":      ["logs-*"],
      "privileges": ["read", "view_index_metadata"],
      "field_security": {
        "grant":  ["@timestamp", "service", "message", "level"],
        "except": []
      },
      "query": "{\"term\":{\"tenant\":\"acme\"}}"
    }
  ]
}

Cluster privileges (common):

Index privileges (common):

7.7 Users

# Create
POST /_security/user/alice
{
  "password":  "changeme",
  "roles":     ["logs_reader"],
  "full_name": "Alice Smith",
  "email":     "[email removed]"
}

# Change password
PUT /_security/user/alice/_password
{ "password": "newpassword" }

# Disable
PUT /_security/user/alice/_disable

# Delete
DELETE /_security/user/alice

7.8 API Keys

API keys are preferred for application credentials. They are scoped, expirable, and can be invalidated without rotating passwords.

# Create an API key
POST /_security/api_key
{
  "name":       "my-app-key",
  "expiration": "30d",
  "role_descriptors": {
    "my-app-role": {
      "indices": [{ "names": ["logs-*"], "privileges": ["read"] }]
    }
  }
}
# Response includes "id" and "api_key" (base64-encoded joined as id:api_key)

# Use in requests:
# Authorization: ApiKey <base64(id:api_key)>

# Invalidate
DELETE /_security/api_key
{ "id": "KEY_ID" }

# Query all active keys
GET /_security/api_key?username=alice

7.9 Service Accounts

Service accounts are non-user security principals for infrastructure services (Fleet, Kibana, Logstash). They use file-backed or index-backed tokens, not passwords.

bin/elasticsearch-service-tokens create elastic/fleet-server my-fleet-token
bin/elasticsearch-service-tokens list elastic/fleet-server
bin/elasticsearch-service-tokens delete elastic/fleet-server my-fleet-token

7.10 Audit Logging

xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include:
  - access_denied
  - authentication_failed
  - run_as_denied
  - tampered_request
  - connection_denied
  - anonymous_access_denied
xpack.security.audit.logfile.events.exclude: []

Audit logs emit as structured JSON to logs/<cluster>_audit.json. Fields include @timestamp, node.id, event.type, event.action, user.name, request.id, url.path, network.client.ip.

7.11 Field-Level and Document-Level Security (FLS/DLS)

FLS — restrict which fields a role can read:

"field_security": {
  "grant":  ["@timestamp", "service", "level"],
  "except": []
}

DLS — restrict which documents a role can read (via a query):

"query": "{\"term\":{\"tenant\":\"acme\"}}"

FLS/DLS are evaluated per-segment at query time. They add overhead proportional to the complexity of the DLS query. Avoid DLS on frequently-updated large indices.

7.12 FIPS Mode

xpack.security.fips_mode.enabled: true
xpack.security.fips_mode.required_providers:
  - "BCFIPS"
  - "BCJSSE"

Requires:

• A FIPS 140-2 validated JVM (Bouncy Castle FIPS or an NSS-backed JDK).
• All keystores must be BCFKS format (not PKCS12 or JKS).
• Passwords must be stored in the ES keystore; the keystore itself must be BCFKS.
• Build with elasticsearch.fips Gradle plugin for distribution-level FIPS compliance.

7.13 Network Isolation

Caution

Transport port (9300) is internal-only. Firewall it from the internet and from application networks. Only Elasticsearch nodes and monitoring sidecars should reach it. Never expose a cleartext port 9200 to the public internet. Always terminate TLS at Elasticsearch itself or at a re-encrypting load balancer — not a TLS-terminating proxy that communicates plaintext to ES.

Recommended network segmentation:

Internet
  → Reverse proxy (TLS termination, optional WAF)
  → Elasticsearch HTTP port 9200 (TLS, internal network only)

Application servers
  → Elasticsearch HTTP port 9200 (TLS, internal network only)

Elasticsearch nodes
  ↔ Transport port 9300 (TLS, cluster-internal network only, firewall from all other networks)

8. Developer Internals

8.1 Request Lifecycle — Write

1. HTTP request arrives on a coordinating node's Netty HTTP pipeline.
2. The REST handler decodes the request and dispatches to TransportBulkAction (or TransportIndexAction for single-doc writes).
3. The coordinating node resolves the target index/alias, resolves aliases, and computes the shard per document using _routing (defaults to _id).
4. Per shard, the write is forwarded to the primary shard's node over the transport layer.
5. The primary applies the write to its Lucene IndexWriter and the translog.
6. The primary fan-outs to replica shards, waits for wait_for_active_shards acknowledgments.
7. The response returns to the coordinating node and then to the client.
8. At the next refresh (periodic or explicit), the new segment becomes searchable.
9. At flush, Lucene commits and the translog is truncated.

8.2 Request Lifecycle — Search

1. HTTP request arrives on the coordinating node.
2. Query is parsed and validated against the index mapping.
3. The coordinating node builds a per-shard execution plan.
4. Query phase: request fans out to one copy (primary or replica) of each target shard. Each shard executes the query locally, scores documents, and returns the top-N doc IDs + scores.
5. Reduce phase: the coordinating node merges shard results, computing the global top-N.
6. Fetch phase: the coordinating node requests the full _source (+ highlighted fields, inner hits) for the final top-N documents from their respective shards.
7. The merged, fetched response is returned to the client.

Aggregations plug into steps 4 (shard-local partial reduce) and 5 (global reduce). ES|QL uses a different columnar engine that evaluates on the coordinator.

8.3 Coordinator — Quorum Details

Implemented in server/src/main/java/org/elasticsearch/cluster/coordination/:

8.4 Allocator Internals

server/src/main/java/org/elasticsearch/cluster/routing/allocation/:

8.5 Lucene Integration

• Each shard holds its own IndexWriter (org.apache.lucene.index.IndexWriter), configured from IndexSettings.
• Segments are mmap'd via org.apache.lucene.store.MMapDirectory — they are OS page-cached, not JVM-heap-cached. This is why keeping heap at only 50% of RAM is critical.
• Custom codecs in server/src/main/java/org/elasticsearch/index/codec/:
  • BKD-tree numerics (doc_values for numeric fields).
  • Bitset compression for sorted set doc values.
  • Keyword ordinals (fast cardinality aggregations).
  • best_compression (zstd via Lucene90StoredFieldsFormat).
• The index.codec setting maps to a registered Lucene codec factory.

8.6 Plugin System

Plugins extend the node via Plugin subclasses registered with the server. All plugins are loaded from the plugins/ directory at startup.

Key extension point interfaces (from server/src/main/java/org/elasticsearch/plugins/):

X-Pack features are all Plugin subclasses loaded exactly like third-party plugins. The x-pack/plugin/core/ module provides shared infrastructure for all X-Pack plugins.

8.7 Entitlement System (9.x)

libs/entitlement/ introduces a runtime capability system that replaces the legacy Java SecurityManager (removed in JDK 18+). Plugins declare what they are permitted to do in an entitlement-policy.yaml file:

# Example entitlement policy
entitlements:
  - type: network_socket
    description: "Allow TCP connections for custom transport"
  - type: file_read
    path: /etc/custom-plugin-config
  - type: file_write
    path: /var/run/custom-plugin

The entitlement agent instruments JDK methods at startup and enforces policies at runtime. Internal tooling ships with the build:

• public-callers-finder — finds third-party calls to JDK methods that need entitlement.
• securitymanager-scanner — finds code still using the old SecurityManager API.

8.8 Settings System Architecture

From server/src/main/java/org/elasticsearch/common/settings/:

• Setting<T> — a typed, validated setting with metadata (property scope, default value, validator, updater).
• Setting.Property — NodeScope (per node), IndexScope (per index), Dynamic (updatable without restart), Deprecated, Filterable (hidden from node info), Consistent (must be same across cluster).
• AbstractScopedSettings — registry for all Setting instances. Validates incoming config, applies transactional updates.
• ClusterSettings — NodeScope registry (cluster-dynamic and static node settings).
• IndexScopedSettings — IndexScope registry.
• Affix settings — wildcard settings like node.attr.* or s3.client.<name>.endpoint. Matched at registration and at runtime.
• Setting key pattern — [-\w]+([.][-\w]+)* (from AbstractScopedSettings).
• Archived settings — settings that are no longer recognized are prefixed with archived. to prevent startup failure while flagging for review.

8.9 REST API Architecture

The REST API surface is specified by 600 JSON files in rest-api-spec/src/main/resources/rest-api-spec/api/. Each file describes:

• Endpoint name and handler class
• HTTP methods and path variants
• Path parameters and query parameters (type, required, default)
• Request body schema reference
• Documentation URL

These specs are consumed by:

• The test harness (rest-api-spec YAML REST tests)
• Client code generators (Java, Python, JavaScript, Go, Ruby, .NET, PHP, Rust)
• The Elasticsearch documentation build

REST handlers implement RestHandler and are registered via ActionPlugin.getRestHandlers(). Transport actions implement TransportAction<Request, Response> and are the business logic layer invoked by REST handlers (and directly by inter-node transport calls).

8.10 Ingest Pipeline Architecture

Ingest pipelines run on ingest-capable nodes before documents are indexed. Pipeline execution in modules/ingest-common/:

1. The write request specifies ?pipeline=my-pipeline or a default pipeline is set on the index.
2. The coordinating node selects an ingest node and forwards the request.
3. The ingest node runs the pipeline: each processor receives a IngestDocument and transforms it in place.
4. On success, the transformed document is forwarded to the shard primary.
5. On OnFailure, the failure handler pipeline runs.

Processor types available (45 built-in from modules/ingest-common/):

Data transformation: convert, gsub, trim, uppercase, lowercase, split, join, bytes, urldecode, html_strip, uri_parts, base64_encode, base64_decode

Extraction: grok, dissect, csv, kv, json, registered_domain, user_agent, network_direction

Enrichment: enrich (lookup from enrich policy), geoip (from modules/ingest-geoip/), ip_location

Routing: reroute, pipeline, drop

Field manipulation: set, append, remove, rename, dot_expander, date, date_index_name, fingerprint, community_id

Control flow: fail, foreach, script, set_security_user, sort

8.11 Painless Scripting

Painless (modules/lang-painless/) is Elasticsearch's built-in scripting language. It compiles to JVM bytecode via ASM. Key characteristics:

• Sandboxed: only whitelisted methods and classes are accessible. The whitelist is defined in modules/lang-painless/src/main/resources/org/elasticsearch/painless/spi/*.txt files.
• Statically typed: type inference with explicit type annotation optional.
• Optimized for ES: built-in access to document fields, scoring context, aggregation state.
• Contexts: filter, score, update, ingest, field, watcher_condition, agg_combine, agg_reduce, transform.

Example Painless script (scripted field):

{
  "script_fields": {
    "full_name": {
      "script": {
        "source": "doc['first_name'].value + ' ' + doc['last_name'].value"
      }
    }
  }
}

Stored scripts:

PUT /_scripts/my-script
{
  "script": {
    "lang": "painless",
    "source": "ctx._source.views += params.increment"
  }
}

8.12 Aggregation Framework

Aggregations are organized in three tiers:

Pipeline aggregations (server/src/main/java/org/elasticsearch/search/aggregations/pipeline/) are evaluated at the coordinating node after all shards have returned their partial results, enabling multi-pass analysis (e.g., moving averages over a date histogram).

9. Integration Points

9.1 REST API Namespaces

The full 600-endpoint surface spans these namespaces:

9.2 Client Libraries

Official Elasticsearch clients use the rest-api-spec JSON files for code generation and test coverage:

All official clients implement:

• Automatic node sniffing or static node list.
• Round-robin or custom node selection.
• Retry on node failures.
• Sniff-on-failure.
• Compression.
• Connection pooling.

9.3 Kibana Integration

Kibana connects to Elasticsearch as the kibana_system built-in user. Its required index patterns:

The kibana_system user must have manage_security on the .kibana* indices for setup. In production, Kibana should connect to Elasticsearch over TLS using the cluster CA.

9.4 Logstash Integration

Logstash uses the logstash_system built-in user for monitoring and a separate API key (or logstash_writer role) for output writes. The Elasticsearch output plugin supports:

• Bulk writes to data streams or legacy index patterns.
• Retry with exponential backoff on 429.
• ILM for automatic index rollover.
• Pipeline (ingest pipeline) specification per output.
• TLS verification against the cluster CA.

9.5 Elastic Agent and Fleet

Fleet is the central management plane for Elastic Agents. The Fleet server connects to Elasticsearch via a service account token:

bin/elasticsearch-service-tokens create elastic/fleet-server my-fleet-token

Agents enroll with Fleet Server and send data to Elasticsearch data streams (logs-*, metrics-*, traces-*) using API keys created by Fleet on enrollment. Data streams follow the ECS (Elastic Common Schema) naming convention.

9.6 Ingest Enrichment Pipelines

Enrich policies allow looking up data from an Elasticsearch index during ingest:

PUT /_enrich/policy/ip-to-geo
{
  "geo_match": {
    "indices":        "ip-ranges",
    "match_field":    "cidr",
    "enrich_fields":  ["country", "org", "asn"]
  }
}
POST /_enrich/policy/ip-to-geo/_execute

Then use the enrich processor in a pipeline:

PUT /_ingest/pipeline/enrich-geo
{
  "processors": [
    {
      "enrich": {
        "policy_name":     "ip-to-geo",
        "field":           "src_ip",
        "target_field":    "geo",
        "ignore_missing":  true
      }
    }
  ]
}

9.7 Watcher (Alerting)

Watcher runs scheduled watches that query Elasticsearch, evaluate conditions, and trigger actions:

PUT /_watcher/watch/low-disk-alert
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
  "input": {
    "http": {
      "request": {
        "host":   "localhost",
        "port":   9200,
        "path":   "/_cat/allocation?v&format=json",
        "scheme": "https",
        "auth": { "basic": { "username": "elastic", "password": "{{ctx.metadata.es_password}}" } }
      }
    }
  },
  "condition": {
    "script": {
      "source": "return ctx.payload.any(n -> Double.parseDouble(n.get('disk.percent', '0')) > 80)"
    }
  },
  "actions": {
    "slack_alert": {
      "slack": {
        "message": {
          "to":   ["#ops-alerts"],
          "text": "Disk usage exceeds 80% on at least one node"
        }
      }
    }
  }
}

9.8 Transforms

Transforms create summary indices from source data. Two modes:

Pivot transform (batch or continuous):

PUT /_transform/hourly-agg
{
  "source":  { "index": ["logs-app-*"] },
  "dest":    { "index": "logs-app-hourly" },
  "sync":    { "time": { "field": "@timestamp", "delay": "60s" } },
  "pivot": {
    "group_by": {
      "hour":    { "date_histogram": { "field": "@timestamp", "calendar_interval": "1h" } },
      "service": { "terms": { "field": "service.name" } }
    },
    "aggregations": {
      "event_count": { "value_count": { "field": "_id" } },
      "avg_duration": { "avg": { "field": "duration_ms" } },
      "p99_duration": { "percentiles": { "field": "duration_ms", "percents": [99] } }
    }
  }
}

Latest transform (takes the latest doc per entity):

PUT /_transform/latest-status
{
  "source": { "index": ["events-*"] },
  "dest":   { "index": "entity-latest-status" },
  "sync":   { "time": { "field": "@timestamp" } },
  "latest": {
    "unique_key": ["entity_id"],
    "sort":       "@timestamp"
  }
}

Troubleshooting Reference

Note

A systematic approach to Elasticsearch problems: (1) check GET /_cluster/health, (2) run GET /_cluster/allocation/explain for unassigned shards, (3) check GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected for saturation, (4) look at GET /_nodes/hot_threads?threads=5 for hot CPUs, (5) check logs in path.logs for ERROR/WARN entries.

Cluster Health

Caution

A red cluster means at least one primary shard is unassigned. Reads and writes to the affected index will return errors. If the primary was on a permanently lost node, recover it with POST /_cluster/reroute using allocate_stale_primary — but this acknowledges potential data loss for writes that were on the lost node between the last flush and failure. Always snapshot before reroute commands on production clusters.

Node Won't Start — Bootstrap Checks

Rejected Execution Exceptions

GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected

A rising rejected count on write = bulk queue full. On search = search queue full.

Warning

Raising queue_size only delays the problem — it increases memory pressure and hides the real capacity issue. Prefer client-side retry with exponential backoff and horizontal scaling over larger queue sizes.

Fix: add nodes, slow the ingest rate, or use async bulk clients that honor back-pressure.

Circuit Breaker Exceptions

circuit_breaking_exception in the response means a request exceeded a memory breaker. The error message names the specific breaker.

Slow Queries

1. Enable slow log (see section 3.11).

2. Check GET /_nodes/hot_threads?threads=5 for hot search threads.

3. GET /_tasks?actions=*search*&detailed=true for in-flight slow searches.

4. Use the Profile API:

   POST /my-index/_search
   { "profile": true, "query": { ... } }
   

   The profile response shows time breakdown per query clause and aggregation phase.

Disk Full — Flood Stage

When the flood-stage watermark is hit, all indices become read_only_allow_delete:

1. Free disk (delete old indices, expand volumes, add nodes).

2. Clear the block:

   PUT /_all/_settings
   { "index.blocks.read_only_allow_delete": null }
   

Recovery Too Slow

GET /_cat/recovery?v&active_only=true

Raise recovery bandwidth:

PUT /_cluster/settings
{ "persistent": { "indices.recovery.max_bytes_per_sec": "200mb" } }

Default is 40mb — conservative. Raise temporarily during deliberate rebalancing.

Shard Corruption

A red shard that cannot recover may be corrupted. Last resort:

bin/elasticsearch-shard remove-corrupted-data

This deletes the corrupted segment and forces recovery from a replica. If no replica exists, data is lost.

Split Brain Recovery

If you have two independent clusters with the same cluster.name after a partition (extremely rare with ≥ 3 masters), the path to recovery is:

1. Stop all nodes.
2. Identify which side has the most recent data (compare GET /_cluster/state?filter_path=metadata.cluster_uuid,metadata.version).
3. On the "winner" nodes, proceed normally.
4. On the "loser" nodes, use bin/elasticsearch-node detach-cluster then add them back to the winner cluster as fresh joiners (they will receive cluster state from the master).

25+ Common Issues and Resolutions

Quick Reference

Cluster Health One-Liners

curl -sk -u elastic:$P https://localhost:9200/_cluster/health?pretty
curl -sk -u elastic:$P https://localhost:9200/_cat/nodes?v
curl -sk -u elastic:$P https://localhost:9200/_cat/indices?v&s=store.size:desc
curl -sk -u elastic:$P https://localhost:9200/_cluster/allocation/explain?pretty
curl -sk -u elastic:$P https://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,node
curl -sk -u elastic:$P https://localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected
curl -sk -u elastic:$P https://localhost:9200/_cat/recovery?v&active_only=true
curl -sk -u elastic:$P https://localhost:9200/_nodes/hot_threads?threads=5
curl -sk -u elastic:$P https://localhost:9200/_health_report?pretty

Essential Environment Variables

Key Ports

CLI Tools Summary

License Tiers

Version Reference