# Apache Spark Administrator Guide

This guide is the operational and architectural reference for **Apache Spark** — for the engineers, architects, and platform operators who deploy, secure, tune, and run Spark in production. It covers the runtime architecture, the cluster deployment models (Standalone, YARN, Kubernetes), the complete configuration surface, security hardening, monitoring, and the internals a team needs to operate Spark at scale.

End-user material (the data APIs, shells, and `spark-submit` usage) lives in the companion **[User Guide](apache__spark__UserGuide.md)**.

> [!NOTE]
> Generated from the development (`master`) branch, self-reporting version **5.0.0-SNAPSHOT** (PySpark `5.0.0.dev0`), built against **Scala 2.13.18** and **Java 17+**. Configuration defaults and feature availability reflect this branch at the documented commit. Released lines (4.x and earlier, also actively built in this tree) may differ — verify against your release.

---

## 1. Executive Summary

Apache Spark is a distributed, general-purpose analytics engine. A Spark application consists of a **driver** process that runs the user program and coordinates work, and a set of **executor** processes that run tasks and hold cached data. A pluggable **cluster manager** allocates the executors.

**Operational model at a glance:**

- **One engine, four cluster managers.** The same application JAR/script runs unchanged on `local`, Standalone, YARN, or Kubernetes — only `--master` and a handful of manager-specific properties change.
- **Configuration is layered.** Site defaults (`conf/spark-defaults.conf`) < command-line `--conf` < programmatic `SparkConf`. Environment-level concerns (memory ceilings, daemon ports, log dirs) come from `conf/spark-env.sh`.
- **Observability is built in.** A live per-application Web UI (default port 4040), event logs replayed by the **History Server** (default 18080), and a metrics system with sinks for JMX, Prometheus, Graphite, StatsD, CSV, and console.
- **Security is opt-in but comprehensive.** RPC authentication (`spark.authenticate`), AES network encryption (`spark.network.crypto.enabled`), local I/O encryption (`spark.io.encryption.enabled`), SSL for web UIs, and Kerberos for Hadoop/YARN.

**The single most important sizing decision** is executor geometry: cores per executor (`spark.executor.cores`), memory per executor (`spark.executor.memory` + overhead), and executor count (`spark.executor.instances` or dynamic allocation). The rest of this guide explains how those interact with the scheduler, the memory manager, and the shuffle subsystem.

---

## 2. Architecture

### 2.1 Runtime topology

```
                         ┌──────────────────────────────────────┐
                         │             DRIVER PROCESS            │
                         │  ┌────────────────────────────────┐   │
   User program ───────▶ │  │ SparkSession / SparkContext    │   │
                         │  ├────────────────────────────────┤   │
                         │  │ DAGScheduler  (stages, RDD/plan)│   │
                         │  │ TaskScheduler (TaskSetManager)  │   │
                         │  │ SchedulerBackend                │   │
                         │  │ BlockManager (master)           │   │
                         │  │ Web UI  :4040                   │   │
                         │  └───────────────┬────────────────┘   │
                         └──────────────────┼────────────────────┘
                                            │ (RPC: Netty)
                  ┌─────────────────────────┼─────────────────────────┐
                  │            CLUSTER MANAGER (one of):                │
                  │   local │ Standalone Master │ YARN RM │ K8s API     │
                  └─────────────────────────┼─────────────────────────┘
                                            │ launches
              ┌─────────────────────────────┼─────────────────────────────┐
              ▼                              ▼                             ▼
     ┌─────────────────┐          ┌─────────────────┐          ┌─────────────────┐
     │   EXECUTOR 1    │          │   EXECUTOR 2    │   ...    │   EXECUTOR N    │
     │  Task threads   │          │  Task threads   │          │  Task threads   │
     │  BlockManager   │          │  BlockManager   │          │  BlockManager   │
     │  Memory (Unified)│         │  Shuffle files  │          │  Cached blocks  │
     └─────────────────┘          └─────────────────┘          └─────────────────┘
```

### 2.2 The execution pipeline

1. **Logical plan.** DataFrame/SQL operations build a logical plan; the RDD API builds an RDD lineage DAG.
2. **Catalyst optimization** (SQL/DataFrame). The Catalyst optimizer rewrites the logical plan (predicate pushdown, column pruning, join reordering with cost-based optimization when `spark.sql.cbo.enabled=true`) and produces a physical plan. **Tungsten** generates whole-stage codegen for tight, cache-friendly execution.
3. **DAGScheduler** splits the plan into **stages** at shuffle boundaries and submits **TaskSets**.
4. **TaskSchedulerImpl** + **SchedulerBackend** assign tasks to executors, respecting locality, the scheduling pool (FIFO/FAIR), and resource profiles. **TaskSetManager** handles retries, speculation, and blacklisting/excludes.
5. **Executors** run tasks, read/write shuffle data via their **BlockManager**, and report heartbeats and metrics back to the driver.
6. **Adaptive Query Execution (AQE)** (`spark.sql.adaptive.enabled`, default on) re-optimizes at runtime between stages — coalescing shuffle partitions, switching join strategies, and splitting skewed partitions using actual statistics.

### 2.3 Memory model (Unified Memory Manager)

Each executor's JVM heap is divided into:

- **Reserved memory** — fixed overhead.
- **Unified region** (`spark.memory.fraction`, default `0.6` of usable heap) — shared between **execution** memory (shuffles, joins, aggregations, sorts) and **storage** memory (cached/persisted blocks). The boundary floats; `spark.memory.storageFraction` (default `0.5`) sets the storage region that is immune to eviction.
- **User memory** — the remainder, for user data structures.
- **Off-heap** — optional, enabled by `spark.memory.offHeap.enabled` with size `spark.memory.offHeap.size`.

On top of the heap, the cluster manager reserves **overhead** memory (`spark.executor.memoryOverhead` / `spark.executor.memoryOverheadFactor`, default factor `0.1`) for the off-heap JVM footprint, Python/R workers, and shuffle buffers.

> [!IMPORTANT]
> Out-of-memory failures on YARN/Kubernetes are frequently *overhead* exhaustion, not heap exhaustion — especially with PySpark (each executor forks Python workers, sized by `spark.executor.pyspark.memory`). If containers are killed for exceeding memory limits, raise `spark.executor.memoryOverhead` before raising `spark.executor.memory`.

### 2.4 Shuffle subsystem

Shuffles redistribute data across partitions (after `groupBy`, `join`, `repartition`). Map tasks write shuffle files locally; reduce tasks fetch them. Key controls:

- `spark.shuffle.manager` — the shuffle implementation.
- `spark.shuffle.compress`, `spark.shuffle.file.buffer` — compression and write buffering.
- `spark.shuffle.checksum.enabled` / `.algorithm` — corruption detection.
- **Push-based shuffle** (`spark.shuffle.push.enabled`) — merges shuffle blocks on remote mergers to reduce small-block random I/O at scale.
- **External Shuffle Service** — lets executors be removed (dynamic allocation, decommissioning) without losing their shuffle files.

---

## 3. Component Inventory

Spark is a large multi-module Maven build (`spark-parent_2.13`). The principal modules:

| Module (path) | Role |
|---|---|
| `core/` | The engine: `SparkContext`, scheduler (`DAGScheduler`/`TaskSchedulerImpl`), `BlockManager`, RDD API, deploy code (Standalone master/worker), shuffle. |
| `sql/catalyst/` | Catalyst — logical plans, analysis, optimizer rules, expressions, the SQL parser. |
| `sql/core/` | DataFrame/Dataset API, physical execution, data sources (Parquet/ORC/JSON/CSV/JDBC), AQE. |
| `sql/api/` | Stable public SQL API surface shared by classic and Connect. |
| `sql/hive/`, `sql/hive-thriftserver/` | Hive catalog integration and the Thrift JDBC/ODBC server (HiveServer2-compatible). |
| `sql/connect/` | Spark Connect — gRPC server and clients (Scala JVM + JDBC). |
| `sql/pipelines/` | Declarative Pipelines engine. |
| `streaming/` | Structured Streaming integration and the legacy DStream API. |
| `mllib/`, `mllib-local/` | Machine learning (`spark.ml` pipelines, `spark.mllib` RDD API, local linear algebra). |
| `graphx/` | Graph processing and built-in algorithms (`org.apache.spark.graphx.lib`). |
| `connector/` | Pluggable connectors: `kafka-0-10`(-sql/-token-provider), `kinesis-asl`, `avro`, `protobuf`, `profiler`, `spark-ganglia-lgpl`. |
| `resource-managers/yarn/`, `.../kubernetes/` | Cluster-manager backends for YARN and Kubernetes. |
| `common/` | Shared libraries: `network-common`, `network-shuffle`, `network-yarn`, `kvstore`, `unsafe`, `sketch`, `variant`, `utils`, `tags`. |
| `launcher/` | Programmatic + script launch (`SparkLauncher`, `SparkSubmitOptionParser`). |
| `repl/` | The Scala interactive shell. |
| `python/`, `R/` | PySpark and SparkR. |
| `hadoop-cloud/` | Cloud object-store (S3A/GCS/ABFS) integration. |

---

## 4. System Requirements

| Component | Requirement |
|---|---|
| **JDK** | Java 17 minimum; Java 21 and 25 are also built/tested. Set `JAVA_HOME` on every node. |
| **Scala** | 2.13.18 (source builds and Scala/Java apps). |
| **Python** | 3.11–3.14 for PySpark (incl. free-threaded `nogil`). Pin with `PYSPARK_PYTHON`. |
| **Memory** | Plan per-node: `SPARK_WORKER_MEMORY` (Standalone) ≥ Σ(executor heap + overhead). Daemons default to `SPARK_DAEMON_MEMORY=1g`. |
| **Storage** | Fast local disk for shuffle/spill (`SPARK_LOCAL_DIRS`); shared/object storage (HDFS, S3) for data and event logs. |
| **Network** | Low-latency intra-cluster network; open the RPC and Web UI ports below. |
| **Hadoop** | Optional; must match the cluster's Hadoop line if using HDFS/YARN. |

### 4.1 Default ports

| Service | Default port | Notes |
|---|---|---|
| Standalone Master RPC | `7077` | `spark://host:7077`; override `SPARK_MASTER_PORT`. |
| Standalone Master Web UI | `8080` | `SPARK_MASTER_WEBUI_PORT`. |
| Standalone Worker Web UI | `8081` | `SPARK_WORKER_WEBUI_PORT`. |
| Application Web UI | `4040` | Increments (4041, …) if busy; lives only while the app runs. |
| History Server | `18080` | Replays event logs. |
| Thrift JDBC/ODBC server | `10000` | HiveServer2-compatible. |
| Spark Connect server | `15002` | gRPC endpoint (`sc://host:15002`). |
| External Shuffle Service | `7337` | When enabled. |
| Driver / BlockManager | random | Pin with `spark.driver.port` / `spark.blockManager.port` behind firewalls. |

---

## 5. Deployment Models

### 5.1 Local mode

No daemons. `--master local[*]` runs driver and executors in one JVM. For development, CI, and small jobs.

### 5.2 Standalone cluster

Spark's built-in cluster manager. Start a master and workers with the `sbin/` scripts.

```bash
# On the master node
./sbin/start-master.sh
# → master at spark://<host>:7077, UI at http://<host>:8080

# On each worker node (point at the master)
./sbin/start-worker.sh spark://<master-host>:7077
```

Or, with a `conf/workers` file listing worker hostnames, launch the whole cluster from the master:

```bash
./sbin/start-all.sh      # master + all workers in conf/workers
./sbin/stop-all.sh
```

Individual daemons: `start-master.sh`/`stop-master.sh`, `start-worker.sh`/`stop-worker.sh`, `start-workers.sh`/`stop-workers.sh`, `workers.sh <cmd>` (run a command on every worker), `decommission-worker.sh`, and the generic `spark-daemon.sh` / `spark-daemons.sh`.

**Standalone configuration** (set in `conf/spark-env.sh`):

| Variable | Purpose |
|---|---|
| `SPARK_MASTER_HOST` | Bind the master to a specific host/IP. |
| `SPARK_MASTER_PORT` / `SPARK_MASTER_WEBUI_PORT` | Non-default master ports. |
| `SPARK_WORKER_CORES` | Cores this worker offers to executors. |
| `SPARK_WORKER_MEMORY` | Total memory this worker offers (e.g. `64g`). |
| `SPARK_WORKER_PORT` / `SPARK_WORKER_WEBUI_PORT` | Non-default worker ports. |
| `SPARK_WORKER_DIR` | Worker scratch/work directory. |
| `SPARK_MASTER_OPTS` / `SPARK_WORKER_OPTS` | Extra `-Dspark.x=y` for master/worker only. |

**High availability & recovery** — `spark.deploy.recoveryMode`:

| Mode | Backing store | Use |
|---|---|---|
| `NONE` (default) | — | No master failover. |
| `FILESYSTEM` | `spark.deploy.recoveryDirectory` | Single-master recovery from a shared dir. |
| `ZOOKEEPER` | `spark.deploy.zookeeper.url` / `.dir` | True multi-master HA via leader election. |
| `ROCKSDB` | local RocksDB | Embedded persistent recovery store. |
| `CUSTOM` | `spark.deploy.recoveryMode.factory` | Pluggable factory. |

Other standalone knobs: `spark.deploy.defaultCores`, `spark.deploy.maxExecutorRetries`, `spark.deploy.spreadOutApps`, `spark.deploy.retainedApplications`, `spark.deploy.retainedDrivers`, `spark.deploy.workerSelectionPolicy`, `spark.cores.max`, `spark.worker.timeout`.

### 5.3 YARN

```bash
export HADOOP_CONF_DIR=/etc/hadoop/conf
./bin/spark-submit --master yarn --deploy-mode cluster \
  --num-executors 20 --executor-cores 4 --executor-memory 8g \
  --queue production --class com.example.App app.jar
```

Key YARN properties (`resource-managers/yarn`): `spark.yarn.am.memory` / `.cores` / `.memoryOverhead` (client-mode AM), `spark.yarn.queue`, `spark.yarn.archive` (a pre-uploaded jars archive — avoids re-uploading Spark on every submit), `spark.yarn.am.nodeLabelExpression`, `spark.yarn.am.waitTime`, and the `spark.yarn.cache.*` distributed-cache settings. Kerberos: `--principal` + `--keytab` (the AM renews delegation tokens).

> [!IMPORTANT]
> Always set `spark.yarn.archive` (or `spark.yarn.jars`) in production. Without it, Spark uploads its entire jar set to HDFS on **every** submission, adding latency and load.

### 5.4 Kubernetes

```bash
./bin/spark-submit --master k8s://https://<api-server>:6443 \
  --deploy-mode cluster --name my-job \
  --conf spark.kubernetes.container.image=myrepo/spark:5.0.0 \
  --conf spark.executor.instances=10 \
  local:///opt/app/app.jar
```

Build images with `bin/docker-image-tool.sh`. Key K8s properties (`resource-managers/kubernetes`):

| Property | Purpose |
|---|---|
| `spark.kubernetes.container.image` | Image for driver+executor (or `.driver.container.image` / executor variant). |
| `spark.kubernetes.container.image.pullPolicy` / `.pullSecrets` | Image pull behavior and registry secrets. |
| `spark.kubernetes.namespace` | Target namespace. |
| `spark.kubernetes.authenticate.driver.serviceAccountName` | Driver service account (RBAC). |
| `spark.kubernetes.allocation.batch.size` / `.delay` | Executor pod allocation pacing. |
| `spark.kubernetes.allocation.maxPendingPods` | Cap on in-flight pending pods. |
| `spark.kubernetes.driver.pod.featureSteps` / executor `.pod.featureSteps` | Pluggable pod customization. |
| `spark.kubernetes.driver.ownPersistentVolumeClaim` | PVC lifecycle tied to driver. |
| `spark.kubernetes.decommission.script` | Graceful executor decommission hook. |

> [!NOTE]
> Use the `local://` URI scheme for application resources baked into the image. The Kubernetes backend supports executor pod templates, PVC reuse, and graceful decommissioning for spot/preemptible nodes.

### 5.5 Dynamic allocation

Let Spark scale executors with workload (works on all cluster managers; requires shuffle tracking or an external shuffle service so shuffle data survives executor removal):

| Property | Default | Purpose |
|---|---|---|
| `spark.dynamicAllocation.enabled` | `false` | Turn on autoscaling. |
| `spark.dynamicAllocation.minExecutors` | `0` | Lower bound. |
| `spark.dynamicAllocation.maxExecutors` | ∞ | Upper bound. |
| `spark.dynamicAllocation.initialExecutors` | = min | Starting count. |
| `spark.dynamicAllocation.executorIdleTimeout` | `60s` | Idle executor removal. |
| `spark.dynamicAllocation.schedulerBacklogTimeout` | `1s` | Backlog before requesting more. |
| `spark.dynamicAllocation.shuffleTracking.enabled` | `false` | Track shuffle files without an external service. |
| `spark.dynamicAllocation.executorAllocationRatio` | `1.0` | Aggressiveness of scale-up. |

### 5.6 Executor decommissioning

Graceful removal (spot instances, scale-down) migrates shuffle/cache blocks off a doomed executor: `spark.decommission.enabled`, `spark.executor.decommission.killInterval`, `spark.executor.decommission.forceKillTimeout`, `spark.executor.decommission.signal`.

---

## 6. Configuration Reference

Configuration sources, in increasing precedence: `conf/spark-defaults.conf` → `--properties-file` → `--conf` / programmatic `SparkConf`. The core engine alone defines **530+** `spark.*` properties (plus the full `spark.sql.*`, `spark.yarn.*`, and `spark.kubernetes.*` families). The tables below cover the high-traffic operational settings; for an exhaustive list, the authoritative source is the `ConfigBuilder` definitions in `core/src/main/scala/org/apache/spark/internal/config/` and the per-module config packages.

### 6.1 Application & driver/executor

| Property | Default | Purpose |
|---|---|---|
| `spark.master` | `local[*]` | Cluster manager URL. |
| `spark.app.name` | — | Application name. |
| `spark.submit.deployMode` | `client` | `client` or `cluster`. |
| `spark.driver.memory` | `1g` | Driver heap. |
| `spark.driver.cores` | `1` | Driver cores (cluster mode). |
| `spark.driver.maxResultSize` | `1g` | Cap on serialized results returned to driver (`0` = unlimited). |
| `spark.driver.memoryOverhead` / `.memoryOverheadFactor` | factor `0.1` | Off-heap overhead for the driver. |
| `spark.executor.memory` | `1g` | Per-executor heap. |
| `spark.executor.cores` | manager-specific | Cores per executor. |
| `spark.executor.instances` | `2` (YARN/K8s) | Static executor count. |
| `spark.executor.memoryOverhead` / `.memoryOverheadFactor` | factor `0.1` | Per-executor off-heap overhead. |
| `spark.executor.pyspark.memory` | unset | Memory cap for PySpark workers. |
| `spark.executor.heartbeatInterval` | `10s` | Executor → driver heartbeats. |
| `spark.executor.maxNumFailures` | manager-specific | Failures before the app aborts. |
| `spark.default.parallelism` | cluster-dependent | Default partition count for RDD ops. |
| `spark.cores.max` | unset | Max total cores (Standalone/Mesos). |

### 6.2 Memory & storage

| Property | Default | Purpose |
|---|---|---|
| `spark.memory.fraction` | `0.6` | Unified (execution+storage) region. |
| `spark.memory.storageFraction` | `0.5` | Eviction-immune storage sub-region. |
| `spark.memory.offHeap.enabled` | `false` | Use off-heap memory. |
| `spark.memory.offHeap.size` | `0` | Off-heap size (required if enabled). |

### 6.3 Shuffle & serialization

| Property | Default | Purpose |
|---|---|---|
| `spark.serializer` | Java | Set to `org.apache.spark.serializer.KryoSerializer`. |
| `spark.kryo.registrationRequired` | `false` | Force explicit Kryo registration. |
| `spark.kryoserializer.buffer` / `.buffer.max` | `64k` / `64m` | Kryo buffers. |
| `spark.shuffle.compress` | `true` | Compress shuffle output. |
| `spark.shuffle.file.buffer` | `32k` | Shuffle write buffer. |
| `spark.shuffle.checksum.enabled` | `true` | Detect shuffle-block corruption. |
| `spark.shuffle.push.enabled` | `false` | Push-based shuffle merging. |

### 6.4 Scheduling

| Property | Default | Purpose |
|---|---|---|
| `spark.scheduler.mode` | `FIFO` | `FIFO` or `FAIR` (pools via `fairscheduler.xml`). |
| `spark.scheduler.allocation.file` | — | Fair-scheduler pool definitions. |
| `spark.speculation` | `false` | Re-launch slow ("straggler") tasks. |
| `spark.speculation.multiplier` / `.quantile` | `1.5` / `0.75` | Speculation thresholds. |
| `spark.task.maxFailures` | `4` | Task retries before stage failure. |
| `spark.locality.wait` | `3s` | Wait for data-local scheduling. |

For **FAIR** scheduling, copy `conf/fairscheduler.xml.template` to `fairscheduler.xml` and define pools (weights, minShare, scheduling mode). The scheduler implementation is `Pool`/`Schedulable`/`SchedulableBuilder` in `core/.../scheduler`.

### 6.5 Event log & History Server

| Property | Default | Purpose |
|---|---|---|
| `spark.eventLog.enabled` | `false` | Persist application events. |
| `spark.eventLog.dir` | `/tmp/spark-events` | Event-log destination (HDFS/S3 in production). |
| `spark.eventLog.compress` | `false` | Compress event logs. |
| `spark.eventLog.rolling.enabled` | `false` | Roll long-running app logs (`.maxFileSize`). |
| `spark.history.fs.logDirectory` | matches eventLog dir | Where the History Server reads logs. |
| `spark.history.fs.cleaner.enabled` | `false` | Auto-purge old logs (`.maxAge`, `.interval`, `.maxNum`). |
| `spark.history.custom.executor.log.url` | — | Template for executor log links. |

### 6.6 Networking & RPC

| Property | Default | Purpose |
|---|---|---|
| `spark.driver.host` / `.bindAddress` | auto | Advertised / bind address of the driver. |
| `spark.driver.port` | random | Driver RPC port (pin behind firewalls). |
| `spark.blockManager.port` | random | Block transfer port. |
| `spark.network.timeout` | `120s` | Default network timeout. |
| `spark.rpc.message.maxSize` | `128` (MB) | Max RPC message size. |
| `spark.rpc.askTimeout` | `spark.network.timeout` | RPC ask timeout. |
| `spark.network.maxRemoteBlockSizeFetchToMem` | — | Threshold to fetch blocks to disk vs memory. |

### 6.7 SQL & Adaptive Query Execution

| Property | Default | Purpose |
|---|---|---|
| `spark.sql.shuffle.partitions` | `200` | Post-shuffle partitions. |
| `spark.sql.adaptive.enabled` | `true` | Runtime re-optimization. |
| `spark.sql.adaptive.coalescePartitions.enabled` | `true` | Merge small shuffle partitions. |
| `spark.sql.adaptive.skewJoin.enabled` | `true` | Split skewed join partitions. |
| `spark.sql.adaptive.advisoryPartitionSizeInBytes` | `64m` | Target post-coalesce partition size. |
| `spark.sql.autoBroadcastJoinThreshold` | `10m` | Max table size for broadcast joins (`-1` disables). |
| `spark.sql.cbo.enabled` | `false` | Cost-based optimization (needs table stats). |
| `spark.sql.ansi.enabled` | varies | ANSI SQL dialect (stricter, standards-compliant). |
| `spark.sql.sources.default` | `parquet` | Default data source. |
| `spark.sql.warehouse.dir` | `spark-warehouse` | Managed-table location. |
| `spark.sql.catalogImplementation` | `in-memory` | `in-memory` or `hive`. |
| `spark.sql.session.timeZone` | JVM TZ | Session time zone. |
| `spark.sql.execution.arrow.pyspark.enabled` | `false` | Arrow for PySpark↔pandas. |

> [!TIP]
> For most SQL workloads, leave AQE on and tune only `spark.sql.shuffle.partitions` (start near total cores × 2–3 for large jobs). AQE's coalescing then right-sizes partitions automatically using runtime statistics.

---

## 7. Environment Variables (`conf/spark-env.sh`)

Copy `conf/spark-env.sh.template` to `conf/spark-env.sh`. This file is sourced by all Spark programs and daemons.

**Read when launching locally / by drivers & executors:**

| Variable | Purpose |
|---|---|
| `HADOOP_CONF_DIR` | Point Spark at Hadoop config (HDFS). |
| `YARN_CONF_DIR` | Point Spark at YARN config. |
| `SPARK_LOCAL_IP` | IP this node binds to. |
| `SPARK_PUBLIC_DNS` | Public DNS of the driver. |
| `SPARK_LOCAL_DIRS` | Scratch dirs for shuffle and RDD spill. |
| `SPARK_CONF_DIR` | Alternate conf dir (default `$SPARK_HOME/conf`). |
| `SPARK_EXECUTOR_CORES` | Cores per executor (default `1`). |
| `SPARK_EXECUTOR_MEMORY` | Memory per executor (default `1G`). |
| `SPARK_DRIVER_MEMORY` | Driver memory (default `1G`). |

**Standalone daemon options:**

| Variable | Purpose |
|---|---|
| `SPARK_MASTER_HOST` / `SPARK_MASTER_PORT` / `SPARK_MASTER_WEBUI_PORT` | Master binding/ports. |
| `SPARK_MASTER_OPTS` | Master-only `-Dspark.x=y`. |
| `SPARK_WORKER_CORES` / `SPARK_WORKER_MEMORY` | Resources this worker offers. |
| `SPARK_WORKER_PORT` / `SPARK_WORKER_WEBUI_PORT` / `SPARK_WORKER_DIR` | Worker binding/ports/scratch. |
| `SPARK_WORKER_OPTS` | Worker-only `-Dspark.x=y`. |
| `SPARK_DAEMON_MEMORY` | Memory for master/worker/history daemons (default `1g`). |
| `SPARK_HISTORY_OPTS` | History-server-only properties. |
| `SPARK_SHUFFLE_OPTS` | External shuffle service properties. |
| `SPARK_DAEMON_JAVA_OPTS` / `SPARK_DAEMON_CLASSPATH` | All-daemon JVM options/classpath. |

**Generic daemon options:**

| Variable | Default | Purpose |
|---|---|---|
| `SPARK_LOG_DIR` | `$SPARK_HOME/logs` | Daemon log directory. |
| `SPARK_LOG_MAX_FILES` | `5` | Log rotation count. |
| `SPARK_PID_DIR` | `/tmp` | PID files. |
| `SPARK_IDENT_STRING` | `$USER` | Instance identifier. |
| `SPARK_NICENESS` | `0` | Daemon scheduling priority. |
| `SPARK_NO_DAEMONIZE` | — | Run in foreground (no PID file) — useful under systemd/containers. |

**Other:** `SPARK_LAUNCHER_OPTS` (launcher JVM options); native BLAS tuning `MKL_NUM_THREADS=1` / `OPENBLAS_NUM_THREADS=1` (recommended with MLlib + native BLAS, per SPARK-21305); beeline: `SPARK_BEELINE_OPTS`, `SPARK_BEELINE_MEMORY` (default `1G`).

---

## 8. Security

Spark security is **off by default** and must be explicitly enabled. Harden in this order:

### 8.1 Authentication & encryption

| Property | Purpose |
|---|---|
| `spark.authenticate` | Enable shared-secret RPC authentication between components. |
| `spark.authenticate.secret` | The shared secret (prefer secret files / manager-provided secrets over inline). |
| `spark.authenticate.secret.file` (+ `.driver.file` / `.executor.file`) | Read the secret from a file. |
| `spark.network.crypto.enabled` | AES-based encryption for all RPC/shuffle network traffic. |
| `spark.network.crypto.saslFallback` | Fall back to SASL for older peers. |
| `spark.io.encryption.enabled` | Encrypt spilled/local I/O (shuffle, cache on disk). |
| `spark.io.encryption.keySizeBits` / `.keygen.algorithm` | Local I/O key parameters. |

> [!CAUTION]
> Enabling `spark.authenticate` and `spark.network.crypto.enabled` is the baseline for any shared or multi-tenant cluster. Without them, RPC traffic is unauthenticated and unencrypted, and the application Web UI / submission endpoints can be abused. On Standalone/Kubernetes, generate the auth secret per-application rather than hardcoding it.

### 8.2 SSL / TLS for web UIs

Configure the `spark.ssl.*` family (keystore/truststore paths and passwords, enabled protocols and cipher suites) to serve the Web UI and History Server over HTTPS. Use a dedicated `spark.ssl.ui.*` namespace where supported.

### 8.3 Kerberos (Hadoop/YARN)

Submit with `--principal <kdc-principal> --keytab <path>`; the application (and YARN AM) obtains and renews delegation tokens for HDFS, Hive, HBase, and Kafka. `--proxy-user` enables impersonation but is **mutually exclusive** with `--principal`/`--keytab`.

### 8.4 Web UI access control & redaction

- ACLs: `spark.acls.enable`, `spark.ui.view.acls`, `spark.modify.acls`, plus `*.groups` variants and `spark.admin.acls`.
- `spark.ui.filters` — install servlet filters (e.g. authentication proxies) in front of the UI.
- **Secret redaction**: `spark.redaction.regex` scrubs matching config values from the UI, logs, and event logs. Confirm it covers your credential key patterns.

> [!WARNING]
> The live application Web UI (port 4040) and History Server expose configuration, environment, and SQL plans. In production, place them behind authentication (`spark.ui.filters` / a reverse proxy) and enable redaction. Do not expose these ports to untrusted networks.

---

## 9. Monitoring & Operations

### 9.1 Web UI & History Server

The live UI (driver, port 4040) shows jobs, stages, tasks, the DAG, the SQL/AQE plan, storage, executors, and environment — but disappears when the application ends. For post-hoc analysis, enable event logging and run the History Server:

```bash
./sbin/start-history-server.sh     # serves http://<host>:18080
./sbin/stop-history-server.sh
```

### 9.2 Metrics system

Spark's metrics system (configured via `conf/metrics.properties`, from the `.template`) exposes driver, executor, master, worker, and application metrics through pluggable **sinks**:

| Sink | Use |
|---|---|
| `PrometheusServlet` | Scrape metrics in Prometheus format (modern default for k8s/observability stacks). |
| `JmxSink` | Expose via JMX. |
| `GraphiteSink` | Push to Graphite. |
| `StatsdSink` | Push to StatsD. |
| `CsvSink` | Periodic CSV files. |
| `Slf4jSink` | Log metrics via SLF4J. |
| `ConsoleSink` | Print to console (debugging). |
| `MetricsServlet` | JSON over the Web UI. |

### 9.3 Logging

Logging uses **Log4j2**. Copy `conf/log4j2.properties.template` to `log4j2.properties` (or use `log4j2-json-layout.properties.template` for structured JSON logs — ideal for log aggregation). Daemon logs land in `SPARK_LOG_DIR`.

### 9.4 The Thrift JDBC/ODBC server

Expose Spark SQL to BI tools and JDBC clients:

```bash
./sbin/start-thriftserver.sh      # HiveServer2-compatible, port 10000
./bin/beeline -u jdbc:hive2://localhost:10000
./sbin/stop-thriftserver.sh
```

### 9.5 The Spark Connect server

```bash
./sbin/start-connect-server.sh    # gRPC, port 15002
./sbin/stop-connect-server.sh
```

Connect-server configuration lives under `spark.connect.*` (e.g. session/store and grpc settings in `sql/connect/server`).

### 9.6 Operational runbook

| Task | Command / action |
|---|---|
| Start Standalone cluster | `./sbin/start-all.sh` |
| Stop Standalone cluster | `./sbin/stop-all.sh` |
| Add a worker | `./sbin/start-worker.sh spark://master:7077` |
| Gracefully drain a worker | `./sbin/decommission-worker.sh` |
| Run a command on all workers | `./sbin/workers.sh <command>` |
| Kill a cluster-mode driver | `spark-submit --kill <id> --master spark://...` |
| Check a driver's status | `spark-submit --status <id> --master spark://...` |
| Inspect a finished app | History Server at `:18080` |

---

## 10. Performance Tuning Checklist

- [ ] Use **Kryo** serialization (`spark.serializer=org.apache.spark.serializer.KryoSerializer`).
- [ ] Keep **AQE on**; tune `spark.sql.shuffle.partitions` for very large jobs.
- [ ] Right-size **executor geometry** — 3–5 cores/executor is a common sweet spot; avoid one fat executor per node (GC pressure) and avoid many tiny executors (overhead).
- [ ] Set **memory overhead** generously for PySpark / off-heap workloads before raising heap.
- [ ] Enable **dynamic allocation** for bursty/multi-tenant clusters (with shuffle tracking or the external shuffle service).
- [ ] Cache (`persist`) only datasets reused across actions; watch the Storage tab for eviction.
- [ ] Prefer **broadcast joins** for small dimension tables (`spark.sql.autoBroadcastJoinThreshold`).
- [ ] Use **push-based shuffle** (`spark.shuffle.push.enabled`) on very large shuffles.
- [ ] Place `spark.local.dir`/`SPARK_LOCAL_DIRS` on fast local disks (NVMe), not the OS root volume.
- [ ] Pre-stage Spark jars on YARN (`spark.yarn.archive`) to cut submission latency.

---

## 11. Troubleshooting (Operational)

### Issue: Executors killed by YARN/K8s for exceeding memory

**Cause:** Container memory = heap + overhead exceeded the manager's limit (often overhead, not heap).
**Resolution:** Raise `spark.executor.memoryOverhead` (or the factor). For PySpark, account for `spark.executor.pyspark.memory`.

### Issue: Job stuck / very slow on a few tasks (skew)

**Cause:** Data skew concentrating work on a few partitions.
**Resolution:** Keep `spark.sql.adaptive.skewJoin.enabled=true`; consider salting keys or repartitioning. Enable speculation (`spark.speculation`) for straggler nodes.

### Issue: `Total size of serialized results … is bigger than spark.driver.maxResultSize`

**Cause:** An action returned too much data to the driver.
**Resolution:** Avoid `collect()` on large data; write to storage. Raise `spark.driver.maxResultSize` only if truly needed (`0` = unlimited, risky).

### Issue: Standalone master failover loses applications

**Cause:** `spark.deploy.recoveryMode=NONE`.
**Resolution:** Use `ZOOKEEPER` (multi-master HA) or `FILESYSTEM`/`ROCKSDB` recovery with a durable `spark.deploy.recoveryDirectory`.

### Issue: YARN submission rejected — no Hadoop config

**Cause:** `HADOOP_CONF_DIR`/`YARN_CONF_DIR` unset.
**Resolution:** Export it (set in `conf/spark-env.sh`).

### Issue: Shuffle fetch failures after executors are removed

**Cause:** Dynamic allocation/decommission removed executors holding shuffle files with no service to serve them.
**Resolution:** Enable the **external shuffle service** or `spark.dynamicAllocation.shuffleTracking.enabled=true`.

### Issue: Web UI / logs leak secrets

**Cause:** Credentials in config values, no redaction.
**Resolution:** Set `spark.redaction.regex` to match your secret keys; restrict UI access with `spark.ui.filters`/ACLs.

---

## 12. Internals (Developer Deep-Dive)

For contributors and platform engineers extending Spark:

- **Configuration system.** Every property is a typed `ConfigEntry` built by `ConfigBuilder("spark.x.y").doc(...).version(...).<type>Conf.createWithDefault(...)` in `core/.../internal/config/` and per-module config packages. The `.doc()` string is the canonical description and the `.version(...)` records when the key was introduced — the authoritative, version-stamped config reference. Negative/removed knowledge and deprecations are tracked in the same definitions.
- **Scheduler.** `DAGScheduler` (stage/DAG construction, shuffle-boundary splitting, retry of failed stages) → `TaskSchedulerImpl` (offer-based task assignment, locality, excludes) → `SchedulerBackend` (manager-specific executor management) → `TaskSetManager` (per-stage task lifecycle, speculation). Fair scheduling via `Pool`/`Schedulable`/`SchedulableBuilder`.
- **Block management.** `BlockManager` on driver and each executor handles cached blocks, broadcast variables, and shuffle data, backed by the `network-common`/`network-shuffle` Netty transport.
- **Catalyst & Tungsten** (`sql/catalyst`, `sql/core`). Parser → analyzer (catalog resolution) → optimizer (rule-based + cost-based) → physical planning → whole-stage codegen. AQE re-plans between materialized stages.
- **Plugin & extension points.** `SparkPlugin`/`DriverPlugin`/`ExecutorPlugin` (`core/.../api/plugin`), custom shuffle (`shuffle/api`), data source v2, custom catalogs (`spark.sql.catalog.*`), custom metrics sinks, and standalone recovery factories.
- **Spark Connect** (`sql/connect`). A gRPC protocol decoupling the client from the driver; the server translates protobuf-encoded relations into Spark plans. Clients exist for Python, Scala/JVM, and JDBC.
- **Language bridges.** PySpark (`python/pyspark`) and SparkR communicate with the JVM through a gateway; PySpark workers are forked per executor and tuned via `PYSPARK_*` / `spark.executor.pyspark.memory`. The pandas API on Spark lives in `python/pyspark/pandas`.

---

## 13. Repository Structure

```
spark/
├── core/             Engine: scheduler, BlockManager, RDD, Standalone deploy, shuffle
├── common/           network-common, network-shuffle, kvstore, unsafe, sketch, variant, utils, tags
├── launcher/         SparkLauncher + SparkSubmitOptionParser
├── repl/             Scala interactive shell
├── sql/
│   ├── api/          Stable public SQL API
│   ├── catalyst/     Logical plans, analyzer, optimizer, expressions, parser
│   ├── core/         DataFrame/Dataset, physical execution, data sources, AQE
│   ├── hive/         Hive catalog integration
│   ├── hive-thriftserver/  JDBC/ODBC (HiveServer2) server
│   ├── connect/      Spark Connect (gRPC server + clients)
│   └── pipelines/    Declarative Pipelines engine
├── streaming/        Structured Streaming + legacy DStreams
├── mllib/ mllib-local/  Machine learning
├── graphx/           Graph processing + algorithms
├── connector/        kafka-0-10, kinesis-asl, avro, protobuf, profiler, ganglia
├── resource-managers/
│   ├── yarn/         YARN backend
│   └── kubernetes/   Kubernetes backend + integration tests + docker
├── hadoop-cloud/     S3A/GCS/ABFS cloud object-store integration
├── python/           PySpark (incl. pandas API, pipelines, errors)
├── R/                SparkR (deprecated)
├── bin/  sbin/       User launchers / cluster daemon scripts
├── conf/             Config templates
├── examples/ data/   Sample programs and datasets
├── dev/ project/     Build, release, and dev tooling
└── docs/             Documentation site sources
```

---

## 14. Potential Improvements & Operational Notes

These are operator-facing observations, not defects:

- **Pin firewall-sensitive ports.** Driver (`spark.driver.port`), block manager (`spark.blockManager.port`), and the external shuffle service (`7337`) default to random/fixed ports; pin them explicitly when running behind firewalls or in locked-down networks.
- **Standalone HA requires deliberate setup.** The default `recoveryMode=NONE` means a master restart drops running applications. Choose `ZOOKEEPER` for true HA in any production Standalone deployment.
- **SparkR is deprecated.** Plan migrations off R toward PySpark/Scala/SQL; avoid building new pipelines on SparkR.
- **Declarative Pipelines is new on the master branch.** Treat the `spark-pipelines` CLI and spec format as evolving; validate against release notes before standardizing on it.
- **`spark.sql.ansi.enabled` semantics differ by line.** ANSI mode changes overflow/cast/null behavior; test workloads before flipping it cluster-wide, as it can change query results and error behavior.
- **Event-log growth.** Long-running apps and busy clusters accumulate event logs quickly; enable `spark.eventLog.rolling.enabled` and the History Server cleaner (`spark.history.fs.cleaner.enabled`) to bound storage.

---

## 15. Quick Reference

### Cluster daemon scripts (`sbin/`)

| Script | Action |
|---|---|
| `start-all.sh` / `stop-all.sh` | Master + all workers. |
| `start-master.sh` / `stop-master.sh` | Standalone master. |
| `start-worker.sh` / `stop-worker.sh` | A single worker (arg: master URL). |
| `start-workers.sh` / `stop-workers.sh` | All workers in `conf/workers`. |
| `start-history-server.sh` / `stop-history-server.sh` | History Server (`:18080`). |
| `start-thriftserver.sh` / `stop-thriftserver.sh` | JDBC/ODBC server (`:10000`). |
| `start-connect-server.sh` / `stop-connect-server.sh` | Spark Connect (`:15002`). |
| `decommission-worker.sh` | Gracefully drain a worker. |
| `workers.sh <cmd>` | Run a command on every worker. |
| `spark-daemon.sh` / `spark-daemons.sh` | Generic daemon control. |

### Essential properties

| Property | Default | Lever |
|---|---|---|
| `spark.executor.memory` | `1g` | Per-executor heap |
| `spark.executor.cores` | manager-specific | Parallelism per executor |
| `spark.executor.instances` | `2` | Static executor count |
| `spark.dynamicAllocation.enabled` | `false` | Autoscaling |
| `spark.sql.shuffle.partitions` | `200` | Shuffle parallelism |
| `spark.sql.adaptive.enabled` | `true` | Runtime optimization |
| `spark.serializer` | Java | Set Kryo |
| `spark.authenticate` | `false` | RPC auth |
| `spark.network.crypto.enabled` | `false` | Wire encryption |
| `spark.eventLog.enabled` | `false` | History Server input |
| `spark.deploy.recoveryMode` | `NONE` | Standalone HA |

---

_Generated June 2026 from apache/spark at a32cda3. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._