exo Admin & Architecture Guide

This guide is the deep reference for operators, architects, and developers running or extending exo — a distributed AI inference engine that turns a set of devices into a single cluster capable of serving language and image models too large for any one machine. It covers the system architecture, every component, configuration, deployment, security posture, operational procedures, and the internals needed to reason about (or replicate) the system.

For installation and day-to-day usage, see the companion exo User Guide.

Note

exo is written in Python 3.13 (orchestration, API, inference glue), Rust (networking and native bindings via PyO3), Swift (the macOS app), and TypeScript/Svelte (the dashboard). It is Apache-2.0 licensed and maintained by exo labs.

Table of Contents

1. Executive Summary
2. System Architecture
3. Component Inventory
4. The Distributed Model: Roles, Election & Event Sourcing
5. Placement & Sharding Engine
6. The Inference Pipeline (Worker → Runner)
7. Networking Layer
8. The API Server
9. Model Cards
10. Configuration Reference
11. Filesystem Layout
12. Requirements & Capacity Planning
13. Deployment
14. Operations
15. Security Analysis
16. Observability & Tracing
17. Performance Tuning
18. Code Quality & Conventions
19. Testing Strategy
20. Known Issues & Improvement Areas
21. Building from Source & Packaging
22. Quick Reference

1. Executive Summary

exo is a peer-to-peer, masterless-by-default cluster in which every node runs the same binary and can take on any role. At startup, nodes discover each other automatically over the local network, elect a coordinator ("master"), and continuously share a view of the cluster's topology, memory, and download state. When a user requests a model, the master computes a placement — how to shard the model across a connected ring of devices given their memory and link speeds — and the relevant workers spin up inference runner subprocesses that execute the model using Apple's MLX framework.

Key architectural properties:

• No central server, no external dependencies. No database, no message broker, no cloud. State is replicated via an append-only event log that flows over a zenoh pub/sub mesh.
• Self-organizing. Discovery, election, topology mapping, and re-placement on failure are automatic.
• Topology-aware. Placement considers per-link types (Thunderbolt/RDMA, Ethernet, Wi-Fi) and per-device memory.
• Multi-protocol API. A single FastAPI server exposes OpenAI Chat Completions, OpenAI Responses, Anthropic Claude Messages, and Ollama dialects, plus image generation/editing and cluster-management endpoints.
• Heterogeneous performance backends. A socket "ring" backend works anywhere; an RDMA-over-Thunderbolt "JACCL" backend delivers low-latency multi-device inference on Apple Silicon.

Important

exo is under heavy active development. The macOS (Apple Silicon, GPU) path is Tier 1 and the most mature. Linux runs on CPU. Several features (prefill/decode disaggregation, image models) are gated behind feature flags and off by default.

2. System Architecture

2.1 Single-node view

Every exo process constructs one Node (src/exo/main.py) that wires together a fixed set of long-running async services under a single anyio task group:

                         ┌──────────────────────────────────────────────┐
                         │                  exo Node                     │
                         │                                               │
   network  ◀───────────┤  Router (zenoh)  ──  EventRouter (event bus)   │
   (other nodes)        │     │                      │                   │
                        │     │     ┌────────────────┼───────────────┐   │
                        │     │     │                │               │   │
                        │  Election │   Master   DownloadCoordinator │   │
                        │           │     │            │             │   │
                        │           │   Worker ── runner subprocs    │   │
                        │           │     │                          │   │
   HTTP clients ◀───────┤          API (FastAPI + dashboard) ────────┘   │
   :52415               │                                                │
                         └──────────────────────────────────────────────┘

The services created in Node.create():

Each node communicates over a set of named topics registered with the router: GLOBAL_EVENTS, LOCAL_EVENTS, COMMANDS, ELECTION_MESSAGES, CONNECTION_MESSAGES, DOWNLOAD_COMMANDS.

2.2 Cluster view

   ┌─────────┐      zenoh mesh (pub/sub) + UDP discovery       ┌─────────┐
   │ Node A  │◀───────────────────────────────────────────────▶│ Node B  │
   │ (master)│                                                  │(worker) │
   └────┬────┘                                                  └────┬────┘
        │   global event log (append-only, replicated)               │
        │                                                            │
        │            ┌─────────┐                                     │
        └───────────▶│ Node C  │◀────────────────────────────────────┘
                     │(worker) │     RDMA/Thunderbolt or socket links
                     └─────────┘     carry model activations between shards

The master holds the authoritative event log; other nodes replicate it. Inference activations (the tensors passed between model shards) flow directly between worker runners over the chosen backend (ring sockets or RDMA), not through the event bus — the event bus carries control plane data (commands, state, tokens/chunks back to the API), while the data plane (model activations) goes peer-to-peer between runners.

2.3 Request lifecycle

client → API.chat_completions()
       → adapter converts request → TextGenerationTaskParams
       → API._send(TextGeneration command) ──(COMMANDS topic)──▶ Master
       → Master._command_processor matches TextGeneration:
            picks least-busy instance for the model → emits TaskCreated event
       → event replicated to Workers
       → Worker.plan_step → plan() decides the task is runnable
            → Worker._start_runner_task → runner subprocess executes shard
       → runner emits ChunkGenerated events (tokens) back through the bus
       → API._apply_state routes chunks to the request's stream queue
       → API streams SSE tokens to the client
       → on completion, API emits TaskFinished → Master emits TaskDeleted

3. Component Inventory

3.1 Repository structure

exo/
├── src/exo/                  # Python core
│   ├── main.py               # Entry point: Node, Args (CLI), main()
│   ├── api/                  # FastAPI server + protocol adapters
│   │   ├── main.py           #   class API, all routes
│   │   ├── adapters/         #   chat_completions, claude, ollama, responses
│   │   ├── types/            #   request/response pydantic models
│   │   └── keepalive.py      #   SSE keep-alive wrapper
│   ├── master/               # Cluster orchestration
│   │   ├── main.py           #   class Master, command/event processors
│   │   ├── placement.py      #   place_instance(), instance lifecycle
│   │   ├── placement_utils.py#   sharding, layer allocation, IP prioritization
│   │   └── image_store.py    #   ephemeral generated-image storage
│   ├── worker/               # Inference execution
│   │   ├── main.py           #   class Worker, plan_step
│   │   ├── plan.py           #   pure planning state machine
│   │   ├── runner/           #   runner supervisor + inference processes
│   │   ├── engines/          #   mlx (LLM) + image (FLUX/Qwen) engines
│   │   └── disaggregated/    #   prefill/decode disaggregation server
│   ├── routing/              # EventRouter, Router, topics
│   ├── download/             # DownloadCoordinator, shard downloader
│   ├── shared/               # Cross-cutting: constants, models, types, election, topology, apply
│   └── utils/                # channels, async_process, info_gatherer, banner, ports, tracing
├── rust/
│   ├── exo_rs/               # PyO3 bindings: Pidfile, networking handle
│   └── networking/           # zenoh-based discovery + swarm
├── app/EXO/                  # Swift macOS menu-bar app
├── dashboard/                # SvelteKit + Tailwind dashboard SPA
├── bench/                    # exo_bench, exo_eval, prefill_decode_bench, eval_tool_calls
├── resources/                # Built-in model cards (TOML)
│   ├── inference_model_cards/#   122 LLM cards
│   └── image_model_cards/    #   18 image-model cards
├── packaging/                # PyInstaller spec + dmg packaging
├── nix/, flake.nix           # Nix build
├── tools/                    # exo-tools helper package
└── docs/                     # api.md, benchmarks, images

3.2 Python module map (HIGH-priority)

3.3 Rust crates

The Rust workspace (Cargo.toml) uses edition 2024, resolver 3, and a patched fork of zenoh (github.com/evanev7/zenoh branch exo) for all zenoh crates. The networking crate is built as a cdylib (for Python import) plus rlib (for stub-gen).

4. The Distributed Model: Roles, Election & Event Sourcing

4.1 Every node runs every role

A node is constructed with a Master, and (unless disabled) a Worker, DownloadCoordinator, and API. Only one node's Master is active at a time; the others sit dormant until election promotes them. This means any node can become the coordinator if the current one fails — there is no special "server."

4.2 Election

The Election service (src/exo/shared/election.py) runs on every node and exchanges ELECTION_MESSAGES. A node started with --force-master is given a huge seniority (1,000,000) so it wins elections; ordinary nodes have seniority 0. Election results stream into Node._elect_loop(), which reacts:

• Becomes master: if not already master, construct and start a Master.
• Loses master: shut down the local Master.
• New master elected (any): rebuild the EventRouter for the new session, recreate the DownloadCoordinator and Worker, and reset() or unpause() the API.

Each election produces a SessionId (master node id + election clock). Events are tagged with the session; the master discards events not from its own session, which prevents stale messages from a prior master from corrupting state.

Note

The election loop is explicitly flagged in-code as entangled logic slated for refactoring. Operators should be aware that rapid master churn (e.g. a flapping network link) triggers repeated worker/API resets and is worth avoiding via stable connectivity.

4.3 Event sourcing

State is derived by folding an append-only event log through a pure apply(state, event) reducer (src/exo/shared/apply.py). The Master is the source of truth: it ingests local events, assigns them monotonic indices, persists them to a DiskEventLog, and re-broadcasts them as GlobalForwarderEvents on the GLOBAL_EVENTS topic. Each node (API, Worker, Master) maintains its own State by applying the global event stream.

State includes: topology, node_memory, node_network, node_backends, instances, tasks, downloads, instance_links, last_seen, and more. Commands (intent) flow on the COMMANDS topic; events (facts) flow on the event topics. The MultiSourceBuffer and OrderedBuffer utilities reorder events from multiple origins by index before applying.

The Master._plan() loop runs every 10 seconds to reconcile reality against state: it deletes instances whose nodes have disconnected and emits NodeTimedOut for nodes not seen in 30 seconds.

Tip

Because the filesystem-backed event log is the persistent memory, an operator debugging cluster behavior can inspect /state and /events on the API to see exactly what the cluster believes is true.

5. Placement & Sharding Engine

Placement (src/exo/master/placement.py, place_instance()) is the heart of exo's "automatic" behavior. Given a PlaceInstance command (model card + requested sharding + backend + min nodes), it selects which devices run the model and how.

5.1 Algorithm

1. Find cycles. Ask the topology for device cycles (rings) with at least min_nodes members. If required_nodes is specified, keep only cycles containing all of them.
2. Memory filter. Keep only cycles whose combined available RAM ≥ the model's storage_size. If none, raise No cycles found with sufficient memory.
3. Sharding constraints:
   • Tensor sharding requires model_card.supports_tensor; cycles must divide hidden_size evenly (and KV-head divisibility unless the model head-parallelises, e.g. DeepSeek V4). Otherwise raise a descriptive ValueError.
   • Specific guards exist: DeepSeek V3.1 (8-bit) can't use pipeline parallelism; Gemma 4 can't use pipeline parallelism (forced to single-node / tensor).
4. Smallest cycles. Prefer the cycles with the fewest nodes (less communication overhead).
5. Backend feasibility. Intersect the model's backends with the requested engine's required backends (MlxRing → Metal/CUDA/CPU; MlxJaccl → Metal only). Every node in a cycle must support a compatible backend, or raise.
6. RDMA filter. For MlxJaccl, keep only cycles that are RDMA-connected and have rdma_ctl enabled on every node; otherwise raise Requested RDMA (MlxJaccl) but no RDMA-connected cycles available.
7. Leaf preference + selection. Prefer cycles containing leaf nodes, then pick the cycle maximizing (download progress for the model, then total available memory) — i.e. favor devices that already have the model and the most free RAM.
8. Single-node fallback. A 1-node cycle is forced to MlxRing + Pipeline.
9. Shard assignment + instance construction (below).

5.2 Layer allocation

allocate_layers_proportionally() distributes the model's layers across nodes proportional to each node's available RAM, using the largest-remainder method, guaranteeing at least one layer per node. Each node's required memory for its layers is validated against availability before the instance is built (Node N has insufficient memory…).

5.3 Sharding strategies

Each assignment maps node_to_runner and runner_to_shard (with ShardMetadata carrying device_rank, world_size, start_layer, end_layer).

5.4 Instance types & connection wiring

• MlxRingInstance — built with hosts_by_node: for each node, a host list where self is 0.0.0.0:<ephemeral_port>, left/right neighbors carry real IPs, and non-neighbors get an RFC-5737 placeholder (198.51.100.1:0). Neighbor IPs are chosen by find_ip_prioritised(..., ring=True), which prefers thunderbolt > ethernet > wifi.
• MlxJacclInstance — built with a jaccl_devices all-to-all RDMA interface matrix and jaccl_coordinators (rank-0 coordinator address per node). Coordinator IP selection prefers ethernet > wifi (ring=False).

Important

The ring backend prioritizes Thunderbolt for the data path (lowest latency for neighbor exchange), while JACCL prioritizes Ethernet for the coordinator control channel. This split is intentional and reflects how each backend uses its links.

6. The Inference Pipeline (Worker → Runner)

6.1 The plan state machine

Worker.plan_step() calls the pure function plan() (src/exo/worker/plan.py) every 100 ms. plan() short-circuits through an ordered set of checks and returns the single next Task to perform (or None):

_cancel_tasks          → cancel any task marked Cancelled
_kill_runner           → Shutdown runners for deleted/failed instances
_create_runner         → CreateRunner for instances assigned to this node
_model_needs_download  → DownloadModel when an idle runner lacks its shard
_init_distributed_backend → ConnectToGroup to form the ring/RDMA group
_load_model            → LoadModel once all shards are downloaded & connected
_ready_to_warmup       → StartWarmup once all ranks are loaded
_pending_tasks         → forward TextGeneration / ImageGeneration / ImageEdits

This produces a runner lifecycle: idle → (download) → connecting → connected → loading → loaded → warming up → ready → running. The handshake between ranks is carefully ordered (rank n-1 initiates connection; rank 0 coordinates warmup) so all devices in an instance come up together.

6.2 Runner subprocesses

Workers don't run models in-process. Worker._create_supervisor() creates a RunnerSupervisor (src/exo/worker/runner/supervisor.py) which manages a separate runner process (started with multiprocessing spawn). This isolates the heavy MLX/Metal execution from the async control plane, so a crashed or hung runner doesn't take down the node. Runners are bounded by EXO_MAX_INSTANCE_RETRIES (5); an instance that exceeds it is requested for deletion.

6.3 Engines

• MLX LLM engine (src/exo/worker/engines/mlx/) — text generation on Apple Silicon via MLX and a forked mlx-lm. Supports the ring and JACCL distributed backends, continuous batching (unless --no-batch), and an optional KV prefix cache.
• Image engines (src/exo/worker/engines/image/) — FLUX and Qwen image models via a forked mflux, with model adapters (ModelAdapter, PromptData) and a generation pipeline. Gated behind EXO_ENABLE_IMAGE_MODELS.

6.4 Multimodal & chunked transfer

Images sent to the API (vision prompts or img2img) are SHA-256 hashed, deduplicated, and split into ≤512 KB chunks (EXO_MAX_CHUNK_SIZE) sent as InputImageChunks; the worker reassembles them by hash into an image cache before running the task. Generated images stream back as ImageChunks and are reassembled by the API.

6.5 Prefill/Decode disaggregation (experimental)

Behind ENABLE_DISAGGREGATION, exo can split inference into separate prefill and decode instances linked via InstanceLinks, with a dedicated disaggregation server (src/exo/worker/disaggregated/server.py). The master routes a decode task to a prefill_endpoint chosen from linked prefill instances by least in-flight load. This is off by default and the placement preview path for it is still TODO.

7. Networking Layer

7.1 Transport: zenoh

exo's Router (src/exo/routing/router.py) wraps zenoh (a patched fork) for pub/sub messaging between nodes. Topics are registered per concern (events, commands, election, downloads). The default zenoh TCP listen port is 52414 (--zenoh-port).

7.2 Discovery

A custom UDP discovery service (Rust, rust/networking/src/discovery.rs) runs on 52413 (--discovery-port). Nodes broadcast Hello/WhatsUp messages within a namespace (default = exo version, override with --namespace / EXO_ZENOH_NAMESPACE); only same-namespace nodes connect. This is how clusters auto-form and how clusters on a shared network stay isolated.

7.3 Topology mapping

Each Worker._poll_connection_updates() actively probes reachability (check_reachable) to other nodes every 10 s, emitting TopologyEdgeCreated/TopologyEdgeDeleted events as links appear and disappear. The info_gatherer collects each node's memory, network interfaces (with interface_type: thunderbolt/ethernet/wifi/unknown), backends, and RDMA control status, broadcast as NodeGatheredInfo. Together these build the Topology graph that placement consumes.

Note

Connection types matter for placement. interface_type values drive the priority ordering in find_ip_prioritised. Mislabeled or undiscovered interfaces degrade placement quality, so accurate interface detection (and physical cabling) directly affects performance.

8. The API Server

The API class (src/exo/api/main.py) is a FastAPI app served by hypercorn bound to 0.0.0.0:<api-port> (default 52415). The SvelteKit dashboard is mounted as static files at /.

8.1 Endpoint catalog

Meta / state

Instance management

Instance links (disaggregation, feature-gated)

Models

Inference

Images (feature-gated by EXO_ENABLE_IMAGE_MODELS)

Downloads & traces

8.2 Request flow internals

Each inference adapter converts the dialect-specific request into a common TextGenerationTaskParams (or image task params), validates that an instance exists for the model (_validate_model_has_instance, 404 otherwise), sends a command to the master, and returns a StreamingResponse. Streaming text uses SSE (text/event-stream) wrapped in with_sse_keepalive (a 10 s keep-alive comment prevents idle disconnects); Ollama streaming uses NDJSON. Tokens arrive as ChunkGenerated events routed by command_id into per-request channels in _apply_state.

8.3 Concurrency & lifecycle

EXO_MAX_CONCURRENT_REQUESTS (default 8) bounds in-flight work. During an election, the API pauses (_pause_on_new_election): _send blocks until unpaused, so requests aren't lost mid-transition. On master change the API is reset() (fresh state + event log) or unpause()d. The image store is cleaned of expired entries every 5 minutes; generated images expire after ~1 hour.

Caution

CORS is configured wide open: allow_origins=["*"], allow_credentials=True, all methods and headers. Combined with the bind to 0.0.0.0 and no authentication, this means anyone who can reach the API port can use (and manage) the cluster. See Security.

9. Model Cards

Models are described by TOML model cards that define metadata and capabilities. exo loads cards from:

• resources/inference_model_cards/ — built-in LLM cards (122 shipped)
• resources/image_model_cards/ — built-in image-model cards (18 shipped)
• ~/.local/share/exo/custom_model_cards/ (EXO_CUSTOM_MODEL_CARDS_DIR) — user-added cards, synced across the cluster

9.1 Card schema (example)

model_id = "mlx-community/Llama-3.2-1B-Instruct-4bit"
n_layers = 16
hidden_size = 2048
num_key_value_heads = 8
supports_tensor = true
tasks = ["TextGeneration"]
family = "llama"
quantization = "4bit"
base_model = "Llama 3.2 1B"
capabilities = ["text"]
context_length = 131072
backends = ["MlxMetal", "MlxCuda", "MlxCpu"]

[storage_size]
in_bytes = 729808896

[sampling_defaults]
temperature = 0.6
top_p = 0.9

9.2 Adding cards

To add a model, drop a TOML card in the appropriate directory, or use POST /models/add with a HuggingFace id — exo fetches the config via ModelCard.fetch_from_hf, writes a custom card, and syncs it cluster-wide (the worker's _reconcile_custom_cards loop keeps the on-disk cache in step with cluster state).

10. Configuration Reference

exo is configured through CLI flags and environment variables — there is no required config file (a config.toml location exists at ~/.config/exo/config.toml but the runtime is driven primarily by flags/env).

10.1 CLI flags

Warning

--bootstrap-peers is disabled — it raises Bootstrap peers has been temporarily removed. Discovery is automatic.

10.2 Environment variables (complete)

Internal constants (not env-tunable): EXO_MAX_CHUNK_SIZE = 512 KB, EXO_MAX_INSTANCE_RETRIES = 5.

Caution

Setting EXO_LIBP2P_NAMESPACE (the old name) causes exo to fail fast at startup with EXO_LIBP2P_NAMESPACE has been removed - use EXO_ZENOH_NAMESPACE instead. Update any automation that still sets it.

11. Filesystem Layout

exo follows the XDG Base Directory spec on Linux; on macOS everything roots at ~/.exo. EXO_HOME overrides the base.

On macOS these live under ~/.exo/ (config/data/cache collapse to one root).

12. Requirements & Capacity Planning

12.1 Platform support tiers

Today: macOS uses the GPU (Metal); Linux runs on CPU.

12.2 Sizing a cluster

The binding constraint is combined available RAM ≥ model storage_size. Plan as follows:

1. Find the target model's storage_size.in_bytes (its card or /models).
2. Ensure the sum of ram_available across the devices you intend to use exceeds it, with headroom for the KV cache and activations.
3. For best throughput, keep the cycle small (exo prefers fewest nodes) and fast-linked (Thunderbolt/RDMA).
4. For tensor parallelism, verify the model supports it and hidden_size divides evenly across the node count.

Tip

The benchmark 2 devices ≈ 1.8× and 4 devices ≈ 3.2× (tensor parallel) is a useful planning heuristic: scaling is sublinear, so add devices for capacity (bigger models) first and speed second.

12.3 macOS memory tuning

For large models that fit in RAM, raising the wired (GPU-accessible) memory limit helps:

sudo sysctl iogpu.wired_limit_mb=N

Choose N larger than the model size in MB but smaller than total RAM.

13. Deployment

exo is not a traditional service you deploy to a server — it's a process you run on each device. Three patterns:

13.1 Foreground (recommended for servers)

Run under a supervisor that keeps it in the foreground:

systemd (Linux)

# /etc/systemd/system/exo.service
[Unit]
Description=exo node
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=exo
WorkingDirectory=/opt/exo
ExecStart=/usr/bin/uv run exo
Restart=on-failure
Environment=EXO_ENABLE_IMAGE_MODELS=false

[Install]
WantedBy=multi-user.target

Important

Do not combine systemd/launchd with --legacy-daemon. The daemon flag double-forks; supervisors expect a foreground process. Use foreground mode and let the supervisor manage lifecycle.

launchd (macOS)

Use a standard launchd plist invoking uv run exo in the foreground, or simply install the macOS app, which manages background execution and the network profile for you.

Legacy init

uv run exo --legacy-daemon

Only for SysV-style init scripts.

13.2 macOS app

brew install --cask exo or the .dmg installs a menu-bar app (macOS 26.2+). It requests permission to install a network LaunchDaemon and configuration profile (for discovery and RDMA). Uninstall cleanly via menu bar → Advanced → Uninstall, or run sudo ./app/EXO/uninstall-exo.sh. (Remember to remove EXO from Login Items manually.)

13.3 Multi-node rollout

1. Deploy exo to each device (same version — namespace defaults to the version string, so mismatched versions won't cluster).
2. Cable devices (Thunderbolt 5 for RDMA) and put them on one network.
3. Start exo on each. They self-discover and elect.
4. Optionally designate coordinator-only nodes with --no-worker.
5. Pin a master with --force-master on a stable, well-connected device if you want deterministic coordination.

Warning

Version skew breaks clustering by default because the discovery namespace defaults to the exo version. Roll out the same version everywhere, or set an explicit shared --namespace.

14. Operations

14.1 Health & inspection

# Cluster's belief about itself
curl http://localhost:52415/state | jq

# Current master
curl http://localhost:52415/node_id

# Running instances (Ollama-style)
curl http://localhost:52415/ollama/api/ps

# Feature flags in effect
curl http://localhost:52415/v1/feature-flags

14.2 Managing model lifecycle

# Start a download to a specific node
curl -X POST http://localhost:52415/download/start -H 'Content-Type: application/json' -d '{...}'

# Cancel / delete a download
curl -X POST http://localhost:52415/download/cancel -H 'Content-Type: application/json' -d '{...}'
curl -X DELETE "http://localhost:52415/download/NODE_ID/mlx-community/My-Model"

# Tear down an instance to reclaim memory
curl -X DELETE http://localhost:52415/instance/INSTANCE_ID

14.3 Logs

Increase verbosity with -v/-vv at launch. hypercorn access logging is disabled; error logging goes to stderr through a custom InterceptLogger.

14.4 Backup & recovery

• Models are large but reproducible (re-downloadable). Back them up only to save bandwidth, or stage them on a shared EXO_MODELS_READ_ONLY_DIRS mount.
• Event logs under event_log/ are operational state; they're rebuilt as the cluster runs and generally don't need backup.
• Custom model cards under custom_model_cards/ are worth backing up if you've curated many.

14.5 Graceful shutdown

SIGINT/SIGTERM trigger Node.shutdown(), which cancels the task group; a second signal forces sys.exit(1). Runners are shut down with their supervisors. The PID file is always released in a finally block.

15. Security Analysis

Caution

exo exposes an unauthenticated, CORS-open control plane on 0.0.0.0. Treat the API port as fully trusted-network-only. Anyone who can reach :52415 can run inference, add/delete models, create/delete instances, and read cluster state. There is no API key, token, or auth middleware.

15.1 Threat model & exposure

Recommendations:

• Bind exo to a trusted network only; never expose :52415 to the public internet. Put it behind a firewall, reverse proxy with auth, or VPN/Tailscale if remote access is needed.
• Use a non-default --namespace to avoid accidental cluster joins and to compartmentalize.
• Run exo as a non-privileged user (the macOS app needs elevated rights only to install its network profile/RDMA config).

15.2 Remote code execution via models

trust_remote_code is disabled by default. Custom models that require it execute repository code on the worker. Only enable it for fully trusted models — this is the single highest-impact RCE vector in normal use.

15.3 Path-traversal hardening

Trace endpoints resolve task ids against EXO_TRACING_CACHE_DIR and reject paths that escape it (Invalid task ID), preventing arbitrary file reads via ../. This is the correct pattern; any new file-serving endpoint should follow it.

15.4 Secrets hygiene

Warning

Issue #2116 ("Hardcoded AWS credentials in Xcode scheme file") was reported and closed. Operators building the macOS app from source should verify no credentials are baked into Xcode scheme files or build artifacts, and scrub any local clones accordingly. Never commit credentials to scheme/plist files.

15.5 Data locality

A genuine security benefit: all inference runs on your own hardware. Prompts and outputs do not leave your network (in --offline mode, not even model downloads). This is a core reason to deploy exo over a hosted API for sensitive workloads.

16. Observability & Tracing

Enable distributed tracing with EXO_TRACING_ENABLED=true. When on, runners emit per-rank TraceEventData; the master collects traces from all expected ranks for a task, merges them (_merge_and_save_traces), and the API persists them to ~/.cache/exo/traces/trace_<task_id>.json.

Trace API:

The /bench/* endpoints additionally return generation stats (prompt/generation tokens-per-second, peak memory) and power usage sampled via PowerSampler, which on Apple Silicon draws on macmon.

Tip

Traces use a Chrome-trace-like schema (name/start/duration/category/rank), so merged traces can be visualized to find per-rank bottlenecks in a distributed run.

17. Performance Tuning

Use bench/exo_bench.py to empirically find the best (sharding, backend, node-count) tuple for a model on your hardware before committing it to production placement.

Note

Known performance pitfalls from the issue tracker: a pipeline-parallel deadlock at 32+ concurrent requests (all_gather blocking), some MacBook Pro pairings running at ~½ expected RDMA speed, and a vision cache key that omits image dimensions. Track these in the upstream issues if they affect you.

18. Code Quality & Conventions

exo enforces unusually strict standards (see Cargo.toml lints, CONTRIBUTING.md, RULES.md):

• Language choice: "prefer Rust unless there's a good reason otherwise." Rust handles networking and performance-critical paths; Python handles orchestration and the API; Swift the app; TypeScript the dashboard.
• Rust lints: Clippy correctness = deny; pedantic, nursery, cargo, and a large restriction set at warn (e.g. unwrap_used, panic, indexing_slicing, arithmetic_side_effects, expect_used, as_conversions). disallowed_script_idents and self_named_module_files are deny.
• Python: full type hints with basedpyright, formatted by ruff, Pydantic models everywhere (FrozenModel base for immutability).
• Architecture style: event sourcing with a pure apply() reducer and pure planning functions (plan()), favoring testable pure functions. The codebase candidly marks "cracks in the event-sourcing architecture" (e.g. the master/node plan loops) for future refactoring.
• Comments explain why, not what.

Common developer commands (justfile):

just fmt           # treefmt / nix fmt
just lint          # ruff check --fix
just check         # basedpyright type check
just test          # pytest src
just sync          # uv sync --all-packages
just rust-rebuild  # regenerate stubs + rebuild exo_rs
just build-dashboard
just package       # PyInstaller bundle
just build-app     # full macOS app build

19. Testing Strategy

Tests live alongside code (src/exo/**/tests/) and in top-level tests/. Coverage spans:

• Placement & sharding (src/exo/master/tests) — layer allocation, cycle selection, shard assignment correctness.
• Worker plan/runner (src/exo/worker/tests/unittests/ — test_plan, test_runner, test_download, test_image, test_mlx).
• API (src/exo/api/tests) and shared apply/types (src/exo/shared/tests).
• Utilities (src/exo/utils/tests) — channels, info gatherer.
• Rust (rust/exo_rs/tests, rust/networking/examples).
• App (app/EXO/EXOTests, EXOUITests) and eval/bench harnesses.

Run with just test (uv run pytest src) or uv run pytest tests. The macOS app and dashboard have their own test targets.

Note

The tmp/old_tests/ directory contains legacy tests (e.g. test_vision_cache.py) kept for reference; the canonical suites are under each package's tests/.

20. Known Issues & Improvement Areas

Drawn from the codebase's own TODOs and the community issue tracker (as of commit 09f9ea3):

Open bugs to be aware of operationally:

• #2151 Vision feature cache key omits image dimensions → wrong inference on shape-colliding images.
• #1973 RDMA ports stop working after idle.
• #2070 Some MacBook Pro pairings run at ~½ expected RDMA speed.
• #2136 DownloadPending entries for retired node ids are never GC'd → infinite _delete_download loop survives restarts.
• #2120 Runner process can disappear while the instance stays registered → unusable instance.
• #2141 Prefill step size should be 1024, not 4096.
• #2108 Pipeline-parallel deadlock at 32+ concurrent requests (all_gather blocks).
• #2122 Claude Messages API: stop_sequences reporting and multi-token stop-sequence leakage.

Architectural TODOs flagged in-source:

• The Node._elect_loop and master/worker plan loops are explicitly marked as entangled / "cracks in the event-sourcing architecture" awaiting refactor — more behaviors should become commands.
• Shard assignments should carry rank information directly (the JACCL coordinator code works around this).
• Prefill/decode disaggregation placement previews are unimplemented (TODO in get_placement_previews).
• Tensor-parallel feasibility check (hidden_size % n == 0, KV-head divisibility) is acknowledged as "good enough for now," not fully correct.
• User-download notification on missing models is a logged TODO, not yet a real notification.

Improvement opportunities:

• Add authentication / authorization to the API for any non-isolated deployment.
• GC retired-node download entries (#2136).
• Surface a placement-preview path for disaggregation.

21. Building from Source & Packaging

21.1 Toolchain

• Python 3.13, uv for dependency management (workspace members: rust/exo_rs, bench, tools).
• Rust nightly (edition 2024) for the native crates; built via PyO3 into the importable exo_rs module.
• Node for the dashboard (SvelteKit + Vite + Tailwind v4).
• macOS: Xcode (Metal toolchain), Homebrew, pinned macmon fork.

21.2 Dependency notes

The MLX stack is pinned to forks for distributed/RDMA fixes: mlx==0.32.0, mlx-lm (fork leo/deepseek-v4), mflux (fork exo2), plus platform-specific CUDA/CPU MLX wheels for Linux. Optional extras select the backend: exo[mlx] (Metal), exo[mlx-cpu], exo[mlx-cuda12], exo[mlx-cuda13]. The zenoh stack is a patched fork across ~25 crates.

21.3 Build steps

just build-dashboard          # npm install + build (required before running)
just rust-rebuild             # stub-gen + rebuild exo_rs
just package                  # PyInstaller bundle (packaging/pyinstaller/exo.spec)
just build-app                # full macOS .app (xcodebuild)

Packaging artifacts: a PyInstaller bundle, a DMG (packaging/dmg/), and the Xcode-built .app. Nix users get a reproducible build via flake.nix (nix run .#exo).

22. Quick Reference

Ports

Backends & sharding

Key constants

Critical operational rules

• Run the same exo version on all nodes (namespace = version by default).
• Never expose :52415 to untrusted networks — no auth, CORS open.
• Don't pair --legacy-daemon with systemd/launchd.
• Build the dashboard before running from source.
• Only enable trust_remote_code for trusted models.

Links

• Source: https://github.com/exo-explore/exo
• API reference: docs/api.md
• Platform roadmap: PLATFORMS.md
• Contributing: CONTRIBUTING.md · Discord: https://discord.gg/TJ4P57arEm
• License: Apache-2.0