# oMLX — Administrator & Developer Guide

> **Audience:** System administrators, DevOps engineers, and developers deploying, operating, extending, or contributing to oMLX.
> **Version:** 0.3.8.dev1 | Apache-2.0 | macOS 15.0+ / Apple Silicon (M1+) / Python 3.10+

---

## Table of Contents

1. [Architecture Overview](#1-architecture-overview)
2. [System Requirements](#2-system-requirements)
3. [Configuration Reference](#3-configuration-reference)
4. [Deployment & Scaling](#4-deployment--scaling)
5. [Operations](#5-operations)
6. [Performance](#6-performance)
7. [Security](#7-security)
8. [Developer Internals](#8-developer-internals)
9. [Integration Points](#9-integration-points)

---

## 1. Architecture Overview

oMLX is a production-grade, OpenAI-compatible multi-model LLM inference server built exclusively for Apple Silicon. The system leverages Apple's MLX framework for unified-memory GPU acceleration and implements a vLLM-style continuous batching architecture adapted for the Metal/MLX runtime.

### 1.1 High-Level System Diagram

```
┌──────────────────────────────────────────────────────────────┐
│                        Client Layer                          │
│   OpenAI SDK / Anthropic SDK / Codex / curl / Admin UI       │
└─────────────────────────┬────────────────────────────────────┘
                          │ HTTP / SSE / WebSocket
┌─────────────────────────▼────────────────────────────────────┐
│                   FastAPI Server (server.py)                  │
│  /v1/chat/completions  /v1/messages  /v1/embeddings           │
│  /v1/rerank  /v1/responses  /v1/audio/*  /health             │
│                  Admin Dashboard (/admin/*)                   │
└─────────────────────────┬────────────────────────────────────┘
                          │
┌─────────────────────────▼────────────────────────────────────┐
│                    Engine Pool (engine_pool.py)               │
│   LRU Model Eviction │ Memory Tracking │ Model Discovery      │
│   Model Types: llm, vlm, embedding, reranker, audio_*        │
└──────┬──────────┬───────────┬───────────┬────────────────────┘
       │          │           │           │
  ┌────▼───┐ ┌───▼────┐ ┌────▼───┐ ┌────▼────────┐
  │Batched │ │  VLM   │ │Embed-  │ │  Audio      │
  │Engine  │ │Engine  │ │ding    │ │ STT/TTS/STS │
  └────┬───┘ └───┬────┘ │Engine  │ └─────────────┘
       │         │      └────────┘
┌──────▼─────────▼──────────────────────────────────────────────┐
│                   Engine Core (engine_core.py)                 │
│   Single MLX Executor Thread (ThreadPoolExecutor max_workers=1)│
│   Prevents Metal command buffer races                          │
└─────────────────────────┬──────────────────────────────────────┘
                          │
┌─────────────────────────▼──────────────────────────────────────┐
│                  Scheduler (scheduler.py)                       │
│   Continuous Batching │ Prefill/Decode Phases │ KV Allocation  │
│   mlx-lm BatchGenerator                                        │
└──────────────┬──────────────────────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────────────────────┐
│                    Cache Subsystem (cache/)                      │
│  PagedCacheManager → BlockAwarePrefixCache → TieredCacheManager │
│  PagedSSDCacheManager → VisionFeatureSSDCache                   │
│  BoundarySnapshotSSDStore → CacheRecoveryManager                │
└─────────────────────────────────────────────────────────────────┘
```

> [!NOTE]
> The single MLX executor thread is a deliberate architectural constraint. All GPU operations are serialized through a `ThreadPoolExecutor(max_workers=1)` to prevent Metal command buffer races — a hard requirement imposed by the MLX framework's threading model. Do not attempt to parallelize GPU calls; use the batch scheduler instead.

### 1.2 Component Inventory

| Component | Module | Role |
|-----------|--------|------|
| CLI | `omlx/cli.py` | Entry point; `serve`, `launch`, `diagnose` subcommands |
| FastAPI Server | `omlx/server.py` | 102 HTTP endpoints across inference, admin, audio, MCP |
| Engine Pool | `omlx/engine_pool.py` | Multi-model LRU manager; model discovery, load/unload |
| Engine Core | `omlx/engine_core.py` | Async inference loop; single-thread GPU serialization |
| Scheduler | `omlx/scheduler.py` | Continuous batching; prefill/decode; KV allocation |
| Cache — Paged | `omlx/cache/paged_cache.py` | Block-based in-memory KV cache with LRU eviction |
| Cache — Prefix | `omlx/cache/prefix_cache.py` | Block-aware prefix sharing for repeated prompt segments |
| Cache — SSD Paged | `omlx/cache/paged_ssd_cache.py` | SSD persistence for KV blocks; async writer thread |
| Cache — Boundary | `omlx/cache/boundary_ssd.py` | SSD offload for boundary cache snapshots |
| Cache — Vision SSD | `omlx/cache/vision_ssd.py` | SSD cache for VLM vision features |
| Cache — Tiered | `omlx/cache/tiered.py` | Orchestrates hot (GPU) + cold (SSD) cache tiers |
| Cache — Hybrid | `omlx/cache/hybrid.py` | Per-layer cache type configuration |
| Cache — Recovery | `omlx/cache/recovery.py` | Crash recovery for SSD cache state |
| LLM Model | `omlx/models/llm.py` | MLX language model wrapper (load, generate, stream) |
| Embedding Model | `omlx/models/embedding.py` | XLMRoBERTa/BERT native + mlx-embeddings fallback |
| Reranker Model | `omlx/models/reranker.py` | CausalLM, Jina, SequenceClassification architectures |
| VLM Adapter | `omlx/models/vlm.py` | mlx-vlm wrapper for vision-language models |
| Batched Engine | `omlx/engine/batched.py` | Continuous batching for LLMs |
| VLM Engine | `omlx/engine/vlm_batched.py` | Vision-language model engine |
| DFlash Engine | `omlx/engine/dflash.py` | DFlash attention engine |
| Embedding Engine | `omlx/engine/embedding.py` | Batch embedding generation |
| Reranker Engine | `omlx/engine/reranker.py` | Document reranking |
| STT Engine | `omlx/engine/stt.py` | Speech-to-text via mlx-audio |
| TTS Engine | `omlx/engine/tts.py` | Text-to-speech via mlx-audio |
| STS Engine | `omlx/engine/sts.py` | Speech-to-speech via mlx-audio |
| API — OpenAI | `omlx/api/openai_models.py` | OpenAI-format Pydantic models |
| API — Anthropic | `omlx/api/anthropic_models.py`, `anthropic_utils.py` | Anthropic Messages API conversion |
| API — Responses | `omlx/api/responses_models.py`, `responses_utils.py` | Responses API with in-memory store |
| Tool Calling | `omlx/api/tool_calling.py` | Qwen/Llama/Gemma format tool call parsing |
| Thinking | `omlx/api/thinking.py` | `<think>` tag extraction for reasoning models |
| Grammar | `omlx/api/grammar.py` | Structured output via xgrammar |
| MCP Client | `omlx/mcp/client.py` | Individual MCP server connections (stdio/SSE/HTTP) |
| MCP Manager | `omlx/mcp/manager.py` | Multi-server MCP connection management |
| MCP Executor | `omlx/mcp/executor.py` | Tool call execution through MCP |
| Admin Routes | `omlx/admin/routes.py` | 80+ admin API endpoints |
| Admin Auth | `omlx/admin/auth.py` | Session auth with API key and sub-keys |
| Benchmarks | `omlx/admin/benchmark.py` | Throughput benchmarking |
| Accuracy Evals | `omlx/admin/accuracy_benchmark.py` | MMLU, HellaSwag, GSM8K, HumanEval, etc. |
| HF Downloader | `omlx/admin/hf_downloader.py` | HuggingFace model download task manager |
| MS Downloader | `omlx/admin/ms_downloader.py` | ModelScope model download task manager |
| HF Uploader | `omlx/admin/hf_uploader.py` | HuggingFace model upload task manager |
| oQ Manager | `omlx/admin/oq_manager.py` | oQ quantization task management |
| Settings | `omlx/settings.py` | GlobalSettings backed by `~/.omlx/settings.json` |
| Config | `omlx/config.py` | Dataclass-based typed configuration |
| Exceptions | `omlx/exceptions.py` | Rich exception hierarchy (363 classes, 1431 functions) |
| Memory Monitor | `omlx/memory_monitor.py` | Memory pressure detection |
| Process Enforcer | `omlx/process_memory_enforcer.py` | Process memory limit enforcement |
| Model Discovery | `omlx/model_discovery.py` | Auto-discovery from directories |
| Model Settings | `omlx/model_settings.py` | Per-model settings with profiles |
| Model Profiles | `omlx/model_profiles.py` | Named setting profiles and templates |
| Model Registry | `omlx/model_registry.py` | Model ownership tracking (prevents concurrent engine use) |
| Logging | `omlx/logging_config.py` | Structured logging, rotation, request context, colored output |
| Optimizations | `omlx/optimizations.py` | Performance optimizations |
| oQ Quantization | `omlx/oq.py` | Level-based quantization with calibration data |
| TurboQuant KV | `omlx/turboquant_kv.py` | KV cache quantization (int8/int4) |
| Prefill Progress | `omlx/prefill_progress.py` | Prefill progress tracking |
| Server Metrics | `omlx/server_metrics.py` | Metrics collection |
| Output Collector | `omlx/output_collector.py` | Low-latency output collection (vLLM pattern) |
| Harmony Adapter | `omlx/adapter/harmony.py` | Harmony (gpt-oss) streaming format parsing |
| Gemma4 Adapter | `omlx/adapter/gemma4.py` | Gemma 4 reasoning channel parsing |
| Output Parser | `omlx/adapter/output_parser.py` | Generic output parsing interface |
| Patch — GatedDelta | `omlx/patches/gated_delta_advance.py` | Fix for qwen3_5 GatedDeltaNet missing `cache.advance()` |
| Patch — IndexCache | `omlx/patches/index_cache.py` | Index cache optimization |
| Patch — SpecPrefill | `omlx/patches/specprefill.py` | Attention-score-based context pruning |
| Patch — TurboQuant | `omlx/patches/turboquant_attention.py` | TurboQuant KV cache quantization |
| Eval Suite | `omlx/eval/` | 16 benchmarks: MMLU, MMLU-Pro, KMMLU, CMMLU, JMMLU, HellaSwag, TruthfulQA, ARC-Challenge, Winogrande, GSM8K, MathQA, HumanEval, MBPP, LiveCodeBench, BBQ, SafetyBench |
| macOS App | `packaging/` | PyObjC menubar app; DMG via venvstacks |
| Homebrew | `Formula/omlx.rb` | Homebrew formula with optional `--with-grammar` |

### 1.3 Request Data Flow

```
Client Request
    │
    ▼
FastAPI Server (server.py)
    │  verify_api_key middleware
    │  resolve model from request body
    ▼
EnginePool.get_or_load(model_id)
    │  LRU eviction if memory pressure
    │  Load model from disk if not resident
    ▼
Engine dispatch (type-based routing)
    │  BatchedEngine (LLM)
    │  VLMBatchedEngine (VLM)
    │  EmbeddingEngine / RerankerEngine
    │  STT/TTS/STSEngine (audio)
    ▼
EngineCore async loop
    │  scheduler.step() called on MLX executor thread
    ▼
Scheduler (scheduler.py)
    │  Prefill phase → BatchGenerator (mlx-lm)
    │  Decode phase → PagedCacheManager
    │  Prefix cache lookup → BlockAwarePrefixCache
    │  SSD offload → PagedSSDCacheManager
    ▼
RequestOutput → OutputCollector
    │
    ▼
StreamingResponse (SSE) or JSONResponse → Client
```

### 1.4 Repository Layout

```
omlx/
├── omlx/                   # Main package
│   ├── api/                # API models and adapters (OpenAI, Anthropic, Responses)
│   │   ├── adapters/       # SSE formatter, OpenAI adapter, Anthropic adapter
│   │   ├── openai_models.py
│   │   ├── anthropic_models.py / anthropic_utils.py
│   │   ├── responses_models.py / responses_utils.py
│   │   ├── tool_calling.py
│   │   ├── thinking.py
│   │   ├── grammar.py
│   │   ├── embedding_models.py / embedding_utils.py
│   │   ├── rerank_models.py
│   │   ├── audio_routes.py
│   │   └── mcp_routes.py
│   ├── cache/              # KV cache management
│   ├── engine/             # Inference engines
│   ├── mcp/                # Model Context Protocol integration
│   ├── models/             # Model wrappers
│   ├── integrations/       # External tool integrations (Codex, OpenCode, etc.)
│   ├── admin/              # Admin dashboard, benchmarks, downloaders
│   │   └── templates/      # Jinja2 HTML (dashboard, chat, login, settings, etc.)
│   ├── adapter/            # Output format adapters
│   ├── patches/            # Runtime monkey-patches
│   ├── eval/               # 16 benchmark implementations
│   ├── utils/              # Hardware, image, tokenizer utilities
│   ├── server.py           # FastAPI application
│   ├── cli.py              # CLI entry point
│   ├── scheduler.py        # Continuous batching scheduler
│   ├── engine_core.py      # Core async inference engine
│   ├── engine_pool.py      # Multi-model engine management
│   ├── settings.py         # GlobalSettings (JSON-backed)
│   ├── config.py           # Typed dataclass configuration
│   ├── exceptions.py       # Exception hierarchy
│   └── ...
├── packaging/              # macOS menubar app (PyObjC + venvstacks)
├── Formula/                # Homebrew formula (omlx.rb)
├── tests/                  # Test suite (pytest)
└── docs/                   # Documentation
```

---

## 2. System Requirements

### 2.1 Hardware Requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU / SoC | Apple M1 (any variant) | Apple M2 Pro / M3 Max / M4 Max |
| Unified Memory | 8 GB | 32 GB+ |
| Storage | 20 GB free (models excluded) | NVMe SSD, 500 GB+ |
| macOS | 15.0 (Sequoia) | Latest stable release |

> [!NOTE]
> oMLX exclusively targets Apple Silicon. There is no x86/Intel path and no NVIDIA/CUDA support. All GPU operations use the Apple Metal backend via the MLX framework.

### 2.2 Software Requirements

| Dependency | Version | Notes |
|------------|---------|-------|
| macOS | 15.0+ | Sequoia required; earlier versions unsupported |
| Python | 3.10+ | 3.11 recommended (Homebrew formula pins 3.11) |
| mlx-lm | Latest | Core LLM inference |
| mlx-vlm | Latest | Vision-language models |
| mlx-audio | Latest | Audio model support (STT/TTS/STS) |
| mlx-embeddings | Latest | Fallback embedding support |
| FastAPI + uvicorn | Latest | HTTP server |
| xgrammar | Optional | Structured output / constrained generation |
| PyObjC | Optional | macOS menubar app (`packaging/`) |
| Rust toolchain | Build only | Required for Homebrew build |

### 2.3 Installation Methods

**Homebrew (recommended for production):**
```bash
brew tap jundot/omlx
brew install omlx

# With structured output support (xgrammar):
brew install omlx --with-grammar

# Run as a background service:
brew services start omlx
```

**pip (development):**
```bash
pip install omlx
# or from source:
git clone https://github.com/jundot/omlx
cd omlx
pip install -e ".[dev]"
```

**macOS App:**
Download the DMG from the releases page. The menubar app manages server lifecycle, preferences, and auto-updates via PyObjC + venvstacks.

---

## 3. Configuration Reference

### 3.1 Settings Priority

```
CLI arguments  >  environment variables  >  ~/.omlx/settings.json  >  built-in defaults
```

> [!IMPORTANT]
> Settings loaded from `~/.omlx/settings.json` are persistent across restarts. Environment variables override JSON settings but are not persisted. CLI arguments are ephemeral and highest priority. Always verify the effective configuration via `GET /api/global-settings` after startup.

### 3.2 CLI Reference

#### `omlx serve` — Start the inference server

```
omlx serve [OPTIONS]

Core:
  --host TEXT                   Bind host (default: 127.0.0.1)
  --port INT                    Bind port (default: 8080)
  --log-level TEXT              Log level: debug|info|warning|error (default: info)
  --model TEXT                  Default model to load at startup
  --model-dir TEXT              Directory to scan for models

Memory:
  --max-model-memory INT        Maximum memory (MB) for model weights
  --max-process-memory INT      Hard process memory ceiling (MB)

Concurrency:
  --max-concurrent-requests INT Maximum simultaneous inference requests
                                (alias: --max-num-seqs)

Cache:
  --cache-enabled BOOL          Enable KV cache (default: true)
  --ssd-cache-dir TEXT          Path for SSD KV cache storage
  --ssd-cache-max-size INT      Maximum SSD cache size (MB)
  --initial-cache-blocks INT    Initial number of KV cache blocks to allocate
  --hot-cache-only BOOL         Disable SSD offload, keep cache in GPU memory only
  --paged-ssd-cache-dir TEXT    Path for paged SSD KV cache
  --paged-ssd-cache-max-size INT Maximum paged SSD cache size (MB)

Inference:
  --max-tokens INT              Default maximum tokens per response
  --temperature FLOAT           Default sampling temperature
  --continuous-batching BOOL    Enable continuous batching (default: true)
  --trust-remote-code BOOL      Allow execution of remote model code

Auth:
  --api-key TEXT                API key for authentication

MCP:
  --mcp-config TEXT             Path to MCP server config (JSON or YAML)

Network:
  --http-proxy TEXT             HTTP proxy URL
  --https-proxy TEXT            HTTPS proxy URL
  --no-proxy TEXT               Comma-separated hostnames to bypass proxy
  --ca-bundle TEXT              Path to CA bundle for TLS verification
  --hf-endpoint TEXT            HuggingFace Hub endpoint override
  --ms-endpoint TEXT            ModelScope endpoint override

Logging:
  --log-dir TEXT                Directory for log file output
  --log-retention-days INT      Days to retain log files
```

#### `omlx launch` — Launch server with model (convenience wrapper)

```
omlx launch MODEL_ID [OPTIONS]
  All options from `omlx serve` apply; model is positional.
```

#### `omlx diagnose` — System diagnostics

```
omlx diagnose
  Prints hardware info, Python environment, MLX version, detected models,
  memory availability, and connectivity checks. No server required.
```

### 3.3 Environment Variables

All 35 environment variables recognized by oMLX:

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `OMLX_HOST` | string | `127.0.0.1` | Bind host for the HTTP server |
| `OMLX_PORT` | int | `8080` | Bind port for the HTTP server |
| `OMLX_LOG_LEVEL` | string | `info` | Log verbosity: `debug`, `info`, `warning`, `error` |
| `OMLX_MODEL` | string | — | Default model ID to load at startup |
| `OMLX_MODEL_DIR` | string | — | Directory scanned for model discovery |
| `OMLX_MAX_MODEL_MEMORY` | int | — | Maximum memory (MB) for model weights |
| `OMLX_MAX_PROCESS_MEMORY` | int | — | Hard process memory ceiling (MB) |
| `OMLX_MAX_CONCURRENT_REQUESTS` | int | — | Max simultaneous inference requests |
| `OMLX_MAX_NUM_SEQS` | int | — | Alias for `OMLX_MAX_CONCURRENT_REQUESTS` |
| `OMLX_CACHE_ENABLED` | bool | `true` | Enable/disable KV cache |
| `OMLX_SSD_CACHE_DIR` | string | — | Path for SSD KV cache storage |
| `OMLX_SSD_CACHE_MAX_SIZE` | int | — | Maximum SSD cache size in MB |
| `OMLX_INITIAL_CACHE_BLOCKS` | int | — | Initial KV cache block count |
| `OMLX_HOT_CACHE_ONLY` | bool | `false` | Disable SSD offload; keep all cache in GPU RAM |
| `OMLX_API_KEY` | string | — | API key for request authentication |
| `OMLX_MCP_CONFIG` | string | — | Path to MCP server configuration file |
| `OMLX_MAX_TOKENS` | int | — | Default maximum tokens per response |
| `OMLX_TEMPERATURE` | float | — | Default sampling temperature |
| `OMLX_TRUST_REMOTE_CODE` | bool | `false` | Allow remote model code execution |
| `OMLX_CONTINUOUS_BATCHING` | bool | `true` | Enable continuous batching scheduler |
| `OMLX_PAGED_SSD_CACHE_DIR` | string | — | Path for paged SSD KV cache |
| `OMLX_PAGED_SSD_CACHE_MAX_SIZE` | int | — | Maximum paged SSD cache size in MB |
| `OMLX_HF_ENDPOINT` | string | `https://huggingface.co` | HuggingFace Hub endpoint (override for mirrors) |
| `OMLX_MS_ENDPOINT` | string | — | ModelScope endpoint override |
| `OMLX_HTTP_PROXY` | string | — | HTTP proxy URL |
| `OMLX_HTTPS_PROXY` | string | — | HTTPS proxy URL |
| `OMLX_NO_PROXY` | string | — | Comma-separated proxy bypass hostnames |
| `OMLX_CA_BUNDLE` | string | — | Path to custom CA certificate bundle |
| `OMLX_LOG_DIR` | string | — | Directory for log file rotation |
| `OMLX_LOG_RETENTION_DAYS` | int | — | Days to retain rotated log files |
| `OMLX_SECRET_KEY` | string | — | Secret key for session signing (admin auth) |
| `MODELSCOPE_DOMAIN` | string | — | ModelScope domain override |
| `DFLASH_MAX_CTX` | int | — | Maximum context length for DFlash attention engine |

### 3.4 Settings File Reference (`~/.omlx/settings.json`)

The settings file is a JSON document with named sections. Unset keys fall back to built-in defaults.

```json
{
  "server": {
    "host": "127.0.0.1",
    "port": 8080,
    "log_level": "info",
    "secret_key": "<random-string>"
  },
  "model": {
    "default_model": "mlx-community/Mistral-7B-Instruct-v0.3-4bit",
    "model_dir": "/Users/you/models",
    "trust_remote_code": false
  },
  "memory": {
    "max_model_memory": null,
    "max_process_memory": null
  },
  "cache": {
    "enabled": true,
    "initial_blocks": 512,
    "hot_cache_only": false,
    "ssd_cache_dir": null,
    "ssd_cache_max_size": null,
    "paged_ssd_cache_dir": null,
    "paged_ssd_cache_max_size": null
  },
  "auth": {
    "api_key": null
  },
  "huggingface": {
    "endpoint": "https://huggingface.co",
    "token": null
  },
  "modelscope": {
    "endpoint": null,
    "token": null
  },
  "network": {
    "http_proxy": null,
    "https_proxy": null,
    "no_proxy": null,
    "ca_bundle": null
  },
  "mcp": {
    "config": null
  },
  "logging": {
    "log_dir": null,
    "log_retention_days": 7
  }
}
```

Manage settings at runtime via the admin API:
```bash
# Read current effective settings
curl http://localhost:8080/api/global-settings

# Update settings (persists to settings.json)
curl -X POST http://localhost:8080/api/global-settings \
  -H "Content-Type: application/json" \
  -d '{"server": {"log_level": "debug"}}'
```

### 3.5 Typed Configuration Classes (`omlx/config.py`)

The internal dataclass hierarchy used for validated runtime configuration:

| Class | Fields |
|-------|--------|
| `ServerConfig` | host, port, log_level, secret_key |
| `ModelConfig` | default_model, model_dir, trust_remote_code |
| `GenerationConfig` | max_tokens, temperature, top_p, repetition_penalty |
| `SchedulerConfig` | max_num_seqs, continuous_batching, max_batch_size |
| `PagedSSDCacheConfig` | dir, max_size_mb, block_size, async_writer |
| `MCPConfig` | config_path, servers (list) |
| `OMLXConfig` | server, model, generation, scheduler, cache, mcp |

### 3.6 Per-Model Settings and Profiles

Per-model settings override global defaults for specific models. Profiles are named preset collections:

```bash
# Set per-model overrides
curl -X PUT http://localhost:8080/api/models/{model_id}/settings \
  -d '{"max_tokens": 4096, "temperature": 0.7}'

# Create a profile
curl -X POST http://localhost:8080/api/models/{model_id}/profiles \
  -d '{"name": "creative", "settings": {"temperature": 1.2, "top_p": 0.95}}'

# Apply a profile
curl -X POST http://localhost:8080/api/models/{model_id}/profiles/creative/apply

# Manage global profile templates
curl http://localhost:8080/api/profile-templates
curl -X POST http://localhost:8080/api/profile-templates \
  -d '{"name": "precise", "settings": {"temperature": 0.1}}'
```

### 3.7 MCP Configuration

MCP (Model Context Protocol) servers are configured via a JSON or YAML file referenced by `OMLX_MCP_CONFIG` or `--mcp-config`:

```json
{
  "servers": [
    {
      "name": "filesystem",
      "transport": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/files"]
    },
    {
      "name": "remote-tools",
      "transport": "sse",
      "url": "https://mcp.example.com/sse"
    },
    {
      "name": "http-tools",
      "transport": "streamable-http",
      "url": "https://tools.example.com/mcp"
    }
  ]
}
```

---

## 4. Deployment & Scaling

### 4.1 Starting the Server

**Basic start:**
```bash
omlx serve --host 0.0.0.0 --port 8080
```

**With a specific model and auth:**
```bash
omlx serve \
  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --api-key "your-secret-key" \
  --host 0.0.0.0 \
  --port 8080
```

**Via Homebrew service (background daemon):**
```bash
brew services start omlx
brew services stop omlx
brew services restart omlx
# View logs:
tail -f $(brew --prefix)/var/log/omlx.log
```

**Via macOS menubar app:**
Install the DMG. The menubar app starts/stops the server, shows status, and opens the admin dashboard.

### 4.2 Multi-Model Configuration

oMLX supports loading multiple models simultaneously, bounded by available unified memory. Models are managed by the Engine Pool with LRU eviction:

```bash
# Start with a model directory — all models discovered automatically
omlx serve --model-dir /Users/you/mlx-models

# The engine pool loads models on demand. Unload explicitly:
curl -X POST http://localhost:8080/v1/models/model-id/unload

# Check what is currently loaded:
curl http://localhost:8080/v1/models
```

**Engine Pool memory management:**
- When a new model is requested and memory is insufficient, the LRU (least recently used) loaded model is evicted.
- `OMLX_MAX_MODEL_MEMORY` sets the ceiling for total model weight memory.
- `OMLX_MAX_PROCESS_MEMORY` is a hard process-level ceiling enforced by `process_memory_enforcer.py`.
- Monitor current memory state via `GET /api/device-info` and `GET /api/stats`.

### 4.3 Model Types and Engine Routing

The Engine Pool routes model IDs to the correct engine type at load time based on model architecture detection:

| Model Type | Engine | Endpoint(s) |
|------------|--------|-------------|
| `llm` | `BatchedEngine` | `/v1/completions`, `/v1/chat/completions` |
| `vlm` | `VLMBatchedEngine` | `/v1/chat/completions` (with image inputs) |
| `embedding` | `EmbeddingEngine` | `/v1/embeddings` |
| `reranker` | `RerankerEngine` | `/v1/rerank` |
| `audio_stt` | `STTEngine` | `/v1/audio/transcriptions` |
| `audio_tts` | `TTSEngine` | `/v1/audio/speech` |
| `audio_sts` | `STSEngine` | `/v1/audio/process` |

### 4.4 Homebrew Formula Details

The Homebrew formula (`Formula/omlx.rb`) specifics:

- Build dependency: `rust` (compile-time only)
- Runtime dependencies: `python@3.11`, macOS arm64
- Optional: `--with-grammar` installs xgrammar for structured output support
- Resources: mlx-audio (with patched mlx-lm version pin)
- Service support: `brew services start omlx` installs a launchd plist

### 4.5 Reverse Proxy / Production Deployment

For production deployments with TLS and authentication:

```nginx
# nginx example
server {
    listen 443 ssl;
    server_name inference.example.com;

    ssl_certificate /etc/ssl/certs/cert.pem;
    ssl_certificate_key /etc/ssl/private/key.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        # Required for SSE streaming:
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}
```

> [!IMPORTANT]
> When running behind a reverse proxy, ensure `proxy_buffering off` for Server-Sent Events (SSE) streaming responses. Buffering breaks streaming token delivery to clients.

---

## 5. Operations

### 5.1 API Endpoints — Full Reference

All 102 endpoints organized by category.

#### Inference API

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/health` | None | Server health check |
| GET | `/api/status` | None | Detailed server status |
| GET | `/v1/models` | Optional | List available models |
| GET | `/v1/models/status` | Optional | Model load status |
| POST | `/v1/models/{model_id}/unload` | API Key | Unload model from memory |
| POST | `/v1/completions` | API Key | Text completions (OpenAI) |
| POST | `/v1/chat/completions` | API Key | Chat completions (OpenAI) |
| POST | `/v1/messages` | API Key | Anthropic Messages API |
| POST | `/v1/messages/count_tokens` | API Key | Token counting (Anthropic) |
| POST | `/v1/responses` | API Key | Responses API (Codex) |
| GET | `/v1/responses/{response_id}` | API Key | Retrieve stored response |
| DELETE | `/v1/responses/{response_id}` | API Key | Delete stored response |
| POST | `/v1/embeddings` | API Key | Generate embeddings (OpenAI) |
| POST | `/v1/rerank` | API Key | Document reranking |
| POST | `/v1/audio/transcriptions` | API Key | Speech-to-text |
| POST | `/v1/audio/speech` | API Key | Text-to-speech |
| POST | `/v1/audio/process` | API Key | Speech-to-speech |

#### Admin API — Authentication & Account

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/api/login` | None | Authenticate with API key |
| POST | `/api/setup-api-key` | None | First-run API key setup |
| POST | `/api/logout` | Session | End session |
| GET | `/auto-login` | None | Cookie-based auto-login |
| POST | `/api/sub-keys` | Session | Create sub-API key |
| DELETE | `/api/sub-keys` | Session | Delete sub-API key |

#### Admin API — Models

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/models` | Session | List all discovered models |
| POST | `/api/models/{id}/unload` | Session | Unload model from memory |
| POST | `/api/models/{id}/load` | Session | Load model into memory |
| POST | `/api/reload` | Session | Reload all active models |
| PUT | `/api/models/{id}/settings` | Session | Update model settings |
| GET | `/api/models/{id}/generation_config` | Session | Get generation config |
| GET | `/api/models/{id}/profiles` | Session | List model profiles |
| POST | `/api/models/{id}/profiles` | Session | Create model profile |
| PUT | `/api/models/{id}/profiles/{name}` | Session | Update model profile |
| DELETE | `/api/models/{id}/profiles/{name}` | Session | Delete model profile |
| POST | `/api/models/{id}/profiles/{name}/apply` | Session | Apply profile as active settings |
| GET | `/api/profile-templates` | Session | List global profile templates |
| POST | `/api/profile-templates` | Session | Create global profile template |
| PUT | `/api/profile-templates/{name}` | Session | Update global profile template |
| DELETE | `/api/profile-templates/{name}` | Session | Delete global profile template |
| POST | `/api/presets/refresh` | Session | Refresh preset list from disk |

#### Admin API — Server & Monitoring

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/server-info` | Session | Server version, uptime, config |
| GET | `/api/global-settings` | Session | Read current settings.json |
| POST | `/api/global-settings` | Session | Update and persist settings.json |
| GET | `/api/logs` | Session | Tail server log output |
| GET | `/api/stats` | Session | Per-model inference statistics |
| POST | `/api/stats/clear` | Session | Clear stats for current session |
| POST | `/api/stats/clear-alltime` | Session | Clear all-time statistics |
| GET | `/api/device-info` | Session | Apple Silicon device info + memory |
| GET | `/api/update-check` | Session | Check for oMLX updates |
| POST | `/api/cache/probe` | Session | Probe cache state |
| POST | `/api/ssd-cache/clear` | Session | Clear SSD KV cache |
| GET | `/api/grammar/parsers` | Session | List active xgrammar parsers |

#### Admin API — HuggingFace Model Management

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/hf/recommended` | Session | Recommended models from HF Hub |
| GET | `/api/hf/search` | Session | Search HuggingFace Hub |
| GET | `/api/hf/model-info` | Session | Model metadata from HF Hub |
| GET | `/api/hf/models` | Session | Locally downloaded HF models |
| POST | `/api/hf/download` | Session | Start model download task |
| GET | `/api/hf/tasks` | Session | List download tasks |
| POST | `/api/hf/cancel/{id}` | Session | Cancel download task |
| POST | `/api/hf/retry/{id}` | Session | Retry failed download |
| DELETE | `/api/hf/task/{id}` | Session | Remove download task record |
| DELETE | `/api/hf/models/{name}` | Session | Delete local model files |

#### Admin API — ModelScope Model Management

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/ms/status` | Session | ModelScope connectivity status |
| GET | `/api/ms/recommended` | Session | Recommended models from ModelScope |
| GET | `/api/ms/search` | Session | Search ModelScope registry |
| GET | `/api/ms/model-info` | Session | Model metadata from ModelScope |
| POST | `/api/ms/download` | Session | Start model download from ModelScope |
| GET | `/api/ms/tasks` | Session | List ModelScope download tasks |
| POST | `/api/ms/cancel/{id}` | Session | Cancel ModelScope download |
| POST | `/api/ms/retry/{id}` | Session | Retry failed ModelScope download |
| DELETE | `/api/ms/task/{id}` | Session | Remove ModelScope task record |

#### Admin API — Benchmarks

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/api/bench/start` | Session | Start throughput benchmark |
| GET | `/api/bench/{id}/stream` | Session | Stream benchmark progress (SSE) |
| POST | `/api/bench/{id}/cancel` | Session | Cancel running benchmark |
| GET | `/api/bench/{id}/results` | Session | Get benchmark results |
| POST | `/api/bench/accuracy/queue/add` | Session | Queue accuracy benchmark task |
| GET | `/api/bench/accuracy/queue/status` | Session | Accuracy benchmark queue status |
| DELETE | `/api/bench/accuracy/queue/{idx}` | Session | Remove task from accuracy queue |
| GET | `/api/bench/accuracy/results` | Session | All accuracy benchmark results |
| POST | `/api/bench/accuracy/results/reset` | Session | Reset accuracy results |
| POST | `/api/bench/accuracy/cancel` | Session | Cancel running accuracy benchmark |
| GET | `/api/bench/accuracy/{id}/stream` | Session | Stream accuracy benchmark (SSE) |

#### Admin API — Quantization & Upload

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/oq/models` | Session | Models eligible for oQ quantization |
| GET | `/api/oq/estimate` | Session | Estimate oQ quantization time |
| POST | `/api/oq/start` | Session | Start oQ quantization task |
| GET | `/api/oq/tasks` | Session | List quantization tasks |
| POST | `/api/oq/cancel/{id}` | Session | Cancel quantization task |
| DELETE | `/api/oq/task/{id}` | Session | Remove quantization task record |
| POST | `/api/upload/validate-token` | Session | Validate HF upload token |
| GET | `/api/upload/oq-models` | Session | Models ready for upload |
| POST | `/api/upload/start` | Session | Start HuggingFace upload task |
| GET | `/api/upload/tasks` | Session | List upload tasks |
| POST | `/api/upload/cancel/{id}` | Session | Cancel upload task |
| DELETE | `/api/upload/task/{id}` | Session | Remove upload task record |

#### MCP Tool Endpoints

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/tools` | None | List all MCP tools |
| GET | `/servers` | None | List MCP server connections |
| POST | `/execute` | API Key | Execute MCP tool call |

### 5.2 Health and Monitoring

```bash
# Quick health check
curl http://localhost:8080/health
# Response: {"status": "ok"}

# Detailed status
curl http://localhost:8080/api/status

# Per-model statistics
curl http://localhost:8080/api/stats

# Device info (memory usage, SoC details)
curl http://localhost:8080/api/device-info

# Server information
curl http://localhost:8080/api/server-info
```

> [!NOTE]
> The `/health` endpoint requires no authentication and is suitable for load-balancer health checks or uptime monitors. The `/api/status` endpoint returns more detail but also requires no auth.

### 5.3 Log Management

oMLX uses structured logging with rotation (`omlx/logging_config.py`):

```bash
# Configure log directory and retention:
omlx serve --log-dir /var/log/omlx --log-retention-days 14

# Or via environment:
OMLX_LOG_DIR=/var/log/omlx OMLX_LOG_RETENTION_DAYS=14 omlx serve

# Stream logs via admin API (last N lines):
curl http://localhost:8080/api/logs

# Features:
# - Rotating file handler with configurable retention
# - Request context injection (request_id, model_id per line)
# - Colored terminal output for development
# - Structured JSON format option for log aggregation
```

### 5.4 Model Discovery

oMLX auto-discovers models from directories using `omlx/model_discovery.py`. A directory is recognized as a model if it contains:
- `config.json` (model architecture configuration)
- `tokenizer.json` or `tokenizer_config.json`
- MLX weight files (`*.safetensors` or `*.npz`)

```bash
# Point to a directory:
omlx serve --model-dir /Users/you/mlx-models

# Trigger a manual discovery refresh:
curl -X POST http://localhost:8080/api/presets/refresh
```

### 5.5 Model Ownership and Registry

The `ModelRegistry` (`omlx/model_registry.py`) prevents concurrent engine use of the same model. When a model is claimed by an engine, subsequent load attempts for the same model ID will block until the owner releases it. This prevents race conditions during parallel request processing.

---

## 6. Performance

### 6.1 Continuous Batching

oMLX implements vLLM-style continuous batching via the `Scheduler` and `EngineCore`. Key parameters:

```bash
# Enable/disable continuous batching (default: enabled)
OMLX_CONTINUOUS_BATCHING=true omlx serve

# Set maximum concurrent sequences in a batch
OMLX_MAX_CONCURRENT_REQUESTS=8 omlx serve
```

The scheduler manages prefill and decode phases independently, allowing new requests to join ongoing decode batches. This maximizes GPU utilization compared to static batching.

### 6.2 KV Cache Architecture

The cache subsystem has six layers:

| Layer | Class | Location | Purpose |
|-------|-------|----------|---------|
| L1 — Paged | `PagedCacheManager` | GPU (unified memory) | Block-based KV cache with LRU eviction |
| L2 — Prefix | `BlockAwarePrefixCache` | GPU (unified memory) | Prefix block sharing across requests |
| L3 — Tiered | `TieredCacheManager` | GPU (hot) + SSD (cold) | Orchestrates hot/cold promotion |
| L4 — Paged SSD | `PagedSSDCacheManager` | NVMe SSD | Async-write paged KV persistence |
| L5 — Boundary SSD | `BoundarySnapshotSSDStore` | NVMe SSD | Boundary cache snapshot offload |
| L6 — Vision SSD | `VisionFeatureSSDCache` | NVMe SSD | VLM vision feature persistence |

```bash
# Configure SSD cache:
omlx serve \
  --ssd-cache-dir ~/.omlx/cache \
  --ssd-cache-max-size 51200 \
  --paged-ssd-cache-dir ~/.omlx/paged-cache \
  --paged-ssd-cache-max-size 102400 \
  --initial-cache-blocks 1024

# Disable SSD offload (pure GPU cache):
omlx serve --hot-cache-only true

# Clear SSD cache at runtime:
curl -X POST http://localhost:8080/api/ssd-cache/clear

# Probe cache state:
curl -X POST http://localhost:8080/api/cache/probe
```

### 6.3 DFlash Attention Engine

The `DFlashEngine` provides an alternative attention implementation for supported model architectures. To configure maximum context length:

```bash
DFLASH_MAX_CTX=131072 omlx serve
```

> [!NOTE]
> DFlash requires specific model architectures. See known issues: models with 58 unmatched parameters will fail to load (issue #886). Verify architecture compatibility before enabling DFlash for new models.

### 6.4 SpecPrefill (Context Pruning)

`omlx/patches/specprefill.py` implements attention-score-based context pruning: low-attention tokens are dropped during prefill to reduce KV cache pressure for long contexts. This is a runtime monkey-patch applied transparently.

### 6.5 oQ Quantization

oMLX includes an on-device quantization pipeline (`omlx/oq.py`) for reducing model memory footprint:

```bash
# Check eligible models:
curl http://localhost:8080/api/oq/models

# Estimate quantization time:
curl "http://localhost:8080/api/oq/estimate?model_id=MODEL_ID&level=4"

# Start quantization:
curl -X POST http://localhost:8080/api/oq/start \
  -d '{"model_id": "MODEL_ID", "level": 4}'

# Monitor tasks:
curl http://localhost:8080/api/oq/tasks
```

Quantization levels are integer-valued (lower = more compression, higher = better quality). Calibration data is used to preserve accuracy at lower bit widths.

> [!WARNING]
> Quantizing a model while it is loaded in an active engine is not supported. Unload the model before starting oQ. Issue #913 documents that oQ on Qwen3.6 27B can break model weights — verify outputs against the unquantized model after any quantization pass.

### 6.6 TurboQuant KV Cache Quantization

`omlx/turboquant_kv.py` quantizes the KV cache itself (int8 or int4) to reduce GPU memory consumption during inference. Configured via the `omlx/patches/turboquant_attention.py` monkey-patch applied at startup.

### 6.7 Performance Benchmarking

```bash
# Start throughput benchmark via admin API:
curl -X POST http://localhost:8080/api/bench/start \
  -d '{"model_id": "MODEL_ID", "num_prompts": 100, "input_len": 256, "output_len": 64}'

# Stream results in real time:
curl http://localhost:8080/api/bench/BENCH_ID/stream

# Accuracy benchmarks (16 datasets):
curl -X POST http://localhost:8080/api/bench/accuracy/queue/add \
  -d '{"model_id": "MODEL_ID", "benchmarks": ["mmlu", "hellaswag", "gsm8k"]}'

curl http://localhost:8080/api/bench/accuracy/queue/status
```

Supported accuracy benchmarks: MMLU, MMLU-Pro, KMMLU, CMMLU, JMMLU, HellaSwag, TruthfulQA, ARC-Challenge, Winogrande, GSM8K, MathQA, HumanEval, MBPP, LiveCodeBench, BBQ, SafetyBench.

---

## 7. Security

### 7.1 Authentication Model

oMLX supports two authentication modes:

**API Key authentication** (inference endpoints):
```bash
# Pass via header:
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"model": "...", "messages": [...]}'

# Configure:
OMLX_API_KEY=your-secret-key omlx serve
```

**Session-based authentication** (admin dashboard):
- Login via `POST /api/login` with the API key
- Session cookie issued; subsequent admin API calls use the session
- Sub-keys allow delegated access with scoped permissions:
  ```bash
  curl -X POST http://localhost:8080/api/sub-keys \
    -d '{"name": "read-only-key", "permissions": ["read"]}'
  ```

> [!WARNING]
> If `OMLX_API_KEY` is not set, the inference endpoints accept unauthenticated requests. Always configure an API key before exposing oMLX on a network interface other than `127.0.0.1`.

### 7.2 Secret Key

`OMLX_SECRET_KEY` signs admin session cookies. If not set, a random value is generated at startup (sessions do not survive restarts). For persistent sessions across restarts, set a fixed secret:

```bash
# Generate a strong secret:
python3 -c "import secrets; print(secrets.token_hex(32))"

# Set in settings.json under server.secret_key, or:
OMLX_SECRET_KEY=<generated-value> omlx serve
```

> [!CAUTION]
> Never commit or log the `OMLX_SECRET_KEY` or `OMLX_API_KEY` values. Rotate them by stopping the server, updating the configuration, and restarting. Existing sessions are invalidated on key rotation.

### 7.3 Trust Remote Code

By default `OMLX_TRUST_REMOTE_CODE=false`. Setting this to `true` allows model `config.json` files to execute arbitrary Python code during model loading (via the `transformers`/`mlx-lm` trust_remote_code path).

Only enable this for models you own or have fully audited. Malicious models can exploit this to execute code with the server process's privileges.

### 7.4 Network Exposure

The default bind address `127.0.0.1` limits access to localhost. To expose on a network interface:

```bash
omlx serve --host 0.0.0.0 --port 8080 --api-key "strong-secret"
```

For internet-facing deployments, place oMLX behind a reverse proxy that handles TLS. Do not expose the admin dashboard (`/admin/*`, `/api/*`) to untrusted networks.

### 7.5 Proxy and TLS Configuration

For environments requiring traffic routing through corporate proxies or custom CA certificates:

```bash
OMLX_HTTPS_PROXY=http://proxy.corp.example.com:8080 \
OMLX_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem \
OMLX_NO_PROXY=localhost,127.0.0.1,internal.example.com \
omlx serve
```

---

## 8. Developer Internals

### 8.1 Exception Hierarchy

`omlx/exceptions.py` defines a rich exception hierarchy. All oMLX exceptions descend from `OMLXError`:

```
OMLXError
├── CacheError
│   ├── CacheEvictionError
│   ├── CacheCorruptionError
│   └── SSDCacheError
├── SchedulerError
│   ├── SchedulerCapacityError
│   └── SchedulerStateError
├── ModelError
│   ├── ModelLoadError
│   ├── ModelNotFoundError
│   ├── ModelArchitectureError
│   └── ModelMemoryError
├── APIError
│   ├── AuthenticationError
│   ├── RateLimitError
│   └── ValidationError
├── EngineError
│   ├── EngineInitError
│   └── EngineExecutionError
└── ConfigError
```

Catch `OMLXError` to handle any oMLX-originating exception. Prefer specific subclasses in code that can meaningfully distinguish failure modes.

### 8.2 Engine Core — Single-Thread GPU Model

The `EngineCore` (`omlx/engine_core.py`) is the critical concurrency bottleneck:

```python
# Conceptual structure (simplified)
class EngineCore:
    def __init__(self):
        self._executor = ThreadPoolExecutor(max_workers=1)  # SINGLE thread
        self._scheduler = Scheduler(...)

    async def step(self):
        # All GPU ops run on the single executor thread
        outputs = await asyncio.get_event_loop().run_in_executor(
            self._executor,
            self._scheduler.step  # MLX/Metal calls happen here
        )
        # Distribute outputs to waiting request collectors
        for output in outputs:
            self._output_collectors[output.request_id].put(output)
```

The `OutputCollector` (`omlx/output_collector.py`) implements a low-latency per-request buffer following the vLLM pattern. The `asyncio` event loop awaits GPU results without blocking; token streaming to clients happens from the output collector as each token is produced.

### 8.3 Scheduler Design

`omlx/scheduler.py` wraps `mlx-lm`'s `BatchGenerator` and implements:
- **Prefill phase**: processes new request prompts in batch
- **Decode phase**: autoregressively generates tokens for in-flight requests
- **KV allocation**: reserves blocks from `PagedCacheManager` before prefill
- **Preemption**: evicts low-priority requests to SSD cache under memory pressure

The scheduler does not implement speculative decoding at the scheduler level; that is handled by the SpecPrefill patch at the attention layer.

### 8.4 Runtime Monkey-Patches

Patches in `omlx/patches/` are applied at import time or during engine initialization:

| Patch | Module | What It Fixes |
|-------|--------|--------------|
| `gated_delta_advance.py` | qwen3_5 models | GatedDeltaNet model missing `cache.advance()` method |
| `index_cache.py` | Cache layer | Index cache read optimization |
| `specprefill.py` | Attention layer | Attention-score-based context pruning |
| `turboquant_attention.py` | Attention layer | Injects int8/int4 KV quantization |

Patches modify MLX or mlx-lm internals at runtime without forking the upstream libraries. When upgrading mlx-lm or mlx-vlm, verify that patched interfaces have not changed.

### 8.5 API Adapters

The `omlx/api/adapters/` layer transforms between wire formats:

- **SSE Formatter**: converts token output to `text/event-stream` chunks in OpenAI format
- **OpenAI Adapter**: normalizes internal `RequestOutput` to OpenAI `ChatCompletionChunk`
- **Anthropic Adapter**: converts internal format to Anthropic `MessageStreamEvent`
- **Base Adapter**: abstract interface for new format additions

Tool call parsing (`omlx/api/tool_calling.py`) handles three model-specific formats:
- **Qwen format**: JSON function calls in assistant messages
- **Llama format**: `<tool_call>` XML-ish tags
- **Gemma format**: Structured function call tokens

Thinking/reasoning extraction (`omlx/api/thinking.py`) strips `<think>...</think>` blocks from model output and surfaces them separately in the API response for reasoning-capable models (e.g., Qwen3, DeepSeek-R1 variants).

### 8.6 Structured Output (Grammar)

Structured output is implemented via xgrammar (`omlx/api/grammar.py`). This feature is optional and requires the `--with-grammar` Homebrew flag or manual xgrammar installation:

```bash
# Request JSON schema-constrained output:
curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "...",
    "messages": [...],
    "response_format": {
      "type": "json_schema",
      "json_schema": {"name": "result", "schema": {"type": "object", ...}}
    }
  }'

# Check active grammar parsers:
curl http://localhost:8080/api/grammar/parsers
```

### 8.7 Module Reference — Core Modules

#### `omlx/settings.py` — GlobalSettings

Singleton backed by `~/.omlx/settings.json`. Sections: server, model, memory, cache, auth, huggingface, modelscope, network, mcp, logging. Settings are loaded once at startup; runtime updates via `POST /api/global-settings` write through to the JSON file and update the in-process singleton.

#### `omlx/config.py` — Typed Configuration

Dataclass hierarchy: `OMLXConfig` aggregates `ServerConfig`, `ModelConfig`, `GenerationConfig`, `SchedulerConfig`, `PagedSSDCacheConfig`, `MCPConfig`. Used internally; not directly user-facing.

#### `omlx/engine_pool.py` — Engine Pool

Maintains a registry of loaded engines keyed by model ID. On `get_or_load(model_id)`:
1. If loaded and not stale: return existing engine
2. If memory insufficient: LRU evict the least recently used engine
3. Load model from disk, instantiate correct engine type, register

#### `omlx/model_discovery.py` — Model Discovery

Recursively scans `--model-dir` for directories meeting the three-file criterion (`config.json` + tokenizer + weights). Results are cached with mtime-based invalidation. Refresh via `POST /api/presets/refresh`.

#### `omlx/memory_monitor.py` — Memory Pressure Detection

Polls unified memory pressure at configurable intervals. Emits `MemoryPressureEvent` to registered listeners (engine pool, scheduler) when pressure crosses thresholds. Drives proactive LRU eviction before hard OOM.

#### `omlx/logging_config.py` — Structured Logging

Configures the root logger with:
- `TimedRotatingFileHandler` (daily rotation, configurable retention)
- `StreamHandler` with colored output (level-colored, for TTY)
- Request context injection via `contextvars` (request_id, model_id appear in every log line within a request scope)

#### `omlx/server_metrics.py` — Metrics Collection

Collects per-model statistics: request count, token throughput (input/output tokens/sec), latency percentiles (p50/p95/p99), cache hit rates. Persisted in memory; reset via admin API. Surfaced at `GET /api/stats`.

### 8.8 Admin Dashboard Frontend

The admin dashboard (`omlx/admin/templates/`) uses:
- **Jinja2** templating (server-side rendered HTML shell)
- **Tailwind CSS** for styling
- **Alpine.js** for reactive UI state
- **Lucide icons** for iconography
- **highlight.js** for code syntax highlighting in chat view
- **KaTeX** for mathematical formula rendering
- **marked.js** for Markdown rendering in chat

Pages: dashboard (overview), chat (interactive model playground), login, settings, models (management), benchmarks, logs.

### 8.9 Testing

Tests live in `tests/`. The pytest suite uses custom markers:

```bash
# Run all tests:
pytest tests/

# Run only fast unit tests (exclude slow and integration):
pytest tests/ -m "not slow and not integration"

# Run integration tests (requires running server):
pytest tests/ -m integration

# Run slow tests (long-running benchmarks/accuracy evals):
pytest tests/ -m slow
```

**Pytest markers:**
- `slow` — long-running tests (benchmark execution, large model loading)
- `integration` — require a live oMLX server; use `--server-url` fixture

**Test structure:**
```
tests/
├── unit/           # Pure unit tests, no server required
├── integration/    # Require live server (marked integration)
├── benchmarks/     # Performance regression tests (marked slow)
└── conftest.py     # Shared fixtures (server_url, api_client, etc.)
```

### 8.10 Contributing

Based on standard conventions in the repository:

**Development setup:**
```bash
git clone https://github.com/jundot/omlx
cd omlx
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Verify environment:
omlx diagnose
```

**Code standards:**
- Python type hints on all public interfaces
- Docstrings for all public classes and functions
- Tests for new engine types and cache variants
- Patches in `omlx/patches/` must include a comment explaining what upstream bug/limitation they address and when they can be removed

**Pull request checklist:**
- `pytest tests/ -m "not slow and not integration"` passes locally
- New environment variables added to the `settings.py` mapping and this guide
- Monkey-patches documented in `omlx/patches/` module docstring
- New API endpoints added to `omlx/admin/routes.py` router inventory

---

## 9. Integration Points

### 9.1 OpenAI SDK Compatibility

oMLX is a drop-in replacement for the OpenAI API for chat, completions, and embeddings:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-omlx-api-key"
)

response = client.chat.completions.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True
)
```

### 9.2 Anthropic SDK Compatibility

The `/v1/messages` and `/v1/messages/count_tokens` endpoints implement the Anthropic Messages API:

```python
import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:8080",
    api_key="your-omlx-api-key"
)

response = client.messages.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}]
)
```

### 9.3 Codex / OpenCode Compatibility

The `/v1/responses` endpoint implements the OpenAI Responses API for Codex and OpenCode compatibility. Responses are stored in-memory and retrievable by ID. The `omlx/integrations/` directory contains additional integration shims for specific tools.

### 9.4 Model Context Protocol (MCP)

oMLX acts as an MCP client, connecting to external tool servers and exposing their tools to inference requests. The `MCPClientManager` maintains persistent connections to all configured servers.

**Using MCP tools in inference:**
```json
{
  "model": "mlx-community/Qwen2.5-7B-Instruct-4bit",
  "messages": [...],
  "tools": "auto"
}
```

When `tools: "auto"` is specified, oMLX injects available MCP tools into the context. Tool calls in model output are routed through `ToolExecutor` to the appropriate MCP server.

**Direct MCP tool execution:**
```bash
# List available tools:
curl http://localhost:8080/tools

# List connected servers:
curl http://localhost:8080/servers

# Execute a tool directly:
curl -X POST http://localhost:8080/execute \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"tool_name": "read_file", "arguments": {"path": "/tmp/test.txt"}}'
```

### 9.5 HuggingFace Ecosystem

oMLX integrates with the HuggingFace Hub for model discovery, downloading, and uploading:

```bash
# Configure HF token (required for gated models):
# In settings.json: huggingface.token = "hf_..."
# Or via admin settings UI

# Download a model:
curl -X POST http://localhost:8080/api/hf/download \
  -d '{"model_id": "mlx-community/Llama-3.2-3B-Instruct-4bit"}'

# Monitor download progress:
curl http://localhost:8080/api/hf/tasks

# Upload a quantized model:
curl -X POST http://localhost:8080/api/upload/start \
  -d '{"model_id": "local-model-name", "repo_id": "your-org/model-name"}'
```

For users in China or with restricted Hub access, override the HuggingFace endpoint:
```bash
OMLX_HF_ENDPOINT=https://hf-mirror.com omlx serve
```

### 9.6 ModelScope Integration

ModelScope (Chinese ML model registry) is a first-class integration:

```bash
# Configure ModelScope token in settings.json:
# modelscope.token = "your-token"
# modelscope.endpoint = "https://modelscope.cn"  (or override)

# Download from ModelScope:
curl -X POST http://localhost:8080/api/ms/download \
  -d '{"model_id": "qwen/Qwen2.5-7B-Instruct"}'
```

---

## Troubleshooting

### Error: `DFlash model fails to load: 58 parameters not matching architecture`

**Cause:** The model's weight tensors do not match the DFlash engine's expected architecture. Typically affects models with non-standard layer configurations (issue #886).

**Solution:** Switch the model to the standard `BatchedEngine` by removing any explicit DFlash configuration for this model. If DFlash is required, check that the model was exported in a DFlash-compatible format.

**Prevention:** Run `omlx diagnose` or attempt a load via `POST /api/models/{id}/load` before putting a model into production use. Verify architecture compatibility in the model's `config.json` before enabling DFlash.

---

### Error: `SpecPrefill system_end calc failed`

**Cause:** The SpecPrefill context pruning patch encountered an invalid boundary calculation, usually from an unexpected prompt structure or token sequence (issue #916).

**Solution:** The request will fall back to full-context prefill automatically. If this error is frequent, disable SpecPrefill by ensuring the `specprefill.py` patch is not applied, or open an issue with the prompt that triggers it.

**Prevention:** Test prompts with unusual structures (e.g., empty system messages, unusual special tokens) against the SpecPrefill patch on upgrade.

---

### Error: `RuntimeError: no Stream(gpu, 0) in current thread`

**Cause:** An MLX GPU operation was called outside the designated single executor thread (issue #904). This can occur on M4 Max or when third-party code calls MLX directly.

**Solution:** Ensure all MLX compute calls go through `EngineCore`'s executor thread. Do not call MLX operations from asyncio coroutines, background threads, or signal handlers directly.

**Prevention:** Custom engine extensions must use `run_in_executor` with the engine's `_executor` to schedule any MLX calls. Never call `mx.eval()` or similar outside the executor thread.

---

### Error: `ToolCallStreamFilter crashes on streaming tool calls`

**Cause:** The tool call stream parser encountered malformed JSON in a partial streaming chunk (issue #882). Typically affects models that split JSON tool call tokens across multiple SSE events.

**Solution:** For affected models, disable streaming for tool-call-heavy workflows (`"stream": false` in the request). Upgrade to the latest patch version, as this was fixed post-#882.

**Prevention:** When adding new tool-calling models, test streaming with multi-step tool calls before deploying.

---

### Error: `Slow generation speed on 0.3.7` (or current version)

**Cause:** Regression in continuous batching throughput (issues #937/#936). May also relate to SSD cache write pressure or prefix cache miss rates.

**Solution:**
1. Check `GET /api/stats` for cache hit rates and token throughput.
2. Try `--hot-cache-only true` to isolate SSD write overhead.
3. Reduce `OMLX_MAX_CONCURRENT_REQUESTS` to decrease scheduler overhead.
4. Clear stats (`POST /api/stats/clear`) and re-benchmark after each change.

**Prevention:** Run throughput benchmarks via `POST /api/bench/start` after every upgrade before moving to production.

---

### Error: `Infinite loop with DFlash + certain models`

**Cause:** DFlash attention engine enters an infinite decode loop with specific model+prompt combinations (issues #911/#934/#935). Root cause is an edge case in DFlash's attention mask handling.

**Solution:** Unload the model (`POST /api/models/{id}/unload`), restart the server if the loop consumes all memory, and switch to the standard `BatchedEngine` for the affected model.

> [!WARNING]
> An infinite loop in the DFlash engine will exhaust unified memory and cause the macOS kernel to terminate the process under memory pressure. Monitor memory via `GET /api/device-info` when testing new models with DFlash.

**Prevention:** Test new models with DFlash on short inputs first. Add the model to an explicit blocklist in model settings if DFlash loops are observed.

---

### Error: `oQ on Qwen3.6 27b breaks model`

**Cause:** The oQ quantization calibration pass for Qwen3.6 27B modifies weight tensors in a way that corrupts output (issue #913). This may affect other large models with similar architectures.

**Solution:** Restore from the original (pre-quantization) model files. Always back up the original model directory before running oQ.

**Prevention:** Before any oQ quantization:
1. Copy the model directory to a backup location.
2. Run a short accuracy benchmark (`POST /api/bench/accuracy/queue/add`) on the original.
3. Run oQ, then repeat the benchmark and compare scores.
4. If accuracy drops significantly, restore from backup.

> [!NOTE]
> The `DELETE /api/hf/models/{name}` endpoint permanently deletes local model files. Confirm you have either the original on HuggingFace or a local backup before deleting a quantized model you wish to revert.

---

*Quick Reference — Common `curl` Commands*

```bash
# Health check
curl http://localhost:8080/health

# List models
curl http://localhost:8080/v1/models

# Chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Hello"}]}'

# Streaming chat
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer KEY" \
  -d '{"model":"MODEL_ID","messages":[...],"stream":true}'

# Embeddings
curl http://localhost:8080/v1/embeddings \
  -H "Authorization: Bearer KEY" \
  -d '{"model":"EMBED_MODEL_ID","input":"text to embed"}'

# Unload model
curl -X POST http://localhost:8080/v1/models/MODEL_ID/unload \
  -H "Authorization: Bearer KEY"

# Stats
curl http://localhost:8080/api/stats

# Device info (memory)
curl http://localhost:8080/api/device-info
```

---

_Generated May 2026 from jundot/omlx at f4ed78d. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._