oMLX — Administrator & Developer Guide

Audience: System administrators, DevOps engineers, and developers deploying, operating, extending, or contributing to oMLX. Version: 0.3.8.dev1 | Apache-2.0 | macOS 15.0+ / Apple Silicon (M1+) / Python 3.10+

Table of Contents

1. Architecture Overview
2. System Requirements
3. Configuration Reference
4. Deployment & Scaling
5. Operations
6. Performance
7. Security
8. Developer Internals
9. Integration Points

1. Architecture Overview

oMLX is a production-grade, OpenAI-compatible multi-model LLM inference server built exclusively for Apple Silicon. The system leverages Apple's MLX framework for unified-memory GPU acceleration and implements a vLLM-style continuous batching architecture adapted for the Metal/MLX runtime.

1.1 High-Level System Diagram

┌──────────────────────────────────────────────────────────────┐
│                        Client Layer                          │
│   OpenAI SDK / Anthropic SDK / Codex / curl / Admin UI       │
└─────────────────────────┬────────────────────────────────────┘
                          │ HTTP / SSE / WebSocket
┌─────────────────────────▼────────────────────────────────────┐
│                   FastAPI Server (server.py)                  │
│  /v1/chat/completions  /v1/messages  /v1/embeddings           │
│  /v1/rerank  /v1/responses  /v1/audio/*  /health             │
│                  Admin Dashboard (/admin/*)                   │
└─────────────────────────┬────────────────────────────────────┘
                          │
┌─────────────────────────▼────────────────────────────────────┐
│                    Engine Pool (engine_pool.py)               │
│   LRU Model Eviction │ Memory Tracking │ Model Discovery      │
│   Model Types: llm, vlm, embedding, reranker, audio_*        │
└──────┬──────────┬───────────┬───────────┬────────────────────┘
       │          │           │           │
  ┌────▼───┐ ┌───▼────┐ ┌────▼───┐ ┌────▼────────┐
  │Batched │ │  VLM   │ │Embed-  │ │  Audio      │
  │Engine  │ │Engine  │ │ding    │ │ STT/TTS/STS │
  └────┬───┘ └───┬────┘ │Engine  │ └─────────────┘
       │         │      └────────┘
┌──────▼─────────▼──────────────────────────────────────────────┐
│                   Engine Core (engine_core.py)                 │
│   Single MLX Executor Thread (ThreadPoolExecutor max_workers=1)│
│   Prevents Metal command buffer races                          │
└─────────────────────────┬──────────────────────────────────────┘
                          │
┌─────────────────────────▼──────────────────────────────────────┐
│                  Scheduler (scheduler.py)                       │
│   Continuous Batching │ Prefill/Decode Phases │ KV Allocation  │
│   mlx-lm BatchGenerator                                        │
└──────────────┬──────────────────────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────────────────────┐
│                    Cache Subsystem (cache/)                      │
│  PagedCacheManager → BlockAwarePrefixCache → TieredCacheManager │
│  PagedSSDCacheManager → VisionFeatureSSDCache                   │
│  BoundarySnapshotSSDStore → CacheRecoveryManager                │
└─────────────────────────────────────────────────────────────────┘

Note

The single MLX executor thread is a deliberate architectural constraint. All GPU operations are serialized through a ThreadPoolExecutor(max_workers=1) to prevent Metal command buffer races — a hard requirement imposed by the MLX framework's threading model. Do not attempt to parallelize GPU calls; use the batch scheduler instead.

1.2 Component Inventory

1.3 Request Data Flow

Client Request
    │
    ▼
FastAPI Server (server.py)
    │  verify_api_key middleware
    │  resolve model from request body
    ▼
EnginePool.get_or_load(model_id)
    │  LRU eviction if memory pressure
    │  Load model from disk if not resident
    ▼
Engine dispatch (type-based routing)
    │  BatchedEngine (LLM)
    │  VLMBatchedEngine (VLM)
    │  EmbeddingEngine / RerankerEngine
    │  STT/TTS/STSEngine (audio)
    ▼
EngineCore async loop
    │  scheduler.step() called on MLX executor thread
    ▼
Scheduler (scheduler.py)
    │  Prefill phase → BatchGenerator (mlx-lm)
    │  Decode phase → PagedCacheManager
    │  Prefix cache lookup → BlockAwarePrefixCache
    │  SSD offload → PagedSSDCacheManager
    ▼
RequestOutput → OutputCollector
    │
    ▼
StreamingResponse (SSE) or JSONResponse → Client

1.4 Repository Layout

omlx/
├── omlx/                   # Main package
│   ├── api/                # API models and adapters (OpenAI, Anthropic, Responses)
│   │   ├── adapters/       # SSE formatter, OpenAI adapter, Anthropic adapter
│   │   ├── openai_models.py
│   │   ├── anthropic_models.py / anthropic_utils.py
│   │   ├── responses_models.py / responses_utils.py
│   │   ├── tool_calling.py
│   │   ├── thinking.py
│   │   ├── grammar.py
│   │   ├── embedding_models.py / embedding_utils.py
│   │   ├── rerank_models.py
│   │   ├── audio_routes.py
│   │   └── mcp_routes.py
│   ├── cache/              # KV cache management
│   ├── engine/             # Inference engines
│   ├── mcp/                # Model Context Protocol integration
│   ├── models/             # Model wrappers
│   ├── integrations/       # External tool integrations (Codex, OpenCode, etc.)
│   ├── admin/              # Admin dashboard, benchmarks, downloaders
│   │   └── templates/      # Jinja2 HTML (dashboard, chat, login, settings, etc.)
│   ├── adapter/            # Output format adapters
│   ├── patches/            # Runtime monkey-patches
│   ├── eval/               # 16 benchmark implementations
│   ├── utils/              # Hardware, image, tokenizer utilities
│   ├── server.py           # FastAPI application
│   ├── cli.py              # CLI entry point
│   ├── scheduler.py        # Continuous batching scheduler
│   ├── engine_core.py      # Core async inference engine
│   ├── engine_pool.py      # Multi-model engine management
│   ├── settings.py         # GlobalSettings (JSON-backed)
│   ├── config.py           # Typed dataclass configuration
│   ├── exceptions.py       # Exception hierarchy
│   └── ...
├── packaging/              # macOS menubar app (PyObjC + venvstacks)
├── Formula/                # Homebrew formula (omlx.rb)
├── tests/                  # Test suite (pytest)
└── docs/                   # Documentation

2. System Requirements

2.1 Hardware Requirements

Note

oMLX exclusively targets Apple Silicon. There is no x86/Intel path and no NVIDIA/CUDA support. All GPU operations use the Apple Metal backend via the MLX framework.

2.2 Software Requirements

2.3 Installation Methods

Homebrew (recommended for production):

brew tap jundot/omlx
brew install omlx

# With structured output support (xgrammar):
brew install omlx --with-grammar

# Run as a background service:
brew services start omlx

pip (development):

pip install omlx
# or from source:
git clone https://github.com/jundot/omlx
cd omlx
pip install -e ".[dev]"

macOS App: Download the DMG from the releases page. The menubar app manages server lifecycle, preferences, and auto-updates via PyObjC + venvstacks.

3. Configuration Reference

3.1 Settings Priority

CLI arguments  >  environment variables  >  ~/.omlx/settings.json  >  built-in defaults

Important

Settings loaded from ~/.omlx/settings.json are persistent across restarts. Environment variables override JSON settings but are not persisted. CLI arguments are ephemeral and highest priority. Always verify the effective configuration via GET /api/global-settings after startup.

3.2 CLI Reference

omlx serve — Start the inference server

omlx serve [OPTIONS]

Core:
  --host TEXT                   Bind host (default: 127.0.0.1)
  --port INT                    Bind port (default: 8080)
  --log-level TEXT              Log level: debug|info|warning|error (default: info)
  --model TEXT                  Default model to load at startup
  --model-dir TEXT              Directory to scan for models

Memory:
  --max-model-memory INT        Maximum memory (MB) for model weights
  --max-process-memory INT      Hard process memory ceiling (MB)

Concurrency:
  --max-concurrent-requests INT Maximum simultaneous inference requests
                                (alias: --max-num-seqs)

Cache:
  --cache-enabled BOOL          Enable KV cache (default: true)
  --ssd-cache-dir TEXT          Path for SSD KV cache storage
  --ssd-cache-max-size INT      Maximum SSD cache size (MB)
  --initial-cache-blocks INT    Initial number of KV cache blocks to allocate
  --hot-cache-only BOOL         Disable SSD offload, keep cache in GPU memory only
  --paged-ssd-cache-dir TEXT    Path for paged SSD KV cache
  --paged-ssd-cache-max-size INT Maximum paged SSD cache size (MB)

Inference:
  --max-tokens INT              Default maximum tokens per response
  --temperature FLOAT           Default sampling temperature
  --continuous-batching BOOL    Enable continuous batching (default: true)
  --trust-remote-code BOOL      Allow execution of remote model code

Auth:
  --api-key TEXT                API key for authentication

MCP:
  --mcp-config TEXT             Path to MCP server config (JSON or YAML)

Network:
  --http-proxy TEXT             HTTP proxy URL
  --https-proxy TEXT            HTTPS proxy URL
  --no-proxy TEXT               Comma-separated hostnames to bypass proxy
  --ca-bundle TEXT              Path to CA bundle for TLS verification
  --hf-endpoint TEXT            HuggingFace Hub endpoint override
  --ms-endpoint TEXT            ModelScope endpoint override

Logging:
  --log-dir TEXT                Directory for log file output
  --log-retention-days INT      Days to retain log files

omlx launch — Launch server with model (convenience wrapper)

omlx launch MODEL_ID [OPTIONS]
  All options from `omlx serve` apply; model is positional.

omlx diagnose — System diagnostics

omlx diagnose
  Prints hardware info, Python environment, MLX version, detected models,
  memory availability, and connectivity checks. No server required.

3.3 Environment Variables

All 35 environment variables recognized by oMLX:

3.4 Settings File Reference (~/.omlx/settings.json)

The settings file is a JSON document with named sections. Unset keys fall back to built-in defaults.

{
  "server": {
    "host": "127.0.0.1",
    "port": 8080,
    "log_level": "info",
    "secret_key": "<random-string>"
  },
  "model": {
    "default_model": "mlx-community/Mistral-7B-Instruct-v0.3-4bit",
    "model_dir": "/Users/you/models",
    "trust_remote_code": false
  },
  "memory": {
    "max_model_memory": null,
    "max_process_memory": null
  },
  "cache": {
    "enabled": true,
    "initial_blocks": 512,
    "hot_cache_only": false,
    "ssd_cache_dir": null,
    "ssd_cache_max_size": null,
    "paged_ssd_cache_dir": null,
    "paged_ssd_cache_max_size": null
  },
  "auth": {
    "api_key": null
  },
  "huggingface": {
    "endpoint": "https://huggingface.co",
    "token": null
  },
  "modelscope": {
    "endpoint": null,
    "token": null
  },
  "network": {
    "http_proxy": null,
    "https_proxy": null,
    "no_proxy": null,
    "ca_bundle": null
  },
  "mcp": {
    "config": null
  },
  "logging": {
    "log_dir": null,
    "log_retention_days": 7
  }
}

Manage settings at runtime via the admin API:

# Read current effective settings
curl http://localhost:8080/api/global-settings

# Update settings (persists to settings.json)
curl -X POST http://localhost:8080/api/global-settings \
  -H "Content-Type: application/json" \
  -d '{"server": {"log_level": "debug"}}'

3.5 Typed Configuration Classes (omlx/config.py)

The internal dataclass hierarchy used for validated runtime configuration:

3.6 Per-Model Settings and Profiles

Per-model settings override global defaults for specific models. Profiles are named preset collections:

# Set per-model overrides
curl -X PUT http://localhost:8080/api/models/{model_id}/settings \
  -d '{"max_tokens": 4096, "temperature": 0.7}'

# Create a profile
curl -X POST http://localhost:8080/api/models/{model_id}/profiles \
  -d '{"name": "creative", "settings": {"temperature": 1.2, "top_p": 0.95}}'

# Apply a profile
curl -X POST http://localhost:8080/api/models/{model_id}/profiles/creative/apply

# Manage global profile templates
curl http://localhost:8080/api/profile-templates
curl -X POST http://localhost:8080/api/profile-templates \
  -d '{"name": "precise", "settings": {"temperature": 0.1}}'

3.7 MCP Configuration

MCP (Model Context Protocol) servers are configured via a JSON or YAML file referenced by OMLX_MCP_CONFIG or --mcp-config:

{
  "servers": [
    {
      "name": "filesystem",
      "transport": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/files"]
    },
    {
      "name": "remote-tools",
      "transport": "sse",
      "url": "https://mcp.example.com/sse"
    },
    {
      "name": "http-tools",
      "transport": "streamable-http",
      "url": "https://tools.example.com/mcp"
    }
  ]
}

4. Deployment & Scaling

4.1 Starting the Server

Basic start:

omlx serve --host 0.0.0.0 --port 8080

With a specific model and auth:

omlx serve \
  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --api-key "your-secret-key" \
  --host 0.0.0.0 \
  --port 8080

Via Homebrew service (background daemon):

brew services start omlx
brew services stop omlx
brew services restart omlx
# View logs:
tail -f $(brew --prefix)/var/log/omlx.log

Via macOS menubar app: Install the DMG. The menubar app starts/stops the server, shows status, and opens the admin dashboard.

4.2 Multi-Model Configuration

oMLX supports loading multiple models simultaneously, bounded by available unified memory. Models are managed by the Engine Pool with LRU eviction:

# Start with a model directory — all models discovered automatically
omlx serve --model-dir /Users/you/mlx-models

# The engine pool loads models on demand. Unload explicitly:
curl -X POST http://localhost:8080/v1/models/model-id/unload

# Check what is currently loaded:
curl http://localhost:8080/v1/models

Engine Pool memory management:

• When a new model is requested and memory is insufficient, the LRU (least recently used) loaded model is evicted.
• OMLX_MAX_MODEL_MEMORY sets the ceiling for total model weight memory.
• OMLX_MAX_PROCESS_MEMORY is a hard process-level ceiling enforced by process_memory_enforcer.py.
• Monitor current memory state via GET /api/device-info and GET /api/stats.

4.3 Model Types and Engine Routing

The Engine Pool routes model IDs to the correct engine type at load time based on model architecture detection:

4.4 Homebrew Formula Details

The Homebrew formula (Formula/omlx.rb) specifics:

• Build dependency: rust (compile-time only)
• Runtime dependencies: python@3.11, macOS arm64
• Optional: --with-grammar installs xgrammar for structured output support
• Resources: mlx-audio (with patched mlx-lm version pin)
• Service support: brew services start omlx installs a launchd plist

4.5 Reverse Proxy / Production Deployment

For production deployments with TLS and authentication:

# nginx example
server {
    listen 443 ssl;
    server_name inference.example.com;

    ssl_certificate /etc/ssl/certs/cert.pem;
    ssl_certificate_key /etc/ssl/private/key.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        # Required for SSE streaming:
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Important

When running behind a reverse proxy, ensure proxy_buffering off for Server-Sent Events (SSE) streaming responses. Buffering breaks streaming token delivery to clients.

5. Operations

5.1 API Endpoints — Full Reference

All 102 endpoints organized by category.

Inference API

Admin API — Authentication & Account

Admin API — Models

Admin API — Server & Monitoring

Admin API — HuggingFace Model Management

Admin API — ModelScope Model Management

Admin API — Benchmarks

Admin API — Quantization & Upload

MCP Tool Endpoints

5.2 Health and Monitoring

# Quick health check
curl http://localhost:8080/health
# Response: {"status": "ok"}

# Detailed status
curl http://localhost:8080/api/status

# Per-model statistics
curl http://localhost:8080/api/stats

# Device info (memory usage, SoC details)
curl http://localhost:8080/api/device-info

# Server information
curl http://localhost:8080/api/server-info

Note

The /health endpoint requires no authentication and is suitable for load-balancer health checks or uptime monitors. The /api/status endpoint returns more detail but also requires no auth.

5.3 Log Management

oMLX uses structured logging with rotation (omlx/logging_config.py):

# Configure log directory and retention:
omlx serve --log-dir /var/log/omlx --log-retention-days 14

# Or via environment:
OMLX_LOG_DIR=/var/log/omlx OMLX_LOG_RETENTION_DAYS=14 omlx serve

# Stream logs via admin API (last N lines):
curl http://localhost:8080/api/logs

# Features:
# - Rotating file handler with configurable retention
# - Request context injection (request_id, model_id per line)
# - Colored terminal output for development
# - Structured JSON format option for log aggregation

5.4 Model Discovery

oMLX auto-discovers models from directories using omlx/model_discovery.py. A directory is recognized as a model if it contains:

• config.json (model architecture configuration)
• tokenizer.json or tokenizer_config.json
• MLX weight files (*.safetensors or *.npz)

# Point to a directory:
omlx serve --model-dir /Users/you/mlx-models

# Trigger a manual discovery refresh:
curl -X POST http://localhost:8080/api/presets/refresh

5.5 Model Ownership and Registry

The ModelRegistry (omlx/model_registry.py) prevents concurrent engine use of the same model. When a model is claimed by an engine, subsequent load attempts for the same model ID will block until the owner releases it. This prevents race conditions during parallel request processing.

6. Performance

6.1 Continuous Batching

oMLX implements vLLM-style continuous batching via the Scheduler and EngineCore. Key parameters:

# Enable/disable continuous batching (default: enabled)
OMLX_CONTINUOUS_BATCHING=true omlx serve

# Set maximum concurrent sequences in a batch
OMLX_MAX_CONCURRENT_REQUESTS=8 omlx serve

The scheduler manages prefill and decode phases independently, allowing new requests to join ongoing decode batches. This maximizes GPU utilization compared to static batching.

6.2 KV Cache Architecture

The cache subsystem has six layers:

# Configure SSD cache:
omlx serve \
  --ssd-cache-dir ~/.omlx/cache \
  --ssd-cache-max-size 51200 \
  --paged-ssd-cache-dir ~/.omlx/paged-cache \
  --paged-ssd-cache-max-size 102400 \
  --initial-cache-blocks 1024

# Disable SSD offload (pure GPU cache):
omlx serve --hot-cache-only true

# Clear SSD cache at runtime:
curl -X POST http://localhost:8080/api/ssd-cache/clear

# Probe cache state:
curl -X POST http://localhost:8080/api/cache/probe

6.3 DFlash Attention Engine

The DFlashEngine provides an alternative attention implementation for supported model architectures. To configure maximum context length:

DFLASH_MAX_CTX=131072 omlx serve

Note

DFlash requires specific model architectures. See known issues: models with 58 unmatched parameters will fail to load (issue #886). Verify architecture compatibility before enabling DFlash for new models.

6.4 SpecPrefill (Context Pruning)

omlx/patches/specprefill.py implements attention-score-based context pruning: low-attention tokens are dropped during prefill to reduce KV cache pressure for long contexts. This is a runtime monkey-patch applied transparently.

6.5 oQ Quantization

oMLX includes an on-device quantization pipeline (omlx/oq.py) for reducing model memory footprint:

# Check eligible models:
curl http://localhost:8080/api/oq/models

# Estimate quantization time:
curl "http://localhost:8080/api/oq/estimate?model_id=MODEL_ID&level=4"

# Start quantization:
curl -X POST http://localhost:8080/api/oq/start \
  -d '{"model_id": "MODEL_ID", "level": 4}'

# Monitor tasks:
curl http://localhost:8080/api/oq/tasks

Quantization levels are integer-valued (lower = more compression, higher = better quality). Calibration data is used to preserve accuracy at lower bit widths.

Warning

Quantizing a model while it is loaded in an active engine is not supported. Unload the model before starting oQ. Issue #913 documents that oQ on Qwen3.6 27B can break model weights — verify outputs against the unquantized model after any quantization pass.

6.6 TurboQuant KV Cache Quantization

omlx/turboquant_kv.py quantizes the KV cache itself (int8 or int4) to reduce GPU memory consumption during inference. Configured via the omlx/patches/turboquant_attention.py monkey-patch applied at startup.

6.7 Performance Benchmarking

# Start throughput benchmark via admin API:
curl -X POST http://localhost:8080/api/bench/start \
  -d '{"model_id": "MODEL_ID", "num_prompts": 100, "input_len": 256, "output_len": 64}'

# Stream results in real time:
curl http://localhost:8080/api/bench/BENCH_ID/stream

# Accuracy benchmarks (16 datasets):
curl -X POST http://localhost:8080/api/bench/accuracy/queue/add \
  -d '{"model_id": "MODEL_ID", "benchmarks": ["mmlu", "hellaswag", "gsm8k"]}'

curl http://localhost:8080/api/bench/accuracy/queue/status

Supported accuracy benchmarks: MMLU, MMLU-Pro, KMMLU, CMMLU, JMMLU, HellaSwag, TruthfulQA, ARC-Challenge, Winogrande, GSM8K, MathQA, HumanEval, MBPP, LiveCodeBench, BBQ, SafetyBench.

7. Security

7.1 Authentication Model

oMLX supports two authentication modes:

API Key authentication (inference endpoints):

# Pass via header:
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"model": "...", "messages": [...]}'

# Configure:
OMLX_API_KEY=your-secret-key omlx serve

Session-based authentication (admin dashboard):

• Login via POST /api/login with the API key
• Session cookie issued; subsequent admin API calls use the session
• Sub-keys allow delegated access with scoped permissions:

  curl -X POST http://localhost:8080/api/sub-keys \
    -d '{"name": "read-only-key", "permissions": ["read"]}'
  

Warning

If OMLX_API_KEY is not set, the inference endpoints accept unauthenticated requests. Always configure an API key before exposing oMLX on a network interface other than 127.0.0.1.

7.2 Secret Key

OMLX_SECRET_KEY signs admin session cookies. If not set, a random value is generated at startup (sessions do not survive restarts). For persistent sessions across restarts, set a fixed secret:

# Generate a strong secret:
python3 -c "import secrets; print(secrets.token_hex(32))"

# Set in settings.json under server.secret_key, or:
OMLX_SECRET_KEY=<generated-value> omlx serve

Caution

Never commit or log the OMLX_SECRET_KEY or OMLX_API_KEY values. Rotate them by stopping the server, updating the configuration, and restarting. Existing sessions are invalidated on key rotation.

7.3 Trust Remote Code

By default OMLX_TRUST_REMOTE_CODE=false. Setting this to true allows model config.json files to execute arbitrary Python code during model loading (via the transformers/mlx-lm trust_remote_code path).

Only enable this for models you own or have fully audited. Malicious models can exploit this to execute code with the server process's privileges.

7.4 Network Exposure

The default bind address 127.0.0.1 limits access to localhost. To expose on a network interface:

omlx serve --host 0.0.0.0 --port 8080 --api-key "strong-secret"

For internet-facing deployments, place oMLX behind a reverse proxy that handles TLS. Do not expose the admin dashboard (/admin/*, /api/*) to untrusted networks.

7.5 Proxy and TLS Configuration

For environments requiring traffic routing through corporate proxies or custom CA certificates:

OMLX_HTTPS_PROXY=http://proxy.corp.example.com:8080 \
OMLX_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem \
OMLX_NO_PROXY=localhost,127.0.0.1,internal.example.com \
omlx serve

8. Developer Internals

8.1 Exception Hierarchy

omlx/exceptions.py defines a rich exception hierarchy. All oMLX exceptions descend from OMLXError:

OMLXError
├── CacheError
│   ├── CacheEvictionError
│   ├── CacheCorruptionError
│   └── SSDCacheError
├── SchedulerError
│   ├── SchedulerCapacityError
│   └── SchedulerStateError
├── ModelError
│   ├── ModelLoadError
│   ├── ModelNotFoundError
│   ├── ModelArchitectureError
│   └── ModelMemoryError
├── APIError
│   ├── AuthenticationError
│   ├── RateLimitError
│   └── ValidationError
├── EngineError
│   ├── EngineInitError
│   └── EngineExecutionError
└── ConfigError

Catch OMLXError to handle any oMLX-originating exception. Prefer specific subclasses in code that can meaningfully distinguish failure modes.

8.2 Engine Core — Single-Thread GPU Model

The EngineCore (omlx/engine_core.py) is the critical concurrency bottleneck:

# Conceptual structure (simplified)
class EngineCore:
    def __init__(self):
        self._executor = ThreadPoolExecutor(max_workers=1)  # SINGLE thread
        self._scheduler = Scheduler(...)

    async def step(self):
        # All GPU ops run on the single executor thread
        outputs = await asyncio.get_event_loop().run_in_executor(
            self._executor,
            self._scheduler.step  # MLX/Metal calls happen here
        )
        # Distribute outputs to waiting request collectors
        for output in outputs:
            self._output_collectors[output.request_id].put(output)

The OutputCollector (omlx/output_collector.py) implements a low-latency per-request buffer following the vLLM pattern. The asyncio event loop awaits GPU results without blocking; token streaming to clients happens from the output collector as each token is produced.

8.3 Scheduler Design

omlx/scheduler.py wraps mlx-lm's BatchGenerator and implements:

• Prefill phase: processes new request prompts in batch
• Decode phase: autoregressively generates tokens for in-flight requests
• KV allocation: reserves blocks from PagedCacheManager before prefill
• Preemption: evicts low-priority requests to SSD cache under memory pressure

The scheduler does not implement speculative decoding at the scheduler level; that is handled by the SpecPrefill patch at the attention layer.

8.4 Runtime Monkey-Patches

Patches in omlx/patches/ are applied at import time or during engine initialization:

Patches modify MLX or mlx-lm internals at runtime without forking the upstream libraries. When upgrading mlx-lm or mlx-vlm, verify that patched interfaces have not changed.

8.5 API Adapters

The omlx/api/adapters/ layer transforms between wire formats:

• SSE Formatter: converts token output to text/event-stream chunks in OpenAI format
• OpenAI Adapter: normalizes internal RequestOutput to OpenAI ChatCompletionChunk
• Anthropic Adapter: converts internal format to Anthropic MessageStreamEvent
• Base Adapter: abstract interface for new format additions

Tool call parsing (omlx/api/tool_calling.py) handles three model-specific formats:

• Qwen format: JSON function calls in assistant messages
• Llama format: <tool_call> XML-ish tags
• Gemma format: Structured function call tokens

Thinking/reasoning extraction (omlx/api/thinking.py) strips <think>...</think> blocks from model output and surfaces them separately in the API response for reasoning-capable models (e.g., Qwen3, DeepSeek-R1 variants).

8.6 Structured Output (Grammar)

Structured output is implemented via xgrammar (omlx/api/grammar.py). This feature is optional and requires the --with-grammar Homebrew flag or manual xgrammar installation:

# Request JSON schema-constrained output:
curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "...",
    "messages": [...],
    "response_format": {
      "type": "json_schema",
      "json_schema": {"name": "result", "schema": {"type": "object", ...}}
    }
  }'

# Check active grammar parsers:
curl http://localhost:8080/api/grammar/parsers

8.7 Module Reference — Core Modules

omlx/settings.py — GlobalSettings

Singleton backed by ~/.omlx/settings.json. Sections: server, model, memory, cache, auth, huggingface, modelscope, network, mcp, logging. Settings are loaded once at startup; runtime updates via POST /api/global-settings write through to the JSON file and update the in-process singleton.

omlx/config.py — Typed Configuration

Dataclass hierarchy: OMLXConfig aggregates ServerConfig, ModelConfig, GenerationConfig, SchedulerConfig, PagedSSDCacheConfig, MCPConfig. Used internally; not directly user-facing.

omlx/engine_pool.py — Engine Pool

Maintains a registry of loaded engines keyed by model ID. On get_or_load(model_id):

1. If loaded and not stale: return existing engine
2. If memory insufficient: LRU evict the least recently used engine
3. Load model from disk, instantiate correct engine type, register

omlx/model_discovery.py — Model Discovery

Recursively scans --model-dir for directories meeting the three-file criterion (config.json + tokenizer + weights). Results are cached with mtime-based invalidation. Refresh via POST /api/presets/refresh.

omlx/memory_monitor.py — Memory Pressure Detection

Polls unified memory pressure at configurable intervals. Emits MemoryPressureEvent to registered listeners (engine pool, scheduler) when pressure crosses thresholds. Drives proactive LRU eviction before hard OOM.

omlx/logging_config.py — Structured Logging

Configures the root logger with:

• TimedRotatingFileHandler (daily rotation, configurable retention)
• StreamHandler with colored output (level-colored, for TTY)
• Request context injection via contextvars (request_id, model_id appear in every log line within a request scope)

omlx/server_metrics.py — Metrics Collection

Collects per-model statistics: request count, token throughput (input/output tokens/sec), latency percentiles (p50/p95/p99), cache hit rates. Persisted in memory; reset via admin API. Surfaced at GET /api/stats.

8.8 Admin Dashboard Frontend

The admin dashboard (omlx/admin/templates/) uses:

• Jinja2 templating (server-side rendered HTML shell)
• Tailwind CSS for styling
• Alpine.js for reactive UI state
• Lucide icons for iconography
• highlight.js for code syntax highlighting in chat view
• KaTeX for mathematical formula rendering
• marked.js for Markdown rendering in chat

Pages: dashboard (overview), chat (interactive model playground), login, settings, models (management), benchmarks, logs.

8.9 Testing

Tests live in tests/. The pytest suite uses custom markers:

# Run all tests:
pytest tests/

# Run only fast unit tests (exclude slow and integration):
pytest tests/ -m "not slow and not integration"

# Run integration tests (requires running server):
pytest tests/ -m integration

# Run slow tests (long-running benchmarks/accuracy evals):
pytest tests/ -m slow

Pytest markers:

• slow — long-running tests (benchmark execution, large model loading)
• integration — require a live oMLX server; use --server-url fixture

Test structure:

tests/
├── unit/           # Pure unit tests, no server required
├── integration/    # Require live server (marked integration)
├── benchmarks/     # Performance regression tests (marked slow)
└── conftest.py     # Shared fixtures (server_url, api_client, etc.)

8.10 Contributing

Based on standard conventions in the repository:

Development setup:

git clone https://github.com/jundot/omlx
cd omlx
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Verify environment:
omlx diagnose

Code standards:

• Python type hints on all public interfaces
• Docstrings for all public classes and functions
• Tests for new engine types and cache variants
• Patches in omlx/patches/ must include a comment explaining what upstream bug/limitation they address and when they can be removed

Pull request checklist:

• pytest tests/ -m "not slow and not integration" passes locally
• New environment variables added to the settings.py mapping and this guide
• Monkey-patches documented in omlx/patches/ module docstring
• New API endpoints added to omlx/admin/routes.py router inventory

9. Integration Points

9.1 OpenAI SDK Compatibility

oMLX is a drop-in replacement for the OpenAI API for chat, completions, and embeddings:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-omlx-api-key"
)

response = client.chat.completions.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True
)

9.2 Anthropic SDK Compatibility

The /v1/messages and /v1/messages/count_tokens endpoints implement the Anthropic Messages API:

import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:8080",
    api_key="your-omlx-api-key"
)

response = client.messages.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}]
)

9.3 Codex / OpenCode Compatibility

The /v1/responses endpoint implements the OpenAI Responses API for Codex and OpenCode compatibility. Responses are stored in-memory and retrievable by ID. The omlx/integrations/ directory contains additional integration shims for specific tools.

9.4 Model Context Protocol (MCP)

oMLX acts as an MCP client, connecting to external tool servers and exposing their tools to inference requests. The MCPClientManager maintains persistent connections to all configured servers.

Using MCP tools in inference:

{
  "model": "mlx-community/Qwen2.5-7B-Instruct-4bit",
  "messages": [...],
  "tools": "auto"
}

When tools: "auto" is specified, oMLX injects available MCP tools into the context. Tool calls in model output are routed through ToolExecutor to the appropriate MCP server.

Direct MCP tool execution:

# List available tools:
curl http://localhost:8080/tools

# List connected servers:
curl http://localhost:8080/servers

# Execute a tool directly:
curl -X POST http://localhost:8080/execute \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"tool_name": "read_file", "arguments": {"path": "/tmp/test.txt"}}'

9.5 HuggingFace Ecosystem

oMLX integrates with the HuggingFace Hub for model discovery, downloading, and uploading:

# Configure HF token (required for gated models):
# In settings.json: huggingface.token = "hf_..."
# Or via admin settings UI

# Download a model:
curl -X POST http://localhost:8080/api/hf/download \
  -d '{"model_id": "mlx-community/Llama-3.2-3B-Instruct-4bit"}'

# Monitor download progress:
curl http://localhost:8080/api/hf/tasks

# Upload a quantized model:
curl -X POST http://localhost:8080/api/upload/start \
  -d '{"model_id": "local-model-name", "repo_id": "your-org/model-name"}'

For users in China or with restricted Hub access, override the HuggingFace endpoint:

OMLX_HF_ENDPOINT=https://hf-mirror.com omlx serve

9.6 ModelScope Integration

ModelScope (Chinese ML model registry) is a first-class integration:

# Configure ModelScope token in settings.json:
# modelscope.token = "your-token"
# modelscope.endpoint = "https://modelscope.cn"  (or override)

# Download from ModelScope:
curl -X POST http://localhost:8080/api/ms/download \
  -d '{"model_id": "qwen/Qwen2.5-7B-Instruct"}'

Troubleshooting

Error: DFlash model fails to load: 58 parameters not matching architecture

Cause: The model's weight tensors do not match the DFlash engine's expected architecture. Typically affects models with non-standard layer configurations (issue #886).

Solution: Switch the model to the standard BatchedEngine by removing any explicit DFlash configuration for this model. If DFlash is required, check that the model was exported in a DFlash-compatible format.

Prevention: Run omlx diagnose or attempt a load via POST /api/models/{id}/load before putting a model into production use. Verify architecture compatibility in the model's config.json before enabling DFlash.

Error: SpecPrefill system_end calc failed

Cause: The SpecPrefill context pruning patch encountered an invalid boundary calculation, usually from an unexpected prompt structure or token sequence (issue #916).

Solution: The request will fall back to full-context prefill automatically. If this error is frequent, disable SpecPrefill by ensuring the specprefill.py patch is not applied, or open an issue with the prompt that triggers it.

Prevention: Test prompts with unusual structures (e.g., empty system messages, unusual special tokens) against the SpecPrefill patch on upgrade.

Error: RuntimeError: no Stream(gpu, 0) in current thread

Cause: An MLX GPU operation was called outside the designated single executor thread (issue #904). This can occur on M4 Max or when third-party code calls MLX directly.

Solution: Ensure all MLX compute calls go through EngineCore's executor thread. Do not call MLX operations from asyncio coroutines, background threads, or signal handlers directly.

Prevention: Custom engine extensions must use run_in_executor with the engine's _executor to schedule any MLX calls. Never call mx.eval() or similar outside the executor thread.

Error: ToolCallStreamFilter crashes on streaming tool calls

Cause: The tool call stream parser encountered malformed JSON in a partial streaming chunk (issue #882). Typically affects models that split JSON tool call tokens across multiple SSE events.

Solution: For affected models, disable streaming for tool-call-heavy workflows ("stream": false in the request). Upgrade to the latest patch version, as this was fixed post-#882.

Prevention: When adding new tool-calling models, test streaming with multi-step tool calls before deploying.

Error: Slow generation speed on 0.3.7 (or current version)

Cause: Regression in continuous batching throughput (issues #937/#936). May also relate to SSD cache write pressure or prefix cache miss rates.

Solution:

1. Check GET /api/stats for cache hit rates and token throughput.
2. Try --hot-cache-only true to isolate SSD write overhead.
3. Reduce OMLX_MAX_CONCURRENT_REQUESTS to decrease scheduler overhead.
4. Clear stats (POST /api/stats/clear) and re-benchmark after each change.

Prevention: Run throughput benchmarks via POST /api/bench/start after every upgrade before moving to production.

Error: Infinite loop with DFlash + certain models

Cause: DFlash attention engine enters an infinite decode loop with specific model+prompt combinations (issues #911/#934/#935). Root cause is an edge case in DFlash's attention mask handling.

Solution: Unload the model (POST /api/models/{id}/unload), restart the server if the loop consumes all memory, and switch to the standard BatchedEngine for the affected model.

Warning

An infinite loop in the DFlash engine will exhaust unified memory and cause the macOS kernel to terminate the process under memory pressure. Monitor memory via GET /api/device-info when testing new models with DFlash.

Prevention: Test new models with DFlash on short inputs first. Add the model to an explicit blocklist in model settings if DFlash loops are observed.

Error: oQ on Qwen3.6 27b breaks model

Cause: The oQ quantization calibration pass for Qwen3.6 27B modifies weight tensors in a way that corrupts output (issue #913). This may affect other large models with similar architectures.

Solution: Restore from the original (pre-quantization) model files. Always back up the original model directory before running oQ.

Prevention: Before any oQ quantization:

1. Copy the model directory to a backup location.
2. Run a short accuracy benchmark (POST /api/bench/accuracy/queue/add) on the original.
3. Run oQ, then repeat the benchmark and compare scores.
4. If accuracy drops significantly, restore from backup.

Note

The DELETE /api/hf/models/{name} endpoint permanently deletes local model files. Confirm you have either the original on HuggingFace or a local backup before deleting a quantized model you wish to revert.

Quick Reference — Common curl Commands

# Health check
curl http://localhost:8080/health

# List models
curl http://localhost:8080/v1/models

# Chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Hello"}]}'

# Streaming chat
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer KEY" \
  -d '{"model":"MODEL_ID","messages":[...],"stream":true}'

# Embeddings
curl http://localhost:8080/v1/embeddings \
  -H "Authorization: Bearer KEY" \
  -d '{"model":"EMBED_MODEL_ID","input":"text to embed"}'

# Unload model
curl -X POST http://localhost:8080/v1/models/MODEL_ID/unload \
  -H "Authorization: Bearer KEY"

# Stats
curl http://localhost:8080/api/stats

# Device info (memory)
curl http://localhost:8080/api/device-info