oMLX — User Guide (jundot/omlx): guide library

oMLX is a production-ready, OpenAI-compatible LLM inference server built for Apple Silicon Macs. It runs entirely on-device using Apple's MLX framework — no cloud API costs, no GPU required, no data leaving your machine. Drop it in as a local replacement for OpenAI's API and your existing tools keep working.

Note

This guide covers everyday use: installing, configuring, serving models, and calling the API. Architecture internals, deployment configuration, and security hardening are in the Admin Guide (Admin Guide).

1. Quick Start

Get a server running and answer your first request in under five minutes.

Prerequisites

Apple Silicon Mac (M1 or later)
macOS 15.0 Sequoia or later
Python 3.10 or later

Install and Start

pip install omlx

Start the server with a model:

omlx serve --model-dir ~/.omlx/models

Tip

On first launch, oMLX creates ~/.omlx/ automatically. Download a model from the admin dashboard at http://127.0.0.1:8000/dashboard before making your first inference request.

Once the server is running, verify it is healthy:

curl http://127.0.0.1:8000/health

Expected response:

{"status": "ok"}

List loaded models:

curl http://127.0.0.1:8000/v1/models

Expected response (once a model is loaded):

{
  "object": "list",
  "data": [
    {
      "id": "llama-3b",
      "object": "model",
      "created": 1746624000,
      "owned_by": "local"
    }
  ]
}

Make your first chat request:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}]
  }'

2. Installation

Method 1: pip (Recommended)

Core install (text generation, embeddings, reranking):

pip install omlx

With audio support (speech-to-text and text-to-speech):

pip install omlx[audio]

With structured output (JSON schema enforcement via xgrammar):

pip install omlx[grammar]

With ModelScope (model downloads from ModelScope Hub):

pip install omlx[modelscope]

Full install (everything):

pip install "omlx[audio,grammar,modelscope]"

Important

oMLX requires macOS 15.0 (Sequoia) or later and an Apple Silicon chip (M1, M2, M3, or M4 series). It will not run on Intel Macs or any other platform.

Method 2: Homebrew

brew tap jundot/tap
brew install jundot/tap/omlx

With structured output (xgrammar) support:

brew install jundot/tap/omlx --with-grammar

Method 3: macOS Menubar App (DMG)

Download the DMG installer from the releases page, open it, drag oMLX to your Applications folder, and launch it. The menubar icon gives you one-click start/stop and quick access to the dashboard.

If the menubar icon does not appear (a known issue with macOS Tahoe and Control Center), run:

omlx diagnose menubar

This prints the steps to restore icon visibility.

Method 4: From Source

git clone https://github.com/jundot/omlx
cd omlx
pip install -e ".[dev]"

Verifying the Install

omlx --version
# omlx 0.3.7

omlx serve --help

Warning

If pip install omlx succeeds but omlx is not found on your PATH, your Python bin directory is not in your shell's PATH. Run python -m omlx serve as a workaround, or add the bin path: export PATH="$(python -m site --user-base)/bin:$PATH".

3. Configuration

oMLX is configured through a combination of CLI flags and environment variables. CLI flags take precedence over environment variables.

Key Environment Variables

Set these in your shell profile (~/.zshrc or ~/.bash_profile) to avoid repeating flags on every launch.

Variable	Purpose	Example
`OMLX_HOST`	Bind host	`0.0.0.0`
`OMLX_PORT`	Bind port	`8000`
`OMLX_MODEL_DIR`	Model storage directory	`/Volumes/FastDrive/models`
`OMLX_MAX_MODEL_MEMORY`	Memory cap for loaded models	`32GB`
`OMLX_MAX_PROCESS_MEMORY`	Total process memory cap	`80%`
`OMLX_MAX_CONCURRENT_REQUESTS`	Request concurrency limit	`4`
`OMLX_API_KEY`	Authentication key	`sk-my-local-key`
`OMLX_LOG_LEVEL`	Logging verbosity	`debug`
`OMLX_LOG_DIR`	Log file directory	`~/.omlx/logs`
`OMLX_LOG_RETENTION_DAYS`	Days to keep log files	`7`
`OMLX_CACHE_ENABLED`	Enable/disable KV cache	`true`
`OMLX_SSD_CACHE_DIR`	SSD prefix cache directory	`/Volumes/SSD/omlx-cache`
`OMLX_SSD_CACHE_MAX_SIZE`	Max SSD cache size	`200GB`
`OMLX_HOT_CACHE_ONLY`	Use RAM-only hot cache	`true`
`OMLX_INITIAL_CACHE_BLOCKS`	Pre-allocated cache blocks	`512`
`OMLX_MCP_CONFIG`	MCP tools config file path	`~/.omlx/mcp.json`
`OMLX_HF_ENDPOINT`	Custom HuggingFace Hub URL	`https://hf-mirror.com`
`OMLX_MS_ENDPOINT`	Custom ModelScope Hub URL	`https://modelscope.cn`
`OMLX_HTTP_PROXY`	HTTP proxy	`http://proxy.corp:8080`
`OMLX_HTTPS_PROXY`	HTTPS proxy	`http://proxy.corp:8080`
`OMLX_NO_PROXY`	Proxy bypass list	`localhost,127.0.0.1`
`OMLX_CA_BUNDLE`	Custom CA bundle PEM	`/etc/ssl/company-ca.pem`
`OMLX_MAX_TOKENS`	Default max tokens	`2048`
`OMLX_TEMPERATURE`	Default temperature	`0.7`
`OMLX_TRUST_REMOTE_CODE`	Allow model remote code	`false`
`OMLX_CONTINUOUS_BATCHING`	Enable continuous batching	`true`
`OMLX_SECRET_KEY`	Admin dashboard secret	`change-me-in-production`

Memory Configuration

Memory management is oMLX's primary tuning surface. Two limits interact:

--max-model-memory — how much memory can be occupied by loaded model weights across all models combined. When a new model is requested and this limit would be exceeded, the least recently used loaded model is evicted (LRU policy).
--max-process-memory — hard ceiling on the entire oMLX process. Defaults to total RAM minus 8 GB, leaving headroom for macOS and other apps.

Practical examples:

# M2 Pro with 16 GB — leave 4 GB for macOS, cap models at 10 GB
omlx serve --max-process-memory 12GB --max-model-memory 10GB

# M3 Max with 128 GB — allow large model weights, disable process cap
omlx serve --max-model-memory 96GB --max-process-memory disabled

# Percentage syntax (useful across different Mac configs)
omlx serve --max-model-memory 70% --max-process-memory 85%

Tip

Use GET /v1/models/status to see live memory usage per model. The admin dashboard at /dashboard shows this visually.

Directory Structure

~/.omlx/
├── models/
│   ├── llama-3b/          → model_id: "llama-3b"
│   │   ├── config.json
│   │   └── model.safetensors
│   ├── qwen-7b/           → model_id: "qwen-7b"
│   └── mistral-7b/        → model_id: "mistral-7b"
├── logs/
└── cache/                 ← SSD prefix cache (if configured)

The model ID used in API requests is the subdirectory name under --model-dir. A model in ~/.omlx/models/llama-3b/ is addressed as "model": "llama-3b" in all API calls.

Enabling API Key Authentication

omlx serve --api-key sk-my-secret-key

Clients must then include the header:

Authorization: Bearer sk-my-secret-key

Or via environment variable for all requests from a shell session:

export OMLX_API_KEY=sk-my-secret-key

4. Core Usage

4.1 Serving Models

Basic Start

omlx serve

Binds to 127.0.0.1:8000 with default settings. Models are loaded on first request.

Listen on All Interfaces (LAN Access)

omlx serve --host 0.0.0.0 --port 8080

Warning

Binding to 0.0.0.0 exposes oMLX to your local network. Always set --api-key when doing this, and ensure your firewall restricts access to trusted hosts. See the Admin Guide for network security configuration.

Custom Model Directory

omlx serve --model-dir /Volumes/ExternalSSD/llm-models

Enable SSD Prefix Cache

The prefix (KV) cache dramatically speeds up requests that share a common prefix — system prompts, few-shot examples, long documents. Results are stored on SSD between server restarts.

omlx serve \
  --paged-ssd-cache-dir /Volumes/FastSSD/omlx-cache \
  --paged-ssd-cache-max-size 200GB \
  --hot-cache-max-size 4GB

--hot-cache-max-size keeps the most recently used cache blocks in RAM for fastest access, spilling older blocks to the SSD directory.

To disable caching entirely:

omlx serve --no-cache

Increase Concurrency

omlx serve --max-concurrent-requests 16

Higher values improve throughput at the cost of increased memory pressure. For interactive use, 4–8 is typically fine; for batch processing, tune this up with monitoring.

Verbose Logging

omlx serve --log-level debug

Log levels in ascending verbosity: error, warning, info, debug, trace.

4.2 Making API Requests

oMLX implements the OpenAI API surface. Any client or library that works with OpenAI works with oMLX — just point the base URL at your local server.

Chat Completions (OpenAI-compatible)

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain paged KV caching in two sentences."}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Python (openai library):

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none"  # or your --api-key value
)

response = client.chat.completions.create(
    model="llama-3b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is MLX?"}
    ]
)
print(response.choices[0].message.content)

Streaming

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Write a haiku about inference."}],
    "stream": true
  }'

Tokens arrive as SSE events:

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Weights"},...}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" flow"},...}]}
...
data: [DONE]

Text Completions

curl http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "prompt": "The capital of France is",
    "max_tokens": 16
  }'

Embeddings

curl http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-embedding-model",
    "input": ["Hello world", "How are you?"]
  }'

Expected response includes a data array with one embedding vector per input string.

Reranking

curl http://127.0.0.1:8000/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-rerank-model",
    "query": "What is paged KV caching?",
    "documents": [
      "Paged KV caching splits the key-value cache into fixed-size blocks.",
      "The Eiffel Tower is in Paris.",
      "Prefix sharing reuses cached computation across requests."
    ]
  }'

Anthropic Messages API

oMLX also accepts Anthropic-formatted requests at /v1/messages:

curl http://127.0.0.1:8000/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "llama-3b",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Hello from Anthropic format!"}]
  }'

Count tokens without generating:

curl http://127.0.0.1:8000/v1/messages/count_tokens \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "How long is this?"}]
  }'

OpenAI Responses API (for Codex CLI)

curl http://127.0.0.1:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "input": "Write a Python function to reverse a string."
  }'

Retrieve a stored response:

curl http://127.0.0.1:8000/v1/responses/{response_id}

Delete a stored response:

curl -X DELETE http://127.0.0.1:8000/v1/responses/{response_id}

Tool Calling (Function Calling)

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'

Structured Output (JSON Schema)

Requires omlx[grammar] install. Forces output to conform to a JSON schema:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Extract: John Smith, age 34, engineer"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person",
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
            "occupation": {"type": "string"}
          },
          "required": ["name", "age", "occupation"]
        }
      }
    }
  }'

Vision-Language Models (VLMs)

Send images in the standard OpenAI multimodal format:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-vlm-model",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe this image."},
          {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/..."}}
        ]
      }
    ]
  }'

4.3 Audio (Speech-to-Text and Text-to-Speech)

Requires omlx[audio] install.

Speech-to-Text (Transcription)

curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=my-stt-model"

Text-to-Speech

curl http://127.0.0.1:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-tts-model",
    "input": "Hello, this is oMLX speaking.",
    "voice": "alloy"
  }' \
  --output speech.mp3

Audio Processing

curl http://127.0.0.1:8000/v1/audio/process \
  -F "file=@audio.wav" \
  -F "model=my-audio-model" \
  -F "task=translate"

4.4 Using the Admin Dashboard

Open http://127.0.0.1:8000/dashboard in your browser (or http://127.0.0.1:8000/login if authentication is configured).

The dashboard provides:

Model management — view all models in --model-dir, load/unload on demand, see live memory usage per model
Model downloads — download models directly from HuggingFace or ModelScope by entering the model ID
Chat interface — test any loaded model at /chat without writing any code
Benchmarks — run accuracy evaluations (MMLU, HellaSwag, GSM8K, HumanEval, etc.) against loaded models
MCP tools — browse configured MCP servers and available tools at /tools and /servers
Settings — adjust server configuration without restarting

Unloading a Model via API

Free a model's memory without restarting the server:

curl -X POST http://127.0.0.1:8000/v1/models/llama-3b/unload

The model stays discoverable in GET /v1/models but is evicted from memory. It reloads automatically on the next request.

Checking Model Memory Status

curl http://127.0.0.1:8000/v1/models/status

Returns each model's load state, memory footprint, and last-access time — useful for diagnosing memory pressure.

4.5 Using the macOS Menubar App

The menubar app wraps the oMLX server in a native macOS experience:

Start/Stop the server from the menu bar without a terminal
Quick links to the dashboard, chat, and logs
Status indicator — green when the server is running and accepting requests
Launch at login option in Preferences

If the menubar icon disappears after a macOS update, run omlx diagnose menubar for platform-specific remediation steps.

4.6 External Tool Integrations

omlx launch configures and starts external AI tools pre-wired to your local oMLX server.

# See what tools are available
omlx launch list

Codex CLI

OpenAI's Codex CLI for AI-assisted terminal coding:

omlx launch codex --model llama-3b

This writes ~/.codex/config.toml with the correct api_base and api_key pointing to your local oMLX server, then launches Codex.

OpenCode

omlx launch opencode --model qwen-7b

Configures OpenCode via environment variables and launches it.

OpenClaw

# Minimal tools profile
omlx launch openclaw --model mistral-7b --tools-profile minimal

# Full tools access
omlx launch openclaw --model mistral-7b --tools-profile full

Available profiles: minimal, coding, messaging, full.

Pi

omlx launch pi --model llama-3b

Point an existing tool at a running oMLX server without relaunching:

omlx launch codex \
  --model llama-3b \
  --host 192.168.1.100 \
  --port 8000 \
  --api-key sk-my-key

4.7 MCP (Model Context Protocol) Tools

MCP lets the LLM call external tools — web search, file access, custom APIs — during inference. Configure tools in a JSON or YAML file and pass it to --mcp-config.

Example MCP Config (`~/.omlx/mcp.json`)

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/Documents"]
    },
    "brave-search": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-brave-search"],
      "env": {
        "BRAVE_API_KEY": "your-brave-api-key"
      }
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_yourtoken"
      }
    }
  }
}

Start the server with MCP enabled:

omlx serve --mcp-config ~/.omlx/mcp.json

Now include MCP tools in chat requests:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Search for the latest MLX release notes."}],
    "tools": "mcp"
  }'

Browse configured servers and tools in the admin dashboard at /servers and /tools, or via the API:

# List MCP servers
curl http://127.0.0.1:8000/v1/mcp/servers

# List available tools
curl http://127.0.0.1:8000/v1/mcp/tools

4.8 Running Accuracy Benchmarks

Run standard benchmarks against any loaded model through the admin dashboard (/dashboard → Benchmarks), or via the API. Available benchmarks include MMLU, HellaSwag, GSM8K, and HumanEval.

Tip

Benchmarks run locally against your loaded model weights. Results reflect the actual model you are running, including any quantization effects from oQ custom quantization.

5. Common Tasks

Download a Model from HuggingFace

Open the admin dashboard: http://127.0.0.1:8000/dashboard
Navigate to the Downloads section
Enter the HuggingFace model ID (e.g., mlx-community/Llama-3.2-3B-Instruct-4bit)
Click Download

To use a custom HuggingFace mirror:

omlx serve --hf-endpoint https://hf-mirror.com
# or
export OMLX_HF_ENDPOINT=https://hf-mirror.com
omlx serve

For ModelScope:

omlx serve --ms-endpoint https://modelscope.cn

Add a Model from Local Files

Copy or symlink model files into a subdirectory under --model-dir:

mkdir -p ~/.omlx/models/my-custom-model
cp /path/to/model/files/* ~/.omlx/models/my-custom-model/

The model becomes available as "model": "my-custom-model" immediately — no restart required.

Set Up a Persistent Server (launchd)

Create ~/Library/LaunchAgents/ai.omlx.server.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>ai.omlx.server</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/omlx</string>
        <string>serve</string>
        <string>--max-model-memory</string>
        <string>24GB</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/omlx.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/omlx.err</string>
</dict>
</plist>

Load it:

launchctl load ~/Library/LaunchAgents/ai.omlx.server.plist

Migrate an Existing OpenAI Integration

In Python with the openai library, change only the base_url:

# Before
client = OpenAI(api_key="sk-...")

# After — everything else stays the same
client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none"
)

In environment-variable-based tools:

export OPENAI_API_BASE=http://127.0.0.1:8000/v1
export OPENAI_API_KEY=none

Use oMLX from Another Machine on Your LAN

# On the Mac running oMLX
omlx serve --host 0.0.0.0 --api-key sk-lan-key

# From another machine
curl http://192.168.1.100:8000/v1/chat/completions \
  -H "Authorization: Bearer sk-lan-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3b", "messages": [{"role": "user", "content": "Hi"}]}'

Use a Corporate HTTP Proxy

omlx serve \
  --http-proxy http://proxy.corp.example.com:8080 \
  --https-proxy http://proxy.corp.example.com:8080 \
  --no-proxy localhost,127.0.0.1,192.168.0.0/16

For a custom CA bundle (self-signed corporate certs):

omlx serve --ca-bundle /path/to/corp-ca-bundle.pem

Enable Continuous Batching

Continuous batching is enabled by default. To verify or force it:

export OMLX_CONTINUOUS_BATCHING=true
omlx serve

This uses a vLLM-style scheduler that interleaves prefill and decode phases across concurrent requests, improving throughput significantly when multiple clients are active simultaneously.

6. Troubleshooting

Error: `RuntimeError: no Stream(gpu, 0) in current thread`

Cause: A model inference call is being made from a thread that does not own an MLX Metal stream. This is a thread-safety violation in the MLX runtime (GitHub issue #904).

Solution: Ensure all model inference calls happen on the main thread or a thread where the MLX stream was initialized. If you are calling oMLX via a custom Python integration that spawns threads, initialize the MLX context in each thread before making requests, or use a thread pool that was started after MLX initialization.

Prevention: Do not share MLX model objects across threads. Use the HTTP API from background threads rather than calling MLX directly.

Error: `SpecPrefill system_end calc failed`

Cause: Speculative prefill encountered an edge case in the system prompt boundary calculation (GitHub issue #916). Typically triggered by unusual system prompt formatting.

Solution: This is a known bug being tracked upstream. As a workaround, avoid empty system prompts and ensure your system message is a non-empty string.

Error: `parameters not matching architecture` (DFlash models)

Cause: A DFlash model variant has a parameter layout that does not match the expected architecture config (GitHub issue #886). The model's config.json references an architecture that the loaded weights do not satisfy.

Solution:

Verify the model files are complete — re-download if the download was interrupted.
Check whether the model requires OMLX_TRUST_REMOTE_CODE=true and, if so, weigh the security implications before enabling it.
If using dflash with a model that exhibits infinite loop behavior, see issues #911/#934/#935 — this combination has known instability in 0.3.7.

Error: `ToolCallStreamFilter crashes on streaming tool calls`

Cause: A streaming response with tool calls hit a parsing edge case in the stream filter (GitHub issue #882).

Solution: Set "stream": false in the request as an immediate workaround. The crash is triggered only during streaming; non-streaming tool calls work correctly.

Prevention: Pin to a patched release once available. Track issue #882 on GitHub.

Slow Token Generation After Upgrading to 0.3.7

Cause: Confirmed regression in 0.3.7 affecting token generation speed on some model/hardware combos (GitHub issue #936). Also observed specifically for Qwen3-TTS (issue #937).

Solution:

Downgrade to 0.3.6 if generation speed is critical: pip install omlx==0.3.6
For Qwen3-TTS specifically, check issue #937 for a workaround config
Monitor the 0.3.8 release for the fix

Error: `translategemma-4b-it-4bit error` / Model Fails to Load

Cause: Certain quantized model variants (e.g., translategemma-4b-it-4bit, Qwen3-VL-Reranker) use architectures not yet supported by the installed oMLX version (GitHub issues #879, #877).

Solution:

Check GET /v1/models/status for a more detailed error message
Verify you are running the latest oMLX version: pip install --upgrade omlx
If the architecture is listed as unsupported in release notes, watch for a future release that adds support

Error on M4 Max Regardless of Model

Cause: An M4 Max-specific issue affecting model loading across all models (GitHub issue #899). Likely a memory layout or Metal API compatibility issue.

Solution: Check the GitHub issue #899 for the latest status and any platform-specific workarounds. As of 0.3.7, this may require a patch release.

Double Request Counting in Logs

Cause: A bug causing each request to be counted twice in usage/logging metrics (GitHub issue #914). Does not affect response correctness — only reporting.

Solution: This is cosmetic for now. Track issue #914 for a fix. Do not use per-request log counts for billing or throttling decisions until resolved.

oMLX Process Using Too Much Memory

Caution

Leaving --max-process-memory unset on systems with less than 16 GB of RAM can allow oMLX to consume enough memory to trigger macOS memory pressure, causing system slowdowns.

Solution:

# Explicitly cap total process memory
omlx serve --max-process-memory 12GB --max-model-memory 8GB

# Check current memory usage
curl http://127.0.0.1:8000/v1/models/status

If memory is not releasing after unloading a model, restart the server — this is a known MLX allocator behavior where the Metal allocator holds freed blocks for reuse.

oQ Quantization Breaks a Model

Cause: oQ custom quantization can produce incorrect results on certain model architectures (confirmed for Qwen3.6 27b in GitHub issue #913).

Solution: Use a pre-quantized model from HuggingFace instead of applying oQ locally. Avoid oQ on Qwen3 models until a fix is confirmed.

Menubar Icon Not Visible

omlx diagnose menubar

This command outputs macOS-version-specific steps to restore visibility (relevant for Tahoe beta and Control Center changes).

7. FAQ

Can oMLX run on an Intel Mac? No. oMLX requires Apple Silicon (M1 or later) for MLX hardware acceleration. There is no CPU-only fallback.

Does it work on macOS 14 Sonoma? No. macOS 15 Sequoia is the minimum required version. Sequoia introduced Metal API features that MLX depends on.

Can I run multiple models simultaneously? Yes. oMLX loads models on demand and keeps them in memory up to --max-model-memory. When the limit is reached, the least recently used model is evicted. All loaded models are accessible at all times — just reference them by name in each request.

How do I know which model is being used if I don't specify one? Always specify the "model" field in your request. oMLX does not have a global default model — omitting it will return a validation error.

Can I use oMLX as a drop-in replacement for the OpenAI API? For chat completions, completions, and embeddings: yes, for most use cases. Some advanced OpenAI features (fine-tuning, assistants, batch API) are not implemented. The /v1/chat/completions and /v1/completions surfaces are the most complete.

What is the difference between --paged-ssd-cache-dir and --hot-cache-max-size? The SSD cache stores computed KV blocks on disk, persisting between server restarts. The hot cache (--hot-cache-max-size) keeps a portion of those blocks in RAM for fastest access. Both work together: RAM-hot blocks are served immediately; SSD blocks require a disk read but are still faster than recomputing from scratch. Set --hot-cache-max-size 0 (the default) to use SSD cache only.

Does oMLX support streaming responses? Yes. Add "stream": true to any chat completions or completions request. Responses arrive as server-sent events (SSE). Note: streaming combined with tool calls has a known crash in 0.3.7 (see Troubleshooting).

What quantization formats are supported? oMLX supports standard MLX quantized formats (4-bit, 8-bit) as well as oQ, the project's custom quantization scheme. Use pre-quantized models from the mlx-community HuggingFace organization for the most reliable results.

Is my data sent anywhere? No. All inference runs locally on your Mac. The only network calls oMLX makes are model downloads (to HuggingFace or ModelScope) and any external calls made by MCP tools you configure. No telemetry or usage data is sent to oMLX's author.

How do I update oMLX?

pip install --upgrade omlx
# or
brew upgrade jundot/tap/omlx

Where are the logs? By default, logs go to stdout. Set OMLX_LOG_DIR to write to files:

export OMLX_LOG_DIR=~/.omlx/logs
export OMLX_LOG_RETENTION_DAYS=14
omlx serve

Can I use this with LangChain or LlamaIndex? Yes. Both support OpenAI-compatible endpoints. Set the openai_api_base (LangChain) or equivalent configuration to http://127.0.0.1:8000/v1.

What does omlx launch list show? It lists the external AI tools that oMLX can configure and launch: codex, opencode, openclaw, and pi. Each tool must be installed separately; oMLX handles only the configuration wiring.

1. Quick Start

Prerequisites

Install and Start

2. Installation

Method 1: pip (Recommended)

Method 2: Homebrew

Method 3: macOS Menubar App (DMG)

Method 4: From Source

Verifying the Install

3. Configuration

Key Environment Variables

Memory Configuration

Directory Structure

Enabling API Key Authentication

4. Core Usage

4.1 Serving Models

Basic Start

Listen on All Interfaces (LAN Access)

Custom Model Directory

Enable SSD Prefix Cache

Increase Concurrency

Verbose Logging

4.2 Making API Requests

Chat Completions (OpenAI-compatible)

Streaming

Text Completions

Embeddings

Reranking

Anthropic Messages API

OpenAI Responses API (for Codex CLI)

Tool Calling (Function Calling)

Structured Output (JSON Schema)

Vision-Language Models (VLMs)

4.3 Audio (Speech-to-Text and Text-to-Speech)

Speech-to-Text (Transcription)

Text-to-Speech

Audio Processing

4.4 Using the Admin Dashboard

Unloading a Model via API

Checking Model Memory Status

4.5 Using the macOS Menubar App

4.6 External Tool Integrations

Codex CLI

OpenCode

OpenClaw

Pi

4.7 MCP (Model Context Protocol) Tools

Example MCP Config (~/.omlx/mcp.json)

4.8 Running Accuracy Benchmarks

5. Common Tasks

Download a Model from HuggingFace

Add a Model from Local Files

Set Up a Persistent Server (launchd)

Migrate an Existing OpenAI Integration

Use oMLX from Another Machine on Your LAN

Use a Corporate HTTP Proxy

Enable Continuous Batching

6. Troubleshooting

Error: RuntimeError: no Stream(gpu, 0) in current thread

Error: SpecPrefill system_end calc failed

Error: parameters not matching architecture (DFlash models)

Error: ToolCallStreamFilter crashes on streaming tool calls

Slow Token Generation After Upgrading to 0.3.7

Error: translategemma-4b-it-4bit error / Model Fails to Load

Error on M4 Max Regardless of Model

Double Request Counting in Logs

oMLX Process Using Too Much Memory

oQ Quantization Breaks a Model

Menubar Icon Not Visible

7. FAQ

Example MCP Config (`~/.omlx/mcp.json`)

Error: `RuntimeError: no Stream(gpu, 0) in current thread`

Error: `SpecPrefill system_end calc failed`

Error: `parameters not matching architecture` (DFlash models)

Error: `ToolCallStreamFilter crashes on streaming tool calls`

Error: `translategemma-4b-it-4bit error` / Model Fails to Load