# oMLX — User Guide

oMLX is a production-ready, OpenAI-compatible LLM inference server built for Apple Silicon Macs. It runs entirely on-device using Apple's MLX framework — no cloud API costs, no GPU required, no data leaving your machine. Drop it in as a local replacement for OpenAI's API and your existing tools keep working.

> [!NOTE]
> This guide covers everyday use: installing, configuring, serving models, and calling the API. Architecture internals, deployment configuration, and security hardening are in the **Admin Guide** ([Admin Guide](/jundot/omlx/admin-guide/)).

---

## 1. Quick Start

Get a server running and answer your first request in under five minutes.

### Prerequisites

- Apple Silicon Mac (M1 or later)
- macOS 15.0 Sequoia or later
- Python 3.10 or later

### Install and Start

```bash
pip install omlx
```

Start the server with a model:

```bash
omlx serve --model-dir ~/.omlx/models
```

> [!TIP]
> On first launch, oMLX creates `~/.omlx/` automatically. Download a model from the admin dashboard at `http://127.0.0.1:8000/dashboard` before making your first inference request.

Once the server is running, verify it is healthy:

```bash
curl http://127.0.0.1:8000/health
```

Expected response:

```json
{"status": "ok"}
```

List loaded models:

```bash
curl http://127.0.0.1:8000/v1/models
```

Expected response (once a model is loaded):

```json
{
  "object": "list",
  "data": [
    {
      "id": "llama-3b",
      "object": "model",
      "created": 1746624000,
      "owned_by": "local"
    }
  ]
}
```

Make your first chat request:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}]
  }'
```

---

## 2. Installation

### Method 1: pip (Recommended)

**Core install** (text generation, embeddings, reranking):

```bash
pip install omlx
```

**With audio support** (speech-to-text and text-to-speech):

```bash
pip install omlx[audio]
```

**With structured output** (JSON schema enforcement via xgrammar):

```bash
pip install omlx[grammar]
```

**With ModelScope** (model downloads from ModelScope Hub):

```bash
pip install omlx[modelscope]
```

**Full install** (everything):

```bash
pip install "omlx[audio,grammar,modelscope]"
```

> [!IMPORTANT]
> oMLX requires macOS 15.0 (Sequoia) or later and an Apple Silicon chip (M1, M2, M3, or M4 series). It will not run on Intel Macs or any other platform.

### Method 2: Homebrew

```bash
brew tap jundot/tap
brew install jundot/tap/omlx
```

With structured output (xgrammar) support:

```bash
brew install jundot/tap/omlx --with-grammar
```

### Method 3: macOS Menubar App (DMG)

Download the DMG installer from the releases page, open it, drag oMLX to your Applications folder, and launch it. The menubar icon gives you one-click start/stop and quick access to the dashboard.

If the menubar icon does not appear (a known issue with macOS Tahoe and Control Center), run:

```bash
omlx diagnose menubar
```

This prints the steps to restore icon visibility.

### Method 4: From Source

```bash
git clone https://github.com/jundot/omlx
cd omlx
pip install -e ".[dev]"
```

### Verifying the Install

```bash
omlx --version
# omlx 0.3.7

omlx serve --help
```

> [!WARNING]
> If `pip install omlx` succeeds but `omlx` is not found on your PATH, your Python `bin` directory is not in your shell's PATH. Run `python -m omlx serve` as a workaround, or add the bin path: `export PATH="$(python -m site --user-base)/bin:$PATH"`.

---

## 3. Configuration

oMLX is configured through a combination of CLI flags and environment variables. CLI flags take precedence over environment variables.

### Key Environment Variables

Set these in your shell profile (`~/.zshrc` or `~/.bash_profile`) to avoid repeating flags on every launch.

| Variable | Purpose | Example |
|---|---|---|
| `OMLX_HOST` | Bind host | `0.0.0.0` |
| `OMLX_PORT` | Bind port | `8000` |
| `OMLX_MODEL_DIR` | Model storage directory | `/Volumes/FastDrive/models` |
| `OMLX_MAX_MODEL_MEMORY` | Memory cap for loaded models | `32GB` |
| `OMLX_MAX_PROCESS_MEMORY` | Total process memory cap | `80%` |
| `OMLX_MAX_CONCURRENT_REQUESTS` | Request concurrency limit | `4` |
| `OMLX_API_KEY` | Authentication key | `sk-my-local-key` |
| `OMLX_LOG_LEVEL` | Logging verbosity | `debug` |
| `OMLX_LOG_DIR` | Log file directory | `~/.omlx/logs` |
| `OMLX_LOG_RETENTION_DAYS` | Days to keep log files | `7` |
| `OMLX_CACHE_ENABLED` | Enable/disable KV cache | `true` |
| `OMLX_SSD_CACHE_DIR` | SSD prefix cache directory | `/Volumes/SSD/omlx-cache` |
| `OMLX_SSD_CACHE_MAX_SIZE` | Max SSD cache size | `200GB` |
| `OMLX_HOT_CACHE_ONLY` | Use RAM-only hot cache | `true` |
| `OMLX_INITIAL_CACHE_BLOCKS` | Pre-allocated cache blocks | `512` |
| `OMLX_MCP_CONFIG` | MCP tools config file path | `~/.omlx/mcp.json` |
| `OMLX_HF_ENDPOINT` | Custom HuggingFace Hub URL | `https://hf-mirror.com` |
| `OMLX_MS_ENDPOINT` | Custom ModelScope Hub URL | `https://modelscope.cn` |
| `OMLX_HTTP_PROXY` | HTTP proxy | `http://proxy.corp:8080` |
| `OMLX_HTTPS_PROXY` | HTTPS proxy | `http://proxy.corp:8080` |
| `OMLX_NO_PROXY` | Proxy bypass list | `localhost,127.0.0.1` |
| `OMLX_CA_BUNDLE` | Custom CA bundle PEM | `/etc/ssl/company-ca.pem` |
| `OMLX_MAX_TOKENS` | Default max tokens | `2048` |
| `OMLX_TEMPERATURE` | Default temperature | `0.7` |
| `OMLX_TRUST_REMOTE_CODE` | Allow model remote code | `false` |
| `OMLX_CONTINUOUS_BATCHING` | Enable continuous batching | `true` |
| `OMLX_SECRET_KEY` | Admin dashboard secret | `change-me-in-production` |

### Memory Configuration

Memory management is oMLX's primary tuning surface. Two limits interact:

- **`--max-model-memory`** — how much memory can be occupied by loaded model weights across all models combined. When a new model is requested and this limit would be exceeded, the least recently used loaded model is evicted (LRU policy).
- **`--max-process-memory`** — hard ceiling on the entire oMLX process. Defaults to total RAM minus 8 GB, leaving headroom for macOS and other apps.

Practical examples:

```bash
# M2 Pro with 16 GB — leave 4 GB for macOS, cap models at 10 GB
omlx serve --max-process-memory 12GB --max-model-memory 10GB

# M3 Max with 128 GB — allow large model weights, disable process cap
omlx serve --max-model-memory 96GB --max-process-memory disabled

# Percentage syntax (useful across different Mac configs)
omlx serve --max-model-memory 70% --max-process-memory 85%
```

> [!TIP]
> Use `GET /v1/models/status` to see live memory usage per model. The admin dashboard at `/dashboard` shows this visually.

### Directory Structure

```
~/.omlx/
├── models/
│   ├── llama-3b/          → model_id: "llama-3b"
│   │   ├── config.json
│   │   └── model.safetensors
│   ├── qwen-7b/           → model_id: "qwen-7b"
│   └── mistral-7b/        → model_id: "mistral-7b"
├── logs/
└── cache/                 ← SSD prefix cache (if configured)
```

The model ID used in API requests is the subdirectory name under `--model-dir`. A model in `~/.omlx/models/llama-3b/` is addressed as `"model": "llama-3b"` in all API calls.

### Enabling API Key Authentication

```bash
omlx serve --api-key sk-my-secret-key
```

Clients must then include the header:

```
Authorization: Bearer sk-my-secret-key
```

Or via environment variable for all requests from a shell session:

```bash
export OMLX_API_KEY=sk-my-secret-key
```

---

## 4. Core Usage

### 4.1 Serving Models

#### Basic Start

```bash
omlx serve
```

Binds to `127.0.0.1:8000` with default settings. Models are loaded on first request.

#### Listen on All Interfaces (LAN Access)

```bash
omlx serve --host 0.0.0.0 --port 8080
```

> [!WARNING]
> Binding to `0.0.0.0` exposes oMLX to your local network. Always set `--api-key` when doing this, and ensure your firewall restricts access to trusted hosts. See the Admin Guide for network security configuration.

#### Custom Model Directory

```bash
omlx serve --model-dir /Volumes/ExternalSSD/llm-models
```

#### Enable SSD Prefix Cache

The prefix (KV) cache dramatically speeds up requests that share a common prefix — system prompts, few-shot examples, long documents. Results are stored on SSD between server restarts.

```bash
omlx serve \
  --paged-ssd-cache-dir /Volumes/FastSSD/omlx-cache \
  --paged-ssd-cache-max-size 200GB \
  --hot-cache-max-size 4GB
```

`--hot-cache-max-size` keeps the most recently used cache blocks in RAM for fastest access, spilling older blocks to the SSD directory.

To disable caching entirely:

```bash
omlx serve --no-cache
```

#### Increase Concurrency

```bash
omlx serve --max-concurrent-requests 16
```

Higher values improve throughput at the cost of increased memory pressure. For interactive use, 4–8 is typically fine; for batch processing, tune this up with monitoring.

#### Verbose Logging

```bash
omlx serve --log-level debug
```

Log levels in ascending verbosity: `error`, `warning`, `info`, `debug`, `trace`.

---

### 4.2 Making API Requests

oMLX implements the OpenAI API surface. Any client or library that works with OpenAI works with oMLX — just point the base URL at your local server.

#### Chat Completions (OpenAI-compatible)

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain paged KV caching in two sentences."}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'
```

**Python (openai library):**

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none"  # or your --api-key value
)

response = client.chat.completions.create(
    model="llama-3b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is MLX?"}
    ]
)
print(response.choices[0].message.content)
```

#### Streaming

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Write a haiku about inference."}],
    "stream": true
  }'
```

Tokens arrive as SSE events:

```
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Weights"},...}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" flow"},...}]}
...
data: [DONE]
```

#### Text Completions

```bash
curl http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "prompt": "The capital of France is",
    "max_tokens": 16
  }'
```

#### Embeddings

```bash
curl http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-embedding-model",
    "input": ["Hello world", "How are you?"]
  }'
```

Expected response includes a `data` array with one embedding vector per input string.

#### Reranking

```bash
curl http://127.0.0.1:8000/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-rerank-model",
    "query": "What is paged KV caching?",
    "documents": [
      "Paged KV caching splits the key-value cache into fixed-size blocks.",
      "The Eiffel Tower is in Paris.",
      "Prefix sharing reuses cached computation across requests."
    ]
  }'
```

#### Anthropic Messages API

oMLX also accepts Anthropic-formatted requests at `/v1/messages`:

```bash
curl http://127.0.0.1:8000/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "llama-3b",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Hello from Anthropic format!"}]
  }'
```

Count tokens without generating:

```bash
curl http://127.0.0.1:8000/v1/messages/count_tokens \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "How long is this?"}]
  }'
```

#### OpenAI Responses API (for Codex CLI)

```bash
curl http://127.0.0.1:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "input": "Write a Python function to reverse a string."
  }'
```

Retrieve a stored response:

```bash
curl http://127.0.0.1:8000/v1/responses/{response_id}
```

Delete a stored response:

```bash
curl -X DELETE http://127.0.0.1:8000/v1/responses/{response_id}
```

#### Tool Calling (Function Calling)

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'
```

#### Structured Output (JSON Schema)

Requires `omlx[grammar]` install. Forces output to conform to a JSON schema:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Extract: John Smith, age 34, engineer"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person",
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
            "occupation": {"type": "string"}
          },
          "required": ["name", "age", "occupation"]
        }
      }
    }
  }'
```

#### Vision-Language Models (VLMs)

Send images in the standard OpenAI multimodal format:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-vlm-model",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe this image."},
          {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/..."}}
        ]
      }
    ]
  }'
```

---

### 4.3 Audio (Speech-to-Text and Text-to-Speech)

Requires `omlx[audio]` install.

#### Speech-to-Text (Transcription)

```bash
curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=my-stt-model"
```

#### Text-to-Speech

```bash
curl http://127.0.0.1:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-tts-model",
    "input": "Hello, this is oMLX speaking.",
    "voice": "alloy"
  }' \
  --output speech.mp3
```

#### Audio Processing

```bash
curl http://127.0.0.1:8000/v1/audio/process \
  -F "file=@audio.wav" \
  -F "model=my-audio-model" \
  -F "task=translate"
```

---

### 4.4 Using the Admin Dashboard

Open `http://127.0.0.1:8000/dashboard` in your browser (or `http://127.0.0.1:8000/login` if authentication is configured).

The dashboard provides:

- **Model management** — view all models in `--model-dir`, load/unload on demand, see live memory usage per model
- **Model downloads** — download models directly from HuggingFace or ModelScope by entering the model ID
- **Chat interface** — test any loaded model at `/chat` without writing any code
- **Benchmarks** — run accuracy evaluations (MMLU, HellaSwag, GSM8K, HumanEval, etc.) against loaded models
- **MCP tools** — browse configured MCP servers and available tools at `/tools` and `/servers`
- **Settings** — adjust server configuration without restarting

#### Unloading a Model via API

Free a model's memory without restarting the server:

```bash
curl -X POST http://127.0.0.1:8000/v1/models/llama-3b/unload
```

The model stays discoverable in `GET /v1/models` but is evicted from memory. It reloads automatically on the next request.

#### Checking Model Memory Status

```bash
curl http://127.0.0.1:8000/v1/models/status
```

Returns each model's load state, memory footprint, and last-access time — useful for diagnosing memory pressure.

---

### 4.5 Using the macOS Menubar App

The menubar app wraps the oMLX server in a native macOS experience:

- **Start/Stop** the server from the menu bar without a terminal
- **Quick links** to the dashboard, chat, and logs
- **Status indicator** — green when the server is running and accepting requests
- **Launch at login** option in Preferences

If the menubar icon disappears after a macOS update, run `omlx diagnose menubar` for platform-specific remediation steps.

---

### 4.6 External Tool Integrations

`omlx launch` configures and starts external AI tools pre-wired to your local oMLX server.

```bash
# See what tools are available
omlx launch list
```

#### Codex CLI

OpenAI's Codex CLI for AI-assisted terminal coding:

```bash
omlx launch codex --model llama-3b
```

This writes `~/.codex/config.toml` with the correct `api_base` and `api_key` pointing to your local oMLX server, then launches Codex.

#### OpenCode

```bash
omlx launch opencode --model qwen-7b
```

Configures OpenCode via environment variables and launches it.

#### OpenClaw

```bash
# Minimal tools profile
omlx launch openclaw --model mistral-7b --tools-profile minimal

# Full tools access
omlx launch openclaw --model mistral-7b --tools-profile full
```

Available profiles: `minimal`, `coding`, `messaging`, `full`.

#### Pi

```bash
omlx launch pi --model llama-3b
```

Point an existing tool at a running oMLX server without relaunching:

```bash
omlx launch codex \
  --model llama-3b \
  --host 192.168.1.100 \
  --port 8000 \
  --api-key sk-my-key
```

---

### 4.7 MCP (Model Context Protocol) Tools

MCP lets the LLM call external tools — web search, file access, custom APIs — during inference. Configure tools in a JSON or YAML file and pass it to `--mcp-config`.

#### Example MCP Config (`~/.omlx/mcp.json`)

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/Documents"]
    },
    "brave-search": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-brave-search"],
      "env": {
        "BRAVE_API_KEY": "your-brave-api-key"
      }
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_yourtoken"
      }
    }
  }
}
```

Start the server with MCP enabled:

```bash
omlx serve --mcp-config ~/.omlx/mcp.json
```

Now include MCP tools in chat requests:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3b",
    "messages": [{"role": "user", "content": "Search for the latest MLX release notes."}],
    "tools": "mcp"
  }'
```

Browse configured servers and tools in the admin dashboard at `/servers` and `/tools`, or via the API:

```bash
# List MCP servers
curl http://127.0.0.1:8000/v1/mcp/servers

# List available tools
curl http://127.0.0.1:8000/v1/mcp/tools
```

---

### 4.8 Running Accuracy Benchmarks

Run standard benchmarks against any loaded model through the admin dashboard (`/dashboard` → Benchmarks), or via the API. Available benchmarks include MMLU, HellaSwag, GSM8K, and HumanEval.

> [!TIP]
> Benchmarks run locally against your loaded model weights. Results reflect the actual model you are running, including any quantization effects from oQ custom quantization.

---

## 5. Common Tasks

### Download a Model from HuggingFace

1. Open the admin dashboard: `http://127.0.0.1:8000/dashboard`
2. Navigate to the Downloads section
3. Enter the HuggingFace model ID (e.g., `mlx-community/Llama-3.2-3B-Instruct-4bit`)
4. Click Download

To use a custom HuggingFace mirror:

```bash
omlx serve --hf-endpoint https://hf-mirror.com
# or
export OMLX_HF_ENDPOINT=https://hf-mirror.com
omlx serve
```

For ModelScope:

```bash
omlx serve --ms-endpoint https://modelscope.cn
```

### Add a Model from Local Files

Copy or symlink model files into a subdirectory under `--model-dir`:

```bash
mkdir -p ~/.omlx/models/my-custom-model
cp /path/to/model/files/* ~/.omlx/models/my-custom-model/
```

The model becomes available as `"model": "my-custom-model"` immediately — no restart required.

### Set Up a Persistent Server (launchd)

Create `~/Library/LaunchAgents/ai.omlx.server.plist`:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>ai.omlx.server</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/omlx</string>
        <string>serve</string>
        <string>--max-model-memory</string>
        <string>24GB</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/omlx.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/omlx.err</string>
</dict>
</plist>
```

Load it:

```bash
launchctl load ~/Library/LaunchAgents/ai.omlx.server.plist
```

### Migrate an Existing OpenAI Integration

In Python with the openai library, change only the `base_url`:

```python
# Before
client = OpenAI(api_key="sk-...")

# After — everything else stays the same
client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none"
)
```

In environment-variable-based tools:

```bash
export OPENAI_API_BASE=http://127.0.0.1:8000/v1
export OPENAI_API_KEY=none
```

### Use oMLX from Another Machine on Your LAN

```bash
# On the Mac running oMLX
omlx serve --host 0.0.0.0 --api-key sk-lan-key

# From another machine
curl http://192.168.1.100:8000/v1/chat/completions \
  -H "Authorization: Bearer sk-lan-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3b", "messages": [{"role": "user", "content": "Hi"}]}'
```

### Use a Corporate HTTP Proxy

```bash
omlx serve \
  --http-proxy http://proxy.corp.example.com:8080 \
  --https-proxy http://proxy.corp.example.com:8080 \
  --no-proxy localhost,127.0.0.1,192.168.0.0/16
```

For a custom CA bundle (self-signed corporate certs):

```bash
omlx serve --ca-bundle /path/to/corp-ca-bundle.pem
```

### Enable Continuous Batching

Continuous batching is enabled by default. To verify or force it:

```bash
export OMLX_CONTINUOUS_BATCHING=true
omlx serve
```

This uses a vLLM-style scheduler that interleaves prefill and decode phases across concurrent requests, improving throughput significantly when multiple clients are active simultaneously.

---

## 6. Troubleshooting

### Error: `RuntimeError: no Stream(gpu, 0) in current thread`

**Cause:** A model inference call is being made from a thread that does not own an MLX Metal stream. This is a thread-safety violation in the MLX runtime (GitHub issue #904).

**Solution:** Ensure all model inference calls happen on the main thread or a thread where the MLX stream was initialized. If you are calling oMLX via a custom Python integration that spawns threads, initialize the MLX context in each thread before making requests, or use a thread pool that was started after MLX initialization.

**Prevention:** Do not share MLX model objects across threads. Use the HTTP API from background threads rather than calling MLX directly.

---

### Error: `SpecPrefill system_end calc failed`

**Cause:** Speculative prefill encountered an edge case in the system prompt boundary calculation (GitHub issue #916). Typically triggered by unusual system prompt formatting.

**Solution:** This is a known bug being tracked upstream. As a workaround, avoid empty system prompts and ensure your system message is a non-empty string.

---

### Error: `parameters not matching architecture` (DFlash models)

**Cause:** A DFlash model variant has a parameter layout that does not match the expected architecture config (GitHub issue #886). The model's `config.json` references an architecture that the loaded weights do not satisfy.

**Solution:**
1. Verify the model files are complete — re-download if the download was interrupted.
2. Check whether the model requires `OMLX_TRUST_REMOTE_CODE=true` and, if so, weigh the security implications before enabling it.
3. If using dflash with a model that exhibits infinite loop behavior, see issues #911/#934/#935 — this combination has known instability in 0.3.7.

---

### Error: `ToolCallStreamFilter crashes on streaming tool calls`

**Cause:** A streaming response with tool calls hit a parsing edge case in the stream filter (GitHub issue #882).

**Solution:** Set `"stream": false` in the request as an immediate workaround. The crash is triggered only during streaming; non-streaming tool calls work correctly.

**Prevention:** Pin to a patched release once available. Track issue #882 on GitHub.

---

### Slow Token Generation After Upgrading to 0.3.7

**Cause:** Confirmed regression in 0.3.7 affecting token generation speed on some model/hardware combos (GitHub issue #936). Also observed specifically for Qwen3-TTS (issue #937).

**Solution:**
- Downgrade to 0.3.6 if generation speed is critical: `pip install omlx==0.3.6`
- For Qwen3-TTS specifically, check issue #937 for a workaround config
- Monitor the 0.3.8 release for the fix

---

### Error: `translategemma-4b-it-4bit error` / Model Fails to Load

**Cause:** Certain quantized model variants (e.g., `translategemma-4b-it-4bit`, Qwen3-VL-Reranker) use architectures not yet supported by the installed oMLX version (GitHub issues #879, #877).

**Solution:**
1. Check `GET /v1/models/status` for a more detailed error message
2. Verify you are running the latest oMLX version: `pip install --upgrade omlx`
3. If the architecture is listed as unsupported in release notes, watch for a future release that adds support

---

### Error on M4 Max Regardless of Model

**Cause:** An M4 Max-specific issue affecting model loading across all models (GitHub issue #899). Likely a memory layout or Metal API compatibility issue.

**Solution:** Check the GitHub issue #899 for the latest status and any platform-specific workarounds. As of 0.3.7, this may require a patch release.

---

### Double Request Counting in Logs

**Cause:** A bug causing each request to be counted twice in usage/logging metrics (GitHub issue #914). Does not affect response correctness — only reporting.

**Solution:** This is cosmetic for now. Track issue #914 for a fix. Do not use per-request log counts for billing or throttling decisions until resolved.

---

### oMLX Process Using Too Much Memory

> [!CAUTION]
> Leaving `--max-process-memory` unset on systems with less than 16 GB of RAM can allow oMLX to consume enough memory to trigger macOS memory pressure, causing system slowdowns.

**Solution:**

```bash
# Explicitly cap total process memory
omlx serve --max-process-memory 12GB --max-model-memory 8GB

# Check current memory usage
curl http://127.0.0.1:8000/v1/models/status
```

If memory is not releasing after unloading a model, restart the server — this is a known MLX allocator behavior where the Metal allocator holds freed blocks for reuse.

---

### oQ Quantization Breaks a Model

**Cause:** oQ custom quantization can produce incorrect results on certain model architectures (confirmed for Qwen3.6 27b in GitHub issue #913).

**Solution:** Use a pre-quantized model from HuggingFace instead of applying oQ locally. Avoid oQ on Qwen3 models until a fix is confirmed.

---

### Menubar Icon Not Visible

```bash
omlx diagnose menubar
```

This command outputs macOS-version-specific steps to restore visibility (relevant for Tahoe beta and Control Center changes).

---

## 7. FAQ

**Can oMLX run on an Intel Mac?**
No. oMLX requires Apple Silicon (M1 or later) for MLX hardware acceleration. There is no CPU-only fallback.

**Does it work on macOS 14 Sonoma?**
No. macOS 15 Sequoia is the minimum required version. Sequoia introduced Metal API features that MLX depends on.

**Can I run multiple models simultaneously?**
Yes. oMLX loads models on demand and keeps them in memory up to `--max-model-memory`. When the limit is reached, the least recently used model is evicted. All loaded models are accessible at all times — just reference them by name in each request.

**How do I know which model is being used if I don't specify one?**
Always specify the `"model"` field in your request. oMLX does not have a global default model — omitting it will return a validation error.

**Can I use oMLX as a drop-in replacement for the OpenAI API?**
For chat completions, completions, and embeddings: yes, for most use cases. Some advanced OpenAI features (fine-tuning, assistants, batch API) are not implemented. The `/v1/chat/completions` and `/v1/completions` surfaces are the most complete.

**What is the difference between `--paged-ssd-cache-dir` and `--hot-cache-max-size`?**
The SSD cache stores computed KV blocks on disk, persisting between server restarts. The hot cache (`--hot-cache-max-size`) keeps a portion of those blocks in RAM for fastest access. Both work together: RAM-hot blocks are served immediately; SSD blocks require a disk read but are still faster than recomputing from scratch. Set `--hot-cache-max-size 0` (the default) to use SSD cache only.

**Does oMLX support streaming responses?**
Yes. Add `"stream": true` to any chat completions or completions request. Responses arrive as server-sent events (SSE). Note: streaming combined with tool calls has a known crash in 0.3.7 (see Troubleshooting).

**What quantization formats are supported?**
oMLX supports standard MLX quantized formats (4-bit, 8-bit) as well as oQ, the project's custom quantization scheme. Use pre-quantized models from the `mlx-community` HuggingFace organization for the most reliable results.

**Is my data sent anywhere?**
No. All inference runs locally on your Mac. The only network calls oMLX makes are model downloads (to HuggingFace or ModelScope) and any external calls made by MCP tools you configure. No telemetry or usage data is sent to oMLX's author.

**How do I update oMLX?**
```bash
pip install --upgrade omlx
# or
brew upgrade jundot/tap/omlx
```

**Where are the logs?**
By default, logs go to stdout. Set `OMLX_LOG_DIR` to write to files:
```bash
export OMLX_LOG_DIR=~/.omlx/logs
export OMLX_LOG_RETENTION_DAYS=14
omlx serve
```

**Can I use this with LangChain or LlamaIndex?**
Yes. Both support OpenAI-compatible endpoints. Set the `openai_api_base` (LangChain) or equivalent configuration to `http://127.0.0.1:8000/v1`.

**What does `omlx launch list` show?**
It lists the external AI tools that oMLX can configure and launch: `codex`, `opencode`, `openclaw`, and `pi`. Each tool must be installed separately; oMLX handles only the configuration wiring.

---

_Generated May 2026 from jundot/omlx at f4ed78d. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._