# LocalAI User Guide

**LocalAI** is the open-source, self-hosted AI engine. It runs large language models, vision models, speech, image, and video generation **on your own hardware** — and exposes them behind APIs that are drop-in compatible with OpenAI, Anthropic, ElevenLabs, and Ollama. No GPU is required, no data leaves your infrastructure, and you only install the parts you actually use.

This guide is for people who want to **run** LocalAI and **use** it: install it, load models, call the API, and use the built-in features (chat, embeddings, speech, images, agents, and more). If you operate LocalAI in production — deployment, scaling, security hardening, distributed clusters, and internals — see the companion **Admin Guide**.

> [!NOTE]
> LocalAI is licensed MIT and created by Ettore Di Giacinto ([@mudler](https://github.com/mudler)). The binary you run is called `local-ai`. The default API port is **8080**.

---

## Table of Contents

1. [Quick Start](#1-quick-start)
2. [What LocalAI Does](#2-what-localai-does)
3. [Installation](#3-installation)
4. [Loading Models](#4-loading-models)
5. [Everyday Usage: Calling the API](#5-everyday-usage-calling-the-api)
6. [Working with the Web UI](#6-working-with-the-web-ui)
7. [Feature Walkthroughs](#7-feature-walkthroughs)
8. [Configuring a Model (YAML)](#8-configuring-a-model-yaml)
9. [The `local-ai` Command Line](#9-the-local-ai-command-line)
10. [Common Tasks (Recipes)](#10-common-tasks-recipes)
11. [Troubleshooting](#11-troubleshooting)
12. [FAQ](#12-faq)
13. [Reference](#13-reference)

---

## 1. Quick Start

The fastest way to a working LocalAI is a container. This starts the server, the web UI, and the model gallery in one command.

=== "CPU only"

    ```bash
    docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
    ```

=== "NVIDIA GPU (CUDA 12)"

    ```bash
    docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
    ```

=== "AMD GPU (ROCm)"

    ```bash
    docker run -ti --name local-ai -p 8080:8080 \
      --device=/dev/kfd --device=/dev/dri --group-add=video \
      localai/localai:latest-gpu-hipblas
    ```

=== "Intel GPU (oneAPI)"

    ```bash
    docker run -ti --name local-ai -p 8080:8080 \
      --device=/dev/dri/card1 --device=/dev/dri/renderD128 \
      localai/localai:latest-gpu-intel
    ```

Once the container is up, open the web UI:

```text
http://localhost:8080
```

Pull and run your first model from the gallery, then chat with it:

```bash
# Inside the container (or with a local binary)
local-ai run llama-3.2-1b-instruct:q4_k_m
```

Call it like the OpenAI API:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-1b-instruct:q4_k_m",
    "messages": [{"role": "user", "content": "Say hello in three languages."}]
  }'
```

Expected response shape:

```json
{
  "object": "chat.completion",
  "model": "llama-3.2-1b-instruct:q4_k_m",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Hello! Bonjour! ¡Hola!"},
      "finish_reason": "stop"
    }
  ]
}
```

> [!TIP]
> Already ran LocalAI before? Don't `docker run` again — restart the existing container with `docker start -i local-ai`. A fresh `docker run` with the same `--name` will fail because the name is taken.

> [!IMPORTANT]
> LocalAI **automatically detects your GPU** and downloads the matching backend on demand the first time a model needs it. The core image is small; backends (llama.cpp, vLLM, diffusers, whisper.cpp, …) are pulled only when used. The first run of a new model type may take a while as the backend image downloads.

---

## 2. What LocalAI Does

LocalAI is **a small core, not a bundle**. The core is a Go server that speaks several popular API dialects. Each model is served by a *backend* — a self-contained engine (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX, kokoro, parakeet…) packaged as its own image and pulled only when a model requires it.

| Capability | What you can do | Primary endpoint |
|---|---|---|
| **Text generation** | Chat and completions with LLMs | `POST /v1/chat/completions` |
| **Vision** | Send images to multimodal models | `POST /v1/chat/completions` (with image parts) |
| **Embeddings** | Turn text into vectors for search/RAG | `POST /v1/embeddings` |
| **Text-to-speech** | Generate audio from text | `POST /v1/audio/speech` |
| **Speech-to-text** | Transcribe audio | `POST /v1/audio/transcriptions` |
| **Speaker diarization** | Label who spoke when | `POST /v1/audio/diarization` |
| **Image generation** | Create and edit images | `POST /v1/images/generations`, `/v1/images/inpainting` |
| **Video generation** | Generate video from prompts | `POST /video` |
| **Object detection** | Detect objects in images | `POST /v1/detection` |
| **Face recognition** | Verify, analyze, register, identify faces | `POST /v1/face/*` |
| **Voice recognition** | Verify, register, identify speakers | `POST /v1/voice/*` |
| **Reranking** | Reorder documents by relevance | `POST /v1/rerank` |
| **Tools / functions** | OpenAI-style tool calling | `POST /v1/chat/completions` (with `tools`) |
| **Realtime audio** | Speech-to-speech with tool calling | `GET /v1/realtime` (WebSocket) |
| **Agents** | Autonomous agents with tools, RAG, MCP | `POST /api/agents/:name/chat` |
| **Tokenization** | Count/inspect tokens | `POST /v1/tokenize` |
| **Constrained output** | Force valid JSON/grammar | request `grammar` / `response_format` |

### Compatible API surfaces

LocalAI implements multiple vendor dialects so existing client libraries work unchanged:

- **OpenAI** — `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/audio/*`, `/v1/images/*`, `/v1/models`, `/v1/responses` (Open Responses API), `/v1/realtime`.
- **Anthropic** — `POST /v1/messages`.
- **Ollama** — `/api/chat`, `/api/generate`, `/api/embed`, `/api/tags`, `/api/show`, `/api/ps`, `/api/version`.
- **ElevenLabs** — `/v1/text-to-speech/:voice-id`, `/v1/sound-generation`.
- **Jina** — `/v1/rerank`.

> [!TIP]
> Point any OpenAI SDK at LocalAI by overriding the base URL to `http://localhost:8080/v1` and using any API key (or a real one if you've enabled auth). The same trick works for Anthropic and Ollama SDKs against their respective paths.

---

## 3. Installation

### 3.1 Containers (recommended)

Containers ship everything needed and handle GPU plumbing. Pick the image tag that matches your hardware:

| Hardware | Image tag |
|---|---|
| CPU only | `localai/localai:latest` |
| NVIDIA CUDA 13 | `localai/localai:latest-gpu-nvidia-cuda-13` |
| NVIDIA CUDA 12 | `localai/localai:latest-gpu-nvidia-cuda-12` |
| NVIDIA Jetson (L4T, CUDA 12) | `localai/localai:latest-nvidia-l4t-arm64` |
| NVIDIA Jetson / DGX Spark (L4T, CUDA 13) | `localai/localai:latest-nvidia-l4t-arm64-cuda-13` |
| AMD ROCm | `localai/localai:latest-gpu-hipblas` |
| Intel oneAPI | `localai/localai:latest-gpu-intel` |
| Vulkan | `localai/localai:latest-gpu-vulkan` |

Persist your models and data with a volume so they survive container restarts:

```bash
docker run -ti --name local-ai -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/data:/data \
  localai/localai:latest
```

> [!NOTE]
> `docker run -ti` keeps the container in the foreground so you can watch logs. Add `-d` to run detached. Use `docker logs -f local-ai` to follow logs afterward.

### 3.2 macOS desktop app

Download the DMG from the [latest release](https://github.com/mudler/LocalAI/releases/latest/download/LocalAI.dmg). It bundles a desktop **Launcher** (a Fyne GUI with a system-tray icon) that starts/stops the server for you.

> [!WARNING]
> The DMG is **not signed by Apple**. After installing, clear the quarantine attribute or macOS will refuse to open it:
> ```bash
> sudo xattr -d com.apple.quarantine /Applications/LocalAI.app
> ```

### 3.3 Verifying the install

After the server starts, confirm it's healthy:

```bash
curl http://localhost:8080/readyz
```

```text
OK
```

Check the version:

```bash
curl http://localhost:8080/version
```

List loaded/known models:

```bash
curl http://localhost:8080/v1/models
```

Setup verification checklist:

- [ ] `GET /readyz` returns `OK`
- [ ] Web UI loads at `http://localhost:8080`
- [ ] `GET /v1/models` returns a (possibly empty) model list
- [ ] A model from the gallery installs and answers a chat request

---

## 4. Loading Models

A "model" in LocalAI is a backend engine plus a configuration. You can load models from several sources, all via `local-ai run` or the web UI.

```bash
# From the model gallery (curated catalog at https://models.localai.io)
local-ai run llama-3.2-1b-instruct:q4_k_m

# From Hugging Face (any GGUF file)
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf

# From the Ollama OCI registry
local-ai run ollama://gemma:2b

# From a YAML config file (local path or URL)
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml

# From a standard OCI registry (e.g. Docker Hub)
local-ai run oci://localai/phi-2:latest
```

| URI scheme | Source | Example |
|---|---|---|
| *(bare name)* | LocalAI model gallery | `llama-3.2-1b-instruct:q4_k_m` |
| `huggingface://` | A file on Hugging Face | `huggingface://owner/repo/file.gguf` |
| `ollama://` | Ollama OCI registry | `ollama://gemma:2b` |
| `oci://` | Any OCI registry | `oci://localai/phi-2:latest` |
| `http(s)://` | A YAML model config | `https://example.com/model.yaml` |

> [!TIP]
> Browse the full catalog with `local-ai models list` or at [models.localai.io](https://models.localai.io). Install from the gallery in the UI under **Models**, or via the API: `POST /models/apply`.

> [!NOTE]
> Hugging Face downloads honor `HF_TOKEN` for gated/private repos and `HF_ENDPOINT` / `HF_MIRROR` to point at a mirror. Set these as environment variables before starting LocalAI.

### Where models live

Models are stored under the path set by `--models-path` (env `LOCALAI_MODELS_PATH`), defaulting to `./models`. Each model has a `<name>.yaml` config file (see [§8](#8-configuring-a-model-yaml)) and its weight files alongside.

---

## 5. Everyday Usage: Calling the API

All examples assume the server is at `http://localhost:8080`. If you've enabled authentication, add `-H "Authorization: Bearer YOUR_API_KEY"` to every request.

### 5.1 Chat completions (OpenAI)

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-1b-instruct:q4_k_m",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7,
    "max_tokens": 200
  }'
```

Stream tokens as they're generated by adding `"stream": true`; LocalAI returns Server-Sent Events:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.2-1b-instruct:q4_k_m",
       "messages": [{"role":"user","content":"Write a haiku."}],
       "stream": true}'
```

```text
data: {"choices":[{"delta":{"content":"Silent"}}]}
data: {"choices":[{"delta":{"content":" pond"}}]}
...
data: [DONE]
```

### 5.2 Vision (multimodal chat)

Send an image alongside text to a multimodal model:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-vision-model",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "What is in this image?"},
      {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}}
    ]}]
  }'
```

### 5.3 Embeddings

```bash
curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "your-embedding-model", "input": "The food was delicious."}'
```

> [!NOTE]
> By default embeddings are returned as float arrays. Pass `"encoding_format": "base64"` to match the OpenAI Node.js SDK default. A model only serves embeddings if its config sets `embeddings: true`.

### 5.4 Text-to-speech

```bash
curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "your-tts-model", "input": "Hello from LocalAI", "voice": "default"}' \
  --output speech.wav
```

### 5.5 Speech-to-text (transcription)

```bash
curl http://localhost:8080/v1/audio/transcriptions \
  -F model="whisper-1" \
  -F file="@recording.wav"
```

For who-spoke-when, use diarization:

```bash
curl http://localhost:8080/v1/audio/diarization \
  -F model="your-diarization-model" \
  -F file="@meeting.wav"
```

### 5.6 Image generation

```bash
curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{"model": "your-image-model",
       "prompt": "a red panda coding at a laptop, digital art",
       "size": "512x512"}'
```

Edit an existing image with inpainting via `POST /v1/images/inpainting`.

### 5.7 Reranking

```bash
curl http://localhost:8080/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{"model": "your-reranker",
       "query": "What is the capital of France?",
       "documents": ["Paris is the capital of France.", "Berlin is in Germany."]}'
```

### 5.8 Tool / function calling

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-1b-instruct:q4_k_m",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {"type": "object",
          "properties": {"city": {"type": "string"}}, "required": ["city"]}
      }
    }]
  }'
```

LocalAI can constrain the model's output so the tool call is always valid JSON (see [Constrained output](#74-constrained-output-grammars)).

### 5.9 Using the Anthropic and Ollama dialects

Anthropic Messages API:

```bash
curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.2-1b-instruct:q4_k_m",
       "max_tokens": 256,
       "messages": [{"role": "user", "content": "Hello!"}]}'
```

Ollama-compatible chat:

```bash
curl http://localhost:8080/api/chat \
  -d '{"model": "llama-3.2-1b-instruct:q4_k_m",
       "messages": [{"role": "user", "content": "Hello!"}]}'
```

> [!TIP]
> If you want LocalAI to behave like a drop-in Ollama server (health check on `/`), start it with `--ollama-api-root-endpoint`. The `/api/*` Ollama endpoints are always available regardless of that flag.

---

## 6. Working with the Web UI

LocalAI ships a built-in web interface (a React app) served at the root URL. From it you can:

- **Chat** with any loaded model (multiple chats with saved history).
- **Browse and install models** from the gallery, with VRAM estimates before you install.
- **Browse and install backends** on the fly from the backend gallery.
- **Import a model by URL** (Hugging Face, OCI, YAML).
- **Edit model configuration** in a YAML editor with autocomplete.
- **Manage agents**, run them, and watch their reasoning stream live.
- **Monitor** backends, view backend logs, and see usage metrics.
- **Fine-tune** and **quantize** models from the UI (where backends are installed).

Open it at:

```text
http://localhost:8080
```

> [!NOTE]
> The UI talks to the same API documented here. If you prefer a headless server, disable it with `--disable-webui` — the API endpoints remain available.

> [!TIP]
> Interactive API documentation (Swagger) is available at `http://localhost:8080/swagger/` when the server is running.

---

## 7. Feature Walkthroughs

### 7.1 The model gallery and backend gallery

LocalAI separates **models** (weights + config) from **backends** (the engine that runs them). Both come from galleries — catalogs defined as JSON lists of `{name, url}`.

The default galleries are:

```json
// models  (env LOCALAI_GALLERIES)
[{"name":"localai", "url":"github:mudler/LocalAI/gallery/index.yaml@master"}]
// backends (env LOCALAI_BACKEND_GALLERIES)
[{"name":"localai", "url":"github:mudler/LocalAI/backend/index.yaml@master"}]
```

List, install, and manage backends from the CLI:

```bash
local-ai backends list                 # show available + installed backends
local-ai backends install <backend>    # install one from the gallery
local-ai backends upgrade               # upgrade all (or name them)
local-ai backends uninstall <backend>  # remove one
```

LocalAI supports **36+ backends** and auto-detects the right one for your hardware. Backends are OCI images pulled on demand; you only download the engines you use.

> [!TIP]
> Enable `--auto-upgrade-backends` to keep backend images current automatically, or run `local-ai backends upgrade` on your own schedule.

### 7.2 Built-in agents

LocalAI includes autonomous AI agents (powered by the LocalAGI/cogito engine) that can use tools, query a knowledge base (RAG), connect to MCP servers, and stream their reasoning. Manage them through the web UI or the API:

| Action | Endpoint |
|---|---|
| Chat with an agent | `POST /api/agents/:name/chat` |
| Stream agent events (SSE) | `GET /api/agents/:name/sse` |
| Pause / resume an agent | `PUT /api/agents/:name/pause`, `/resume` |
| Inspect agent status | `GET /api/agents/:name/status` |
| Import / export agents | `POST /api/agents/import`, `GET /api/agents/:name/export` |

You can also run an agent **standalone**, without the full server:

```bash
# Run an agent from the pool registry by name, send one prompt, print the answer
local-ai agent run my-agent --prompt "Summarize today's news"

# Or load an agent from a JSON config file
local-ai agent run --config ./agent.json --prompt "Hello"

# List the agents in your pool registry (agents/pool.json)
local-ai agent list
```

> [!NOTE]
> Standalone agents read from `agents/pool.json` by default (override with `--state-dir`). They connect back to a LocalAI API for inference — set `--api-url` and `--api-key` if the API isn't the local default. Browse and download shareable agent configs from the [Agent Hub](https://agenthub.localai.io).

### 7.3 Model Context Protocol (MCP)

LocalAI can act as an MCP client — letting a model call tools exposed by external MCP servers — and as an MCP host. MCP-augmented chat has dedicated endpoints:

```bash
curl http://localhost:8080/v1/mcp/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model", "messages": [{"role":"user","content":"List my files"}]}'
```

Discover what a model's configured MCP servers expose:

```bash
curl http://localhost:8080/v1/mcp/servers/your-model
curl http://localhost:8080/v1/mcp/prompts/your-model
curl http://localhost:8080/v1/mcp/resources/your-model
```

MCP servers are declared per model in the model YAML under the `mcp:` block (see [§8](#8-configuring-a-model-yaml)). Disable MCP entirely with `--disable-mcp`.

### 7.4 Constrained output (grammars)

To force a model to emit only valid structured output (e.g. always-valid JSON for tool calls), LocalAI supports **constrained grammars**. Provide a grammar in the request or model config, or use `response_format` for JSON. This is what makes function calling reliable on local models. The `grammar` field is part of the model's LLM config and can also be set per request.

### 7.5 Realtime speech-to-speech

The Realtime API provides bidirectional audio with tool calling over a WebSocket:

```text
GET /v1/realtime
POST /v1/realtime/sessions
POST /v1/realtime/transcription_session
```

This enables low-latency voice assistants where the model listens and speaks, optionally calling tools mid-conversation.

### 7.6 Voice and face recognition

LocalAI can verify, analyze, register, and identify both **faces** and **voices**:

```bash
# Register a face under an identity
curl http://localhost:8080/v1/face/register -F model="your-face-model" -F file="@alice.jpg" -F name="alice"
# Identify a face
curl http://localhost:8080/v1/face/identify -F model="your-face-model" -F file="@unknown.jpg"
```

Parallel endpoints exist for voice: `/v1/voice/register`, `/v1/voice/identify`, `/v1/voice/verify`, `/v1/voice/analyze`, `/v1/voice/embed`, `/v1/voice/forget`.

### 7.7 Voice activity detection (VAD)

Detect speech segments in audio with Silero-VAD:

```bash
curl http://localhost:8080/v1/vad \
  -H "Content-Type: application/json" \
  -d '{"model": "silero-vad", "audio": [/* PCM samples */]}'
```

### 7.8 Fine-tuning and quantization

From the UI (or the API) you can fine-tune models with TRL and quantize them to GGUF:

| Action | Endpoint |
|---|---|
| Start a fine-tune job | `POST /finetuning/jobs` |
| Track fine-tune progress (SSE) | `GET /finetuning/jobs/:id/progress` |
| Export a fine-tuned model | `POST /finetuning/jobs/:id/export` |
| Start a quantization job | `POST /quantization/jobs` |
| Track quantization progress (SSE) | `GET /quantization/jobs/:id/progress` |

Quantization accepts a Hugging Face model name or local path, a backend (`llama-cpp-quantization`), and a quantization type (`q4_k_m`, `q5_k_m`, `q8_0`, `f16`, …), producing a GGUF you can then import and serve.

---

## 8. Configuring a Model (YAML)

Most model behavior is controlled by a per-model YAML file in your models directory. The simplest config just names the model and its backend:

```yaml
name: my-llama
backend: llama-cpp
parameters:
  model: llama-3.2-1b-instruct.Q4_K_M.gguf
```

A richer example showing common options:

```yaml
name: my-llama
backend: llama-cpp
f16: true
threads: 8
context_size: 4096
gpu_layers: 35
mmap: true
embeddings: false

parameters:           # request defaults (the OpenAI-style "parameters" block)
  temperature: 0.7
  top_p: 0.9
  top_k: 40
  max_tokens: 512
  seed: -1

template:
  chat: |
    {{.Input}}
  completion: |
    {{.Input}}

stopwords:
  - "</s>"
  - "<|im_end|>"
```

### 8.1 Top-level model keys

| Key | Type | Meaning |
|---|---|---|
| `name` | string | Model name used in API requests |
| `backend` | string | Backend engine (e.g. `llama-cpp`, `vllm`, `diffusers`, `whisper`) |
| `f16` | bool | Use half-precision (often a GPU acceleration toggle) |
| `threads` | int | Threads for parallel computation |
| `embeddings` | bool | Whether this model serves the embeddings endpoint |
| `description`, `usage` | string | Human-readable notes shown in the UI |
| `disabled` | bool | Hide/disable the model |
| `pinned` | bool | Keep the model from being auto-evicted/cleaned |
| `download_files` | list | Files to fetch when the model is installed |
| `known_usecases` | list | Declared capabilities (chat, embeddings, image, …) |

### 8.2 The `parameters:` block (request defaults)

These map to OpenAI request fields and set defaults for the model. Clients can override any of them per request.

| Key | Meaning |
|---|---|
| `temperature` | Sampling temperature |
| `top_p`, `top_k`, `min_p` | Nucleus / top-k / min-p sampling |
| `max_tokens` (`max_completion_tokens`) | Max tokens to generate |
| `seed` | Random seed (deterministic output when fixed) |
| `repeat_penalty`, `repeat_last_n` | Repetition control |
| `frequency_penalty`, `presence_penalty` | OpenAI penalties |
| `typical_p`, `tfz` | Typical / tail-free sampling |
| `n` | Number of results to return |
| `language` | Language hint (e.g. transcription) |
| `negative_prompt` | Negative prompt (image/diffusion) |
| `logit_bias`, `logprobs`, `top_logprobs` | Logit bias and log-probability output |

### 8.3 Inline LLM options (llama.cpp / vLLM / transformers)

These live at the top level of the config and apply to most LLM backends:

| Key | Meaning |
|---|---|
| `gpu_layers` | Number of layers to offload to GPU |
| `context_size` | Context window size |
| `mmap`, `mmlock` | Memory-map / lock model weights |
| `low_vram` | Reduce VRAM usage |
| `main_gpu`, `tensor_split` | Multi-GPU placement |
| `flash_attention` | Enable flash attention |
| `cache_type_k`, `cache_type_v` | KV-cache quantization types |
| `prompt_cache_path`, `prompt_cache_all`, `prompt_cache_ro` | Prompt caching |
| `mirostat`, `mirostat_eta`, `mirostat_tau` | Mirostat sampling |
| `rope_scaling`, `rope_freq_base`, `yarn_*` | RoPE / YaRN context extension |
| `lora_adapter`, `lora_adapters`, `lora_scale(s)`, `lora_base` | LoRA adapters |
| `grammar` | Constrain output to a grammar |
| `stopwords` | Stop sequences |
| `cutstrings`, `trimspace`, `trimsuffix`, `extract_regex` | Post-process output text |
| `mmproj` | Multimodal projector file (vision) |
| `reranking` | Serve as a reranker |
| `draft_model`, `n_draft` | Speculative decoding |

vLLM-specific keys include `gpu_memory_utilization`, `tensor_parallel_size`, `max_model_len`, `dtype`, `quantization`, `trust_remote_code`, `enforce_eager`, `swap_space`, `limit_mm_per_prompt`, and `engine_args` (a raw passthrough to the engine constructor).

### 8.4 The `template:` block

LocalAI applies Go templates to format prompts for each backend. Set `chat`, `chat_message`, `completion`, `edit`, `function`, and `multimodal` templates, or set `use_tokenizer_template: true` to use the model's own chat template (common for vLLM and transformers). `reply_prefix` prepends text to every reply.

### 8.5 Specialized config blocks

| Block | Purpose |
|---|---|
| `diffusers:` | Image/video generation (pipeline type, scheduler, clip skip, controlnet, img2img) |
| `tts:` | Text-to-speech (voice, audio path) |
| `function:` | Function-calling behavior |
| `reasoning:` | Reasoning/thinking extraction |
| `mcp:` | MCP servers this model can call (remote + stdio) |
| `agent:` | Agent behavior bound to the model |
| `pii:` | PII detection/filtering |
| `router:` | Route requests across candidate models by policy |
| `proxy:` | Forward to an external provider (cloud-proxy backend) |
| `limits:` | Admission control (`max_concurrent`, `retry_after_seconds`) |
| `grpc:` | Backend connection retries (`attempts`, `attempts_sleep_time`) |

> [!TIP]
> You rarely need to write a full YAML by hand. Installing from the gallery generates a working config, and the web UI's **Edit Model** page gives you a YAML editor with autocomplete and VRAM estimation.

---

## 9. The `local-ai` Command Line

`local-ai` is the single binary. Running it with no subcommand runs the server (the `run` command). See `local-ai <command> --help` for full per-command details.

| Command | Purpose |
|---|---|
| `run` | Run LocalAI (the default command if none is given) |
| `models` | Manage models and definitions |
| `backends` | Manage backends (`list`, `install`, `uninstall`, `upgrade`) |
| `tts` | Convert text to speech from the CLI |
| `transcript` | Convert audio to text from the CLI |
| `sound-generation` | Generate audio files from text or audio |
| `agent` | Run agents standalone (`run`, `list`) without the full server |
| `worker` | Start a worker for distributed mode (backend-agnostic) |
| `p2p-worker` | Run a worker to distribute load via P2P (llama.cpp only) |
| `agent-worker` | Start an agent worker for distributed mode |
| `federated` | Run LocalAI in federated mode |
| `explorer` | Run the P2P explorer |
| `mcp-server` | Run the admin tool surface as a stdio MCP server (controls a remote LocalAI) |
| `util` | Utility commands |
| `completion` | Generate shell completion scripts (bash, zsh, fish) |

### 9.1 Common `run` flags

Every flag has an environment-variable equivalent (shown in parentheses), so you can configure LocalAI entirely through the environment — ideal for containers.

| Flag | Env var | Default | Purpose |
|---|---|---|---|
| `--address` | `LOCALAI_ADDRESS` | `:8080` | Bind address for the API server |
| `--models-path` | `LOCALAI_MODELS_PATH` | `./models` | Where models are stored |
| `--backends-path` | `LOCALAI_BACKENDS_PATH` | `./backends` | Where backends are stored |
| `--threads` (`-t`) | `LOCALAI_THREADS` | — | Threads for computation (use physical core count) |
| `--context-size` | `LOCALAI_CONTEXT_SIZE` | — | Default context size |
| `--f16` | `LOCALAI_F16` | false | Enable GPU acceleration (half precision) |
| `--cors` | `LOCALAI_CORS` | false | Enable CORS |
| `--upload-limit` | `LOCALAI_UPLOAD_LIMIT` | 15 | Upload size limit (MB) |
| `--api-key` | `LOCALAI_API_KEY` | — | One or more API keys to require for access |
| `--disable-webui` | `LOCALAI_DISABLE_WEBUI` | false | Serve API only, no web UI |
| `--galleries` | `LOCALAI_GALLERIES` | localai | Model galleries (JSON) |
| `--preload-models` | `LOCALAI_PRELOAD_MODELS` | — | Models to apply at startup (JSON) |
| `--load-to-memory` | `LOCALAI_LOAD_TO_MEMORY` | — | Models to load into memory at startup |
| `--p2p` | `LOCALAI_P2P` | false | Enable P2P mode |
| `--debug` | `LOCALAI_LOG_LEVEL=debug` | — | Verbose logging |

Run with a model on the command line:

```bash
local-ai run --address :9090 --threads 8 --context-size 8192 llama-3.2-1b-instruct:q4_k_m
```

> [!TIP]
> LocalAI auto-loads environment variables from `.env`, `localai.env`, `~/localai.env`, `~/.config/localai.env`, and `/etc/localai.env` at startup. Put your `LOCALAI_*` settings there instead of a long command line.

### 9.2 CLI for one-off media tasks

```bash
# Text to speech without the server
local-ai tts --help

# Audio to text without the server
local-ai transcript --help
```

### 9.3 Shell completion

```bash
local-ai completion bash > /etc/bash_completion.d/local-ai
local-ai completion zsh  > "${fpath[1]}/_local-ai"
local-ai completion fish > ~/.config/fish/completions/local-ai.fish
```

---

## 10. Common Tasks (Recipes)

### Run fully on CPU

Omit any GPU image/flags. LocalAI runs on CPU by default; just don't set `--f16`/`gpu_layers` and use a CPU image.

### Use LocalAI from the OpenAI Python SDK

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="llama-3.2-1b-instruct:q4_k_m",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
```

### Preload models at startup

```bash
docker run -ti -p 8080:8080 \
  -e LOCALAI_PRELOAD_MODELS='[{"id":"localai@llama-3.2-1b-instruct:q4_k_m"}]' \
  localai/localai:latest
```

### Keep a model resident in memory

```bash
local-ai run --load-to-memory llama-3.2-1b-instruct:q4_k_m llama-3.2-1b-instruct:q4_k_m
```

### Limit how many backends stay loaded

```bash
# Keep at most 2 backends in memory; least-recently-used are evicted
local-ai run --max-active-backends 2
```

### Require an API key

```bash
local-ai run --api-key "sk-my-secret-key"
```

Then call with `-H "Authorization: Bearer sk-my-secret-key"`.

### Free GPU memory when idle

```bash
local-ai run --watchdog-idle --watchdog-idle-timeout 10m
```

---

## 11. Troubleshooting

Troubleshooting entries follow the format **Error → Cause → Solution → Prevention**.

### Error: container name already in use

**Cause:** You ran `docker run --name local-ai ...` a second time; the name is still taken by the stopped container.
**Solution:** Restart the existing container instead:
```bash
docker start -i local-ai
```
**Prevention:** Use `docker start` to resume, and only `docker run` for first creation.

### Error: macOS says the app "cannot be opened" / is damaged

**Cause:** The DMG is not signed by Apple, so Gatekeeper quarantines it.
**Solution:**
```bash
sudo xattr -d com.apple.quarantine /Applications/LocalAI.app
```
**Prevention:** Run the `xattr` command immediately after installing.

### Error: model responds but the first request is very slow

**Cause:** The matching backend image is downloaded on demand the first time a model type is used.
**Solution:** Wait for the backend to finish downloading (watch the logs). Subsequent requests are fast.
**Prevention:** Preload models/backends at startup, or pre-install the backend with `local-ai backends install <backend>`.

### Error: GPU is not being used

**Cause:** Wrong image (CPU image instead of a GPU one), missing device flags, or `gpu_layers`/`--f16` not set.
**Solution:** Use the GPU image for your hardware (see [§3.1](#31-containers-recommended)), pass the device flags, and set `gpu_layers` in the model config. Issue [#8934](https://github.com/mudler/LocalAI/issues/8934) tracks an Intel-GPU detection case.
**Prevention:** Verify `--gpus all` (NVIDIA) or the `--device` flags (AMD/Intel) are present and the backend matches your accelerator.

### Error: embeddings return the wrong number of dimensions

**Cause:** Some embedding models (e.g. Jina-v3) need configuration to return full-dimension vectors; see issue [#8721](https://github.com/mudler/LocalAI/issues/8721).
**Solution:** Check the model's gallery config/documentation for the correct embedding settings, and ensure `embeddings: true` is set on a model intended for embeddings.
**Prevention:** Validate embedding dimensions against the model card before using them in a vector store.

### Error: diffusers / stable-diffusion image generation fails

**Cause:** Known backend regressions in some versions; see issue [#9979](https://github.com/mudler/LocalAI/issues/9979).
**Solution:** Update LocalAI and the diffusers backend (`local-ai backends upgrade`), and check the backend logs in the UI.
**Prevention:** Keep backends current; test image generation after upgrades.

### Error: server refuses to start when binding to a public address

**Cause:** LocalAI refuses to listen on a public-internet address with no authentication configured (a safety guard). Loopback, RFC 1918 LAN, ULA, link-local, and CGNAT/Tailscale ranges are allowed.
**Solution:** Configure authentication (set `--api-key` or enable auth), or — only if you understand the risk — pass `--allow-insecure-public-bind`.
**Prevention:** Always put auth in front of a publicly bound server.

### Error: out of memory / system RAM misreported in containers

**Cause:** In some container runtimes (e.g. LXD) system RAM is reported incorrectly; see issue [#8059](https://github.com/mudler/LocalAI/issues/8059).
**Solution:** Set `context_size`, `gpu_layers`, and `--max-active-backends` conservatively; enable the memory reclaimer (`--memory-reclaimer`).
**Prevention:** Size models to your hardware and cap concurrent backends.

### Error: web UI logs are noisy / health checks flood the log

**Cause:** UI polling and health-check paths log at INFO by default in older builds.
**Solution:** Update LocalAI; polling/health/resource paths now log at DEBUG (hidden by default).
**Prevention:** Keep LocalAI current.

> [!TIP]
> When in doubt, check the per-backend logs in the web UI (or `GET /api/backend-logs/:modelId`) — backend startup failures (missing libraries, GLIBCXX mismatches, etc.) show up there, not in the main server log.

---

## 12. FAQ

**Q: Do I need a GPU?**
No. LocalAI runs on CPU-only hardware. A GPU (NVIDIA, AMD, Intel, Apple Silicon, or Vulkan) accelerates inference but is optional.

**Q: Is it really compatible with the OpenAI API?**
Yes — it implements the OpenAI chat, completions, embeddings, audio, and images endpoints, plus Anthropic, Ollama, ElevenLabs, and Jina dialects. Point existing SDKs at LocalAI by changing the base URL.

**Q: Where do models come from?**
The curated LocalAI gallery (`local-ai models list` / [models.localai.io](https://models.localai.io)), Hugging Face (`huggingface://`), the Ollama registry (`ollama://`), or any OCI registry (`oci://`).

**Q: Why is the download so big the first time?**
The core is small; each backend engine is a separate image pulled on demand. The first use of a new model type downloads that backend once.

**Q: How do I run more than one model?**
Load several; LocalAI manages them. Use `--max-active-backends` to cap how many stay in memory at once (least-recently-used are evicted).

**Q: Can I keep data out of the cloud entirely?**
Yes — LocalAI is privacy-first and self-hosted; your data never leaves your infrastructure. The only network calls are to fetch models/backends you explicitly request.

**Q: How do I require authentication?**
Start with `--api-key` for static keys, or enable full user auth (OIDC/GitHub/local) for multi-user setups (see the Admin Guide).

**Q: Does it support streaming?**
Yes — add `"stream": true` to chat/completions requests to receive Server-Sent Events.

**Q: Can the model call tools/functions?**
Yes — pass `tools` in the request. LocalAI can constrain output with grammars so tool-call JSON is always valid.

**Q: What's an "agent" vs. a model?**
A model answers single requests. An agent (LocalAGI/cogito) is autonomous — it plans, uses tools, queries a knowledge base, and can connect to MCP servers. Run agents from the UI, the API, or standalone with `local-ai agent run`.

**Q: How do I update?**
Pull the latest image (`docker pull localai/localai:latest`) and recreate the container, or `local-ai backends upgrade` to refresh backends.

**Q: Can I disable the web UI?**
Yes — `--disable-webui`. The API stays available.

**Q: Where are generated images/audio saved?**
Under the path set by `LOCALAI_GENERATED_CONTENT_PATH` (default `/tmp/generated/content`).

**Q: How do I free GPU memory when models are idle?**
Enable the idle watchdog (`--watchdog-idle --watchdog-idle-timeout 10m`) or the memory reclaimer (`--memory-reclaimer`).

**Q: Can I run it on a Mac?**
Yes — download the signed-but-unnotarized DMG and clear the quarantine attribute (see [§3.2](#32-macos-desktop-app)). Apple Silicon is supported via MLX and llama.cpp backends.

**Q: How do I see what the API offers interactively?**
Open `http://localhost:8080/swagger/`.

---

## 13. Reference

### Selected API endpoints

| Method | Path | Purpose |
|---|---|---|
| POST | `/v1/chat/completions` | Chat (OpenAI) |
| POST | `/v1/completions` | Text completion (OpenAI) |
| POST | `/v1/embeddings` | Embeddings (OpenAI) |
| POST | `/v1/messages` | Messages (Anthropic) |
| POST | `/api/chat`, `/api/generate` | Chat/generate (Ollama) |
| POST | `/v1/audio/speech` | Text-to-speech |
| POST | `/v1/audio/transcriptions` | Speech-to-text |
| POST | `/v1/audio/diarization` | Speaker diarization |
| POST | `/v1/images/generations` | Image generation |
| POST | `/v1/images/inpainting` | Image inpainting/editing |
| POST | `/video` | Video generation |
| POST | `/v1/detection` | Object detection |
| POST | `/v1/face/*`, `/v1/voice/*` | Face / voice recognition |
| POST | `/v1/rerank` | Reranking |
| POST | `/v1/vad` | Voice activity detection |
| POST | `/v1/tokenize` | Tokenization |
| GET | `/v1/models` | List models |
| GET | `/v1/realtime` | Realtime audio (WebSocket) |
| POST | `/v1/responses` | Open Responses API |
| POST | `/v1/mcp/chat/completions` | MCP-augmented chat |
| GET | `/swagger/` | Interactive API docs |
| GET | `/readyz`, `/healthz` | Health checks |
| GET | `/metrics` | Prometheus metrics |
| GET | `/version` | Server version |

### Common environment variables

| Variable | Purpose |
|---|---|
| `LOCALAI_ADDRESS` | Server bind address (default `:8080`) |
| `LOCALAI_MODELS_PATH` | Model storage directory |
| `LOCALAI_BACKENDS_PATH` | Backend storage directory |
| `LOCALAI_THREADS` | Computation threads |
| `LOCALAI_CONTEXT_SIZE` | Default context size |
| `LOCALAI_API_KEY` | Required API key(s) |
| `LOCALAI_GALLERIES` | Model galleries (JSON) |
| `LOCALAI_BACKEND_GALLERIES` | Backend galleries (JSON) |
| `LOCALAI_F16` | Enable half-precision |
| `LOCALAI_CORS` | Enable CORS |
| `HF_TOKEN` | Hugging Face token for gated downloads |
| `HF_ENDPOINT` / `HF_MIRROR` | Hugging Face mirror |

### Model URI schemes

| Scheme | Source |
|---|---|
| *(name)* | LocalAI gallery |
| `huggingface://` | Hugging Face file |
| `ollama://` | Ollama OCI registry |
| `oci://` | Any OCI registry |
| `http(s)://` | YAML config URL |

### Useful links

- Documentation: <https://localai.io/>
- Model catalog: <https://models.localai.io/>
- Agent Hub: <https://agenthub.localai.io>
- Source: <https://github.com/mudler/LocalAI>
- Community: [Discord](https://discord.gg/uJAeKSAGDy)

---

_Generated June 2026 from mudler/LocalAI at 9f11b09. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._