# GPT4All User Guide

GPT4All is an open-source ecosystem by Nomic AI that lets you run large language models entirely on your own hardware — no cloud, no API keys, no data leaving your machine. It ships as a polished desktop application for Windows, macOS, and Linux, plus SDKs for Python and Node.js, and a terminal-based CLI. Whether you want a private ChatGPT replacement or need to embed local LLM inference into your application, GPT4All delivers production-ready tooling backed by the llama.cpp engine.

---

## 1. Quick Start

Get up and running in under five minutes with the desktop application.

**Step 1 — Download the installer** from [gpt4all.io](https://gpt4all.io) for your operating system.

**Step 2 — Install and launch.** Run the installer; GPT4All opens as a single-instance application (only one window at a time).

**Step 3 — Download a model.** On first launch, GPT4All presents the model gallery. Pick a model that fits your hardware:

| Your RAM | Recommended Model | Size |
|----------|-------------------|------|
| 8 GB | Llama 3.2 1B Instruct | ~1 GB |
| 16 GB | Llama 3 8B Instruct | ~4.7 GB |
| 32 GB+ | DeepSeek-R1 Distill Qwen 14B | ~9 GB |

**Step 4 — Start chatting.** Select the downloaded model, type a message, and press Enter.

> [!TIP]
> If you have a dedicated GPU (NVIDIA, AMD, or Intel), GPT4All will automatically detect and use it for faster inference. You can verify GPU usage in the application settings.

---

## 2. Installation

### 2.1 Desktop Application

=== "Windows"

1. Download the `.exe` installer from [gpt4all.io](https://gpt4all.io).
2. Run the installer and follow the prompts.
3. Launch GPT4All from the Start Menu or desktop shortcut.

=== "macOS"

1. Download the `.dmg` from [gpt4all.io](https://gpt4all.io).
2. Open the `.dmg` and drag GPT4All to Applications.
3. On first launch, right-click and select Open to bypass Gatekeeper.

=== "Linux"

1. Download the `.run` installer from [gpt4all.io](https://gpt4all.io).
2. Make it executable and run:

```bash
chmod +x gpt4all-installer-linux-vX.Y.Z.run
./gpt4all-installer-linux-vX.Y.Z.run
```

3. Alternatively, install via Flatpak:

```bash
flatpak install flathub io.gpt4all.gpt4all
flatpak run io.gpt4all.gpt4all
```

> [!NOTE]
> The Linux `.run` installer may not create a desktop shortcut automatically on some distributions (see [#3640](https://github.com/nomic-ai/gpt4all/issues/3640), [#3456](https://github.com/nomic-ai/gpt4all/issues/3456)). You can launch GPT4All directly from the installation directory or create a `.desktop` file manually.

### 2.2 Python SDK

Requires Python 3.8 or higher.

```bash
pip install gpt4all
```

For NVIDIA GPU acceleration:

```bash
pip install gpt4all[cuda]
```

This installs `nvidia-cuda-runtime-cu11` and `nvidia-cublas-cu11` as additional dependencies.

> [!IMPORTANT]
> The `[cuda]` extra targets CUDA 11. If your system has only CUDA 12 installed, you may encounter library loading errors (see [#3033](https://github.com/nomic-ai/gpt4all/issues/3033)). Install the CUDA 11 compatibility libraries or use CPU inference.

Verify the installation:

```python
from gpt4all import GPT4All
model = GPT4All("orca-mini-3b-gguf2-q4_0.gguf")
print(model.generate("Hello, world!", max_tokens=50))
```

### 2.3 Node.js SDK

Requires Node.js 18 or higher.

```bash
npm install gpt4all
```

The package includes native bindings built with `node-addon-api` and `node-gyp-build`. On first use, it downloads the llama.cpp backend library automatically.

Verify the installation:

```javascript
const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("orca-mini-3b-gguf2-q4_0.gguf");
    const result = await createCompletion(model, [
        { role: "user", content: "Hello!" }
    ]);
    console.log(result.choices[0].message.content);
    model.dispose();
}
main();
```

### 2.4 CLI

The CLI is bundled with the Python SDK. After installing the Python package:

```bash
python -m gpt4all repl
```

Or check the version:

```bash
python -m gpt4all version
```

---

## 3. Configuration

### 3.1 Desktop Application Settings

GPT4All stores settings via Qt's QSettings system in INI format. The settings file location varies by OS:

| OS | Settings Location |
|----|-------------------|
| Windows | `%APPDATA%\nomic.ai\GPT4All.ini` |
| macOS | `~/Library/Preferences/io.gpt4all.gpt4all.plist` |
| Linux | `~/.config/nomic.ai/GPT4All.conf` |

Key configurable settings include:

| Setting | Description | Default |
|---------|-------------|---------|
| Model directory | Where GGUF model files are stored | OS-specific (e.g., `~/.local/share/nomic.ai/GPT4All/` on Linux) |
| GPU backend | Vulkan, CUDA, or Kompute | Auto-detected |
| Context length | Tokens of conversation history retained | Model-dependent (minimum: 8) |
| API server port | Port for the built-in OpenAI-compatible server | Off by default |
| Language | Interface language | System locale |

Available interface languages: English (US), Spanish (Mexico), Italian (Italy), Portuguese (Brazil), Romanian (Romania), Chinese Simplified, Chinese Traditional.

### 3.2 Model Storage

All GGUF model files are stored in a single, user-configurable directory. Models are downloaded from `https://gpt4all.io/models/gguf/` and model metadata is fetched from `https://gpt4all.io/models/models3.json`.

To change the model directory in the desktop app, go to **Settings** and update the model path. All SDKs also accept a custom model path.

### 3.3 Python SDK Configuration

The `GPT4All` constructor accepts configuration at instantiation:

```python
from gpt4all import GPT4All

model = GPT4All(
    model_name="mistral-7b-instruct-v0.1.Q4_0.gguf",
    device="gpu"           # "cpu", "gpu", "amd", "nvidia", "intel"
)
```

Models are cached in `~/.cache/gpt4all/` by default. To use a different directory:

```python
model = GPT4All(
    model_name="mistral-7b-instruct-v0.1.Q4_0.gguf",
    model_path="/path/to/your/models/"
)
```

### 3.4 Node.js SDK Configuration

```javascript
const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf", {
    modelPath: "/path/to/models",    // Custom model directory
    allowDownload: true,              // Auto-download if missing (default: true)
    verbose: false,                   // Verbose logging
    device: "gpu",                    // "cpu", "gpu", "amd", "nvidia", "intel"
    nCtx: 2048,                       // Context window size (default: 2048)
    ngl: 100                          // Number of GPU layers (default: 100)
});
```

### 3.5 Environment Variables

| Variable | Purpose |
|----------|---------|
| `GPT4ALL_VERBOSE_LLAMACPP` | Set to any value to enable verbose llama.cpp backend logging |

> [!TIP]
> Enable `GPT4ALL_VERBOSE_LLAMACPP` when debugging model loading issues — it prints detailed information about GGUF file parsing, tensor allocation, and GPU memory mapping.

### 3.6 Remote / API Model Configuration

GPT4All's desktop app supports connecting to external OpenAI-compatible endpoints (OpenAI, Groq, Mistral, etc.) via the ChatAPI class. Configure these in the application settings by providing the API endpoint URL and your API key. This lets you use cloud models alongside local ones in the same interface.

---

## 4. Core Usage

### 4.1 Desktop Application

#### Starting a Conversation

1. Select a model from the model dropdown at the top of the chat window.
2. Type your message and press Enter or click Send.
3. The model generates a response locally on your hardware.

GPT4All enforces a single-instance policy — attempting to launch a second instance activates the existing window instead.

#### Managing Models

The model gallery lists all available models with their sizes, descriptions, and hardware requirements. Models use the GGUF format and are sourced from the GPT4All model repository.

**Popular models include:**

| Model | Parameters | Quantized Size | Best For |
|-------|-----------|----------------|----------|
| GPT4All Reasoner v1 | — | Varies | Reasoning tasks |
| Llama 3 8B Instruct | 8B | ~4.7 GB | General purpose |
| Llama 3.2 1B Instruct | 1B | ~0.8 GB | Low-resource devices |
| Llama 3.2 3B Instruct | 3B | ~2 GB | Balanced performance |
| Mistral Instruct | 7B | ~4 GB | Instruction following |
| Nous Hermes 2 Mistral | 7B | ~4 GB | Diverse tasks |
| DeepSeek-R1 Distill Qwen 7B | 7B | ~4.7 GB | Reasoning and code |
| Phi-3 Mini | 3.8B | ~2.2 GB | Compact, fast |
| StarCoder 2 7B | 7B | ~4 GB | Code generation |

To download a model, click the download icon next to it in the gallery. To remove a model, delete its `.gguf` file from the model directory.

#### Supported Model Architectures

The llama.cpp backend supports a wide range of architectures:

- **General:** LLaMA, Falcon, GPT-2, GPT-NeoX, MPT, Bloom, StableLM
- **Instruction-tuned:** Qwen, Qwen2, Qwen2 MoE, Phi-2, Phi-3, Gemma, Gemma 2, InternLM2, Command-R
- **Code:** StarCoder, StarCoder 2, CodeShell, Refact
- **Specialized:** Baichuan, Granite, GraniteMoE, JAIS, ChatGLM, Orion, Xverse, OLMo, OLMoE, OpenELM, DeepSeek2
- **Embedding:** BERT, Nomic-BERT

Maximum supported GGUF version: 3.

#### LocalDocs (RAG)

LocalDocs lets you ground model responses in your own documents. The model retrieves relevant snippets from your files and includes them as context.

**Setting up a LocalDocs collection:**

1. Go to **LocalDocs** in the settings or sidebar.
2. Click **Add Collection** and select a folder containing your documents.
3. GPT4All indexes the folder contents. Supported file types:
   - `.txt` — Plain text
   - `.md` — Markdown
   - `.rst` — reStructuredText
   - `.xlsx` — Excel spreadsheets
4. Enable the collection in your chat to start asking questions grounded in your documents.

> [!NOTE]
> LocalDocs embedding runs locally. Large document collections may take significant time to index on first setup. There is currently no visual progress indicator for the embedding process ([#2579](https://github.com/nomic-ai/gpt4all/issues/2579)). OpenDocument formats (.odt, .ods) are not yet supported ([#3477](https://github.com/nomic-ai/gpt4all/issues/3477)).

#### File Attachments

You can attach files directly to a chat message for the model to read and respond to. Supported formats:

- `.txt` — Plain text files
- `.md` — Markdown files
- `.rst` — reStructuredText files
- `.xlsx` — Excel spreadsheets

Click the attachment icon in the chat input area to select a file.

#### Code Interpreter

GPT4All includes a built-in JavaScript code interpreter tool. When enabled, the model can write and execute JavaScript code to answer questions that require computation, data processing, or logic.

The interpreter enforces a maximum of 3 consecutive tool calls per turn to prevent runaway execution.

#### Built-in API Server

GPT4All includes an OpenAI-compatible API server that runs on localhost. Enable it in settings and configure the port.

**Available endpoints:**

| Method | Path | Description |
|--------|------|-------------|
| GET | `/v1/models` | List available models |
| GET | `/v1/models/<model>` | Get model details |
| POST | `/v1/completions` | Text completion |
| POST | `/v1/chat/completions` | Chat completion |

**Request parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `model` | string | (required) | Model name |
| `max_tokens` | integer | 16 | Maximum tokens to generate |
| `temperature` | float | 1.0 | Sampling temperature (0–2) |
| `top_p` | float | 1.0 | Nucleus sampling (0–1) |
| `min_p` | float | 0.0 | Minimum probability threshold (0–1) |
| `n` | integer | 1 | Number of completions |

**Example — curl request to the local API:**

```bash
curl http://localhost:4891/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-7b-instruct-v0.1.Q4_0.gguf",
    "messages": [{"role": "user", "content": "Explain quantum computing briefly."}],
    "max_tokens": 200,
    "temperature": 0.7
  }'
```

> [!WARNING]
> The API server binds to **localhost only** — it is not accessible from other machines on your network. The following OpenAI API features are **not supported**: `stream`, `frequency_penalty`, `presence_penalty`, `seed`, `stop`, `logit_bias`, `logprobs`, `tools`, `tool_choice`, `functions`, and `response_format`. Requests using these parameters will be ignored or may produce errors.

#### Chat History

Conversations are automatically saved and can be resumed. The chat serialization format uses version 12 with magic number `0xF5D553CC`. Chat history is stored locally alongside the application data.

### 4.2 Python SDK

#### Basic Text Generation

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")
response = model.generate("What is the capital of France?", max_tokens=100)
print(response)
```

#### Chat Completion (Messages Format)

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain photosynthesis in simple terms."}
]
response = model.chat_completion(messages)
print(response["choices"][0]["message"]["content"])
```

#### Chat Sessions (Multi-turn Conversations)

Use the chat session context manager to maintain conversation history across multiple exchanges:

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

with model.chat_session():
    response1 = model.generate("My name is Alice.")
    print(response1)

    response2 = model.generate("What is my name?")
    print(response2)  # The model remembers "Alice"
```

#### Streaming Responses

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

for token in model.generate("Write a haiku about coding.", streaming=True):
    print(token, end="", flush=True)
print()
```

#### Generation Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_tokens` | int | 200 | Maximum tokens to generate |
| `temp` | float | 0.7 | Sampling temperature |
| `top_k` | int | 40 | Top-K sampling |
| `top_p` | float | 0.9 | Nucleus sampling |
| `min_p` | float | 0.0 | Minimum probability threshold |
| `repeat_penalty` | float | 1.18 | Repetition penalty |
| `repeat_last_n` | int | 64 | Window for repetition penalty |
| `n_batch` | int | 8 | Batch size for prompt processing |
| `streaming` | bool | False | Stream tokens one at a time |

#### GPU Inference

```python
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")
```

Valid device values: `"cpu"`, `"gpu"`, `"amd"`, `"nvidia"`, `"intel"`.

### 4.3 Node.js SDK

#### Loading a Model

```javascript
const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf", {
        device: "gpu",
        nCtx: 4096,
        ngl: 100
    });

    // Use the model...

    model.dispose();  // Always dispose when done
}
```

#### Chat Completion

```javascript
const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const result = await createCompletion(model, [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "What is machine learning?" }
    ]);

    console.log(result.choices[0].message.content);
    model.dispose();
}
main();
```

#### Streaming Responses

```javascript
const { loadModel, createCompletionStream } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const stream = createCompletionStream(model, [
        { role: "user", content: "Tell me a story." }
    ]);

    stream.tokens.on("data", (token) => {
        process.stdout.write(token);
    });

    await stream.result;
    console.log("\nDone.");
    model.dispose();
}
main();
```

#### Async Generator

```javascript
const { loadModel, createCompletionGenerator } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const generator = createCompletionGenerator(model, [
        { role: "user", content: "List 5 programming languages." }
    ]);

    for await (const chunk of generator) {
        process.stdout.write(chunk);
    }

    model.dispose();
}
main();
```

#### Embeddings

```javascript
const { loadModel, createEmbedding } = require("gpt4all");

async function main() {
    const model = await loadModel("nomic-embed-text-v1.5.f16.gguf");

    const embedding = await createEmbedding(model, "Hello, world!", {
        prefix: "search_document",
        dimensionality: 768
    });

    console.log(embedding.embeddings[0].length);  // 768
    model.dispose();
}
main();
```

**Embedding prefix options** (for Nomic Embed models):

| Prefix | Use Case |
|--------|----------|
| `search_query` | Query text for semantic search |
| `search_document` | Document text being indexed |
| `classification` | Text classification tasks |
| `clustering` | Text clustering tasks |

**Long text handling** (`longTextMode`):

| Mode | Behavior |
|------|----------|
| `"mean"` | Average embeddings across chunks |
| `"truncate"` | Truncate text to context window |

#### Chat Sessions (Conversation Memory)

```javascript
const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const session = await model.createChatSession({
        systemPrompt: "You are a Python expert."
    });

    const r1 = await createCompletion(session, [
        { role: "user", content: "What is a decorator?" }
    ]);
    console.log(r1.choices[0].message.content);

    const r2 = await createCompletion(session, [
        { role: "user", content: "Give me an example." }
    ]);
    console.log(r2.choices[0].message.content);

    model.dispose();
}
main();
```

#### Downloading Models Programmatically

```javascript
const { downloadModel } = require("gpt4all");

const controller = downloadModel("mistral-7b-instruct-v0.1.Q4_0.gguf", {
    modelPath: "./models"
});

controller.promise.then(() => {
    console.log("Download complete!");
}).catch((err) => {
    console.error("Download failed:", err);
});

// Cancel if needed:
// controller.cancel();
```

Downloads support automatic resume — if a partial file exists, the download continues from where it left off.

#### Default Prompt Parameters

The Node.js SDK uses these defaults when no overrides are provided:

| Parameter | Default |
|-----------|---------|
| `temp` | 0.1 |
| `topK` | 40 |
| `topP` | 0.9 |
| `minP` | 0 |
| `repeatPenalty` | 1.18 |
| `repeatLastN` | 10 |
| `nBatch` | 100 |

Default prompt template: `### Human:\n%1\n\n### Assistant:\n`

### 4.4 CLI

#### Starting the REPL

```bash
python -m gpt4all repl
```

With options:

```bash
python -m gpt4all repl --model mistral-7b-instruct-v0.1.Q4_0.gguf --device gpu --n-threads 8
```

| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--model` | `-m` | `mistral-7b-instruct-v0.1.Q4_0.gguf` | Model filename |
| `--n-threads` | `-t` | System default | CPU threads for inference |
| `--device` | `-d` | `cpu` | Device: `gpu`, `amd`, `nvidia`, `intel` |

#### REPL Commands

| Command | Action |
|---------|--------|
| `/help` | Show available commands |
| `/reset` | Clear conversation history |
| `/clear` | Clear the screen |
| `/exit` | Quit the REPL |

The CLI uses streaming output by default, printing tokens as they are generated with these default parameters: `max_tokens=200`, `temp=0.9`, `top_k=40`, `top_p=0.9`, `min_p=0`, `repeat_penalty=1.1`, `repeat_last_n=64`, `n_batch=9`.

---

## 5. Common Tasks

### Recipe 1: Private Document Q&A

Use LocalDocs in the desktop app to query your own documents without sending data to the cloud.

1. Open GPT4All and go to **LocalDocs**.
2. Create a collection pointing to a folder with your `.txt`, `.md`, `.rst`, or `.xlsx` files.
3. Wait for indexing to complete.
4. Start a new chat, enable the collection, and ask questions about your documents.

### Recipe 2: Replace OpenAI API with Local Inference

Point any OpenAI-compatible client at GPT4All's built-in API server.

1. Enable the API server in GPT4All settings and note the port (e.g., 4891).
2. Configure your application:

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:4891/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="mistral-7b-instruct-v0.1.Q4_0.gguf",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=100
)
print(response.choices[0].message.content)
```

> [!CAUTION]
> Remember that streaming, function calling, and several other OpenAI API features are not implemented in GPT4All's server. Test your integration thoroughly before switching from a cloud API.

### Recipe 3: Batch Processing with Python

Process multiple prompts efficiently using the Python SDK:

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

prompts = [
    "Summarize the benefits of exercise.",
    "Explain how a compiler works.",
    "Write a Python function to reverse a string."
]

for prompt in prompts:
    response = model.generate(prompt, max_tokens=200, temp=0.7)
    print(f"Q: {prompt}\nA: {response}\n{'='*60}\n")
```

### Recipe 4: Building a Chatbot with Node.js

```javascript
const { loadModel, createCompletion } = require("gpt4all");
const readline = require("readline");

async function chatbot() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");
    const session = await model.createChatSession({
        systemPrompt: "You are a friendly assistant."
    });

    const rl = readline.createInterface({
        input: process.stdin,
        output: process.stdout
    });

    function ask() {
        rl.question("You: ", async (input) => {
            if (input.toLowerCase() === "exit") {
                model.dispose();
                rl.close();
                return;
            }
            const result = await createCompletion(session, [
                { role: "user", content: input }
            ]);
            console.log(`Bot: ${result.choices[0].message.content}\n`);
            ask();
        });
    }
    ask();
}
chatbot();
```

### Recipe 5: Semantic Search with Embeddings

Generate embeddings for a document corpus using the Node.js SDK:

```javascript
const { loadModel, createEmbedding } = require("gpt4all");

async function buildIndex(documents) {
    const model = await loadModel("nomic-embed-text-v1.5.f16.gguf");

    const embeddings = [];
    for (const doc of documents) {
        const result = await createEmbedding(model, doc, {
            prefix: "search_document",
            dimensionality: 768
        });
        embeddings.push(result.embeddings[0]);
    }

    model.dispose();
    return embeddings;
}

async function search(query, documents, index) {
    const model = await loadModel("nomic-embed-text-v1.5.f16.gguf");

    const queryEmb = await createEmbedding(model, query, {
        prefix: "search_query",
        dimensionality: 768
    });

    // Compute cosine similarity (simplified)
    const scores = index.map((docEmb, i) => {
        const dot = docEmb.reduce((sum, v, j) => sum + v * queryEmb.embeddings[0][j], 0);
        return { index: i, score: dot, text: documents[i] };
    });

    model.dispose();
    return scores.sort((a, b) => b.score - a.score);
}
```

### Recipe 6: GPU Selection for Multi-GPU Systems

```python
from gpt4all import GPT4All

# Use any available GPU
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

# Prefer NVIDIA
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="nvidia")

# Prefer AMD
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="amd")
```

### Recipe 7: Adjusting Generation Quality

Control creativity and consistency through temperature and sampling parameters:

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

# Creative writing — high temperature, broad sampling
creative = model.generate(
    "Write a poem about the ocean.",
    max_tokens=300, temp=1.2, top_k=80, top_p=0.95
)

# Factual answer — low temperature, narrow sampling
factual = model.generate(
    "What year was Python created?",
    max_tokens=50, temp=0.1, top_k=10, top_p=0.5
)

# Code generation — moderate settings with repeat penalty
code = model.generate(
    "Write a Python quicksort function.",
    max_tokens=400, temp=0.3, repeat_penalty=1.2
)
```

### Recipe 8: Using Custom GGUF Models

Any GGUF model from Hugging Face or other sources can be used:

1. Download the `.gguf` file to your model directory (`~/.cache/gpt4all/` by default).
2. Load it by filename:

```python
from gpt4all import GPT4All

model = GPT4All("my-custom-model.Q4_K_M.gguf")
response = model.generate("Hello!", max_tokens=100)
```

> [!IMPORTANT]
> Only GGUF format (up to version 3) is supported. Older GGML format files will not load. Convert legacy models using the `convert` scripts in the llama.cpp repository.

### Recipe 9: System Prompts for Specialized Behavior

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

messages = [
    {"role": "system", "content": "You are a SQL expert. Always respond with valid SQL queries and brief explanations."},
    {"role": "user", "content": "Find all users who signed up in the last 30 days."}
]

response = model.chat_completion(messages)
print(response["choices"][0]["message"]["content"])
```

### Recipe 10: Resumable Model Downloads (Node.js)

If a download is interrupted, it resumes automatically on the next attempt:

```javascript
const { downloadModel } = require("gpt4all");

const controller = downloadModel("llama-3-8b-instruct.Q4_0.gguf");

// Monitor progress
controller.promise
    .then(() => console.log("Download complete!"))
    .catch((err) => {
        if (err.message.includes("cancel")) {
            console.log("Download cancelled. Run again to resume.");
        } else {
            console.error("Error:", err);
        }
    });

// Cancel if needed (file is preserved for resume)
setTimeout(() => controller.cancel(), 30000);
```

### Recipe 11: Integrating with Express.js

Wrap GPT4All in a simple REST API:

```javascript
const express = require("express");
const { loadModel, createCompletion } = require("gpt4all");

const app = express();
app.use(express.json());

let model;

app.post("/chat", async (req, res) => {
    if (!model) {
        model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");
    }

    const result = await createCompletion(model, req.body.messages || [
        { role: "user", content: req.body.prompt || "Hello" }
    ]);

    res.json(result);
});

app.listen(3000, () => console.log("Server on http://localhost:3000"));
```

### Recipe 12: Multi-turn CLI Conversation Script

```python
from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

with model.chat_session():
    while True:
        user_input = input("\nYou: ").strip()
        if user_input.lower() in ("exit", "quit"):
            break
        response = model.generate(user_input, max_tokens=300, temp=0.7)
        print(f"\nAssistant: {response}")
```

---

## 6. Troubleshooting

### 6.1 Installation Issues

**Problem: Linux installer does not run or create a desktop shortcut**

The `.run` installer may fail silently on some Linux distributions, particularly those without certain GUI libraries.

- Ensure `libxcb`, `libGL`, and `libfontconfig` are installed.
- Run the installer from a terminal to see error output.
- If the installer succeeds but no shortcut appears, create a `.desktop` file manually pointing to the GPT4All binary.
- Consider using the Flatpak package as an alternative: `flatpak install flathub io.gpt4all.gpt4all`.

Related: [#3640](https://github.com/nomic-ai/gpt4all/issues/3640), [#3456](https://github.com/nomic-ai/gpt4all/issues/3456).

**Problem: Application fails to start after installation**

- On Windows, run as administrator or check that your antivirus is not blocking the executable.
- On macOS, right-click the app and choose Open to bypass Gatekeeper on first launch.
- On Linux, check for missing shared libraries with `ldd` on the GPT4All binary.

Related: [#3456](https://github.com/nomic-ai/gpt4all/issues/3456).

**Problem: Python SDK installation fails on pip**

```bash
pip install --upgrade pip setuptools wheel
pip install gpt4all
```

If building native extensions fails, ensure you have a C++ compiler installed (MSVC on Windows, gcc/clang on Linux/macOS).

**Problem: Node.js SDK `node-gyp` build errors**

Ensure you have the native build tools:

=== "Windows"
```bash
npm install -g windows-build-tools
```

=== "macOS"
```bash
xcode-select --install
```

=== "Linux"
```bash
sudo apt install build-essential python3
```

### 6.2 Model Loading Issues

**Problem: Model download stalls or fails**

- Check your internet connection and available disk space.
- Models are downloaded from `https://gpt4all.io/models/gguf/` — ensure this domain is not blocked by firewall or proxy.
- Delete any partial `.part` file in the model directory and retry.

**Problem: "Unsupported GGUF version" error**

GPT4All supports GGUF versions up to 3. If you downloaded a model with a newer GGUF version, you need to either update GPT4All or find a compatible quantization of the model.

**Problem: Model loads but produces garbage output**

- Verify the model file is not corrupted (re-download it).
- Some models require specific prompt templates. Check the model card for the correct chat format.
- Reduce the context length if you are running low on RAM — an insufficient allocation can cause buffer corruption.

**Problem: "Could not find model" error in Python SDK**

The SDK looks for models in `~/.cache/gpt4all/` by default. Either:
1. Place the `.gguf` file in that directory, or
2. Specify `model_path` explicitly:

```python
model = GPT4All("my-model.gguf", model_path="/path/to/models/")
```

### 6.3 GPU and Performance Issues

**Problem: Vulkan error — `vk::Device::createFence: ErrorInitializationFailed`**

This occurs when the Vulkan driver is outdated or incompatible.

- Update your GPU drivers to the latest version.
- If using an integrated GPU, try forcing CPU mode or switching to a discrete GPU.
- On Linux, ensure the Vulkan ICD loader is correctly installed (`vulkan-tools` package provides `vulkaninfo` for diagnosis).

Related: [#3259](https://github.com/nomic-ai/gpt4all/issues/3259).

**Problem: CUDA 12 installed but GPT4All reports missing CUDA 11**

The Python SDK's `[cuda]` extra installs CUDA 11 runtime libraries. If your system has only CUDA 12:

- Install the CUDA 11 compatibility libraries alongside CUDA 12.
- Or use CPU inference by not installing the `[cuda]` extra.
- Or use the desktop application, which handles CUDA detection automatically.

Related: [#3033](https://github.com/nomic-ai/gpt4all/issues/3033).

**Problem: GPU not detected or inference falls back to CPU**

- Verify GPU support with `GPT4ALL_VERBOSE_LLAMACPP=1` to see backend selection logs.
- Ensure your GPU drivers are up to date.
- Check that the correct device string is used: `"gpu"`, `"nvidia"`, `"amd"`, or `"intel"`.
- On desktop, check Settings to confirm GPU acceleration is enabled.

Related: [#1676](https://github.com/nomic-ai/gpt4all/issues/1676) (Intel GPU support).

**Problem: Out of memory (OOM) when loading large models**

- Choose a smaller quantization (Q4_0 is smallest, Q8_0 is largest per parameter).
- Reduce `nCtx` (context window) — each token in context consumes memory.
- Close other applications to free RAM/VRAM.
- Consider a smaller model: 1B–3B parameter models run comfortably on 8 GB RAM.

**Problem: Inference is very slow**

- Enable GPU acceleration if not already active.
- Reduce `max_tokens` to limit generation length.
- Use a smaller model for faster inference.
- On CPU, increase the number of threads with `--n-threads` (CLI) or `n_threads` parameter.
- Ensure memory-mapped I/O is working (GPT4All uses `mmap` by default).

### 6.4 API Server Issues

**Problem: Cannot connect to the built-in API server**

- Verify the server is enabled in GPT4All settings.
- Check the port number — use `curl http://localhost:<port>/v1/models` to test.
- The server binds to localhost only; it is inaccessible from other machines by design.

**Problem: `TypeError: 'OpenAIResponse' object is not iterable` with `stream=True`**

Streaming is not implemented in the GPT4All API server. Remove `stream=True` from your request or set it to `false`.

Related: [#704](https://github.com/nomic-ai/gpt4all/issues/704).

**Problem: API returns empty or truncated responses**

- Increase `max_tokens` in your request (default is only 16).
- Ensure the model is fully loaded before sending requests.

### 6.5 Desktop Application Issues

**Problem: `chat.exe` process continues running after closing the application**

On Windows, the chat process may persist in the background after the window is closed.

- Open Task Manager and manually end the `chat.exe` process.
- This is a known issue being tracked.

Related: [#3656](https://github.com/nomic-ai/gpt4all/issues/3656).

**Problem: Cannot resize or maximize the window**

This was reported in v3.0.0 and may affect certain window managers on Linux.

- Try toggling fullscreen mode.
- Update to the latest GPT4All release (v3.10.0+).

Related: [#2712](https://github.com/nomic-ai/gpt4all/issues/2712).

**Problem: Cannot input Chinese characters (fcitx) on Linux**

Input methods like fcitx may not work correctly with the Qt6 application.

- Try setting `QT_IM_MODULE=fcitx` before launching GPT4All:

```bash
QT_IM_MODULE=fcitx ./GPT4All
```

- Alternatively, try the `ibus` input method.

Related: [#2967](https://github.com/nomic-ai/gpt4all/issues/2967).

**Problem: GUI Warning Issue on startup**

If you see Qt warnings on application startup, these are typically non-fatal and can be ignored. If the application fails to render:

- Ensure your display drivers are current.
- Try running with `QT_QPA_PLATFORM=xcb` (Linux) to force X11 rendering.

Related: [#3644](https://github.com/nomic-ai/gpt4all/issues/3644).

### 6.6 LocalDocs Issues

**Problem: LocalDocs indexing takes very long**

- Embedding runs locally on your hardware. Large collections (thousands of documents) can take hours on CPU.
- Enable GPU acceleration to speed up embedding.
- Start with a smaller, focused collection.

**Problem: LocalDocs retrieval returns irrelevant results**

- Ensure your question is specific enough to match document content.
- Check that the documents were indexed correctly (re-index the collection).
- Use `.txt` or `.md` files for best results — the parser handles these most reliably.

**Problem: File type not supported in LocalDocs**

Currently supported: `.txt`, `.md`, `.rst`, `.xlsx`. OpenDocument formats (`.odt`, `.ods`, `.odp`) and PDF are not supported for LocalDocs indexing.

Related: [#3477](https://github.com/nomic-ai/gpt4all/issues/3477).

### 6.7 Python SDK Issues

**Problem: PyPI package appears out of date**

The PyPI release may lag behind the GitHub source. Check the latest version at [pypi.org/project/gpt4all](https://pypi.org/project/gpt4all/).

Related: [#3650](https://github.com/nomic-ai/gpt4all/issues/3650).

**Problem: Chat session context not persisting**

Ensure you are using the `chat_session()` context manager:

```python
# WRONG — no session, each call is independent
model.generate("My name is Alice.")
model.generate("What is my name?")  # Won't remember

# RIGHT — session preserves context
with model.chat_session():
    model.generate("My name is Alice.")
    model.generate("What is my name?")  # Remembers "Alice"
```

---

## 7. FAQ

**Q: Is GPT4All truly offline? Does it phone home?**

Yes, all inference runs locally on your hardware. The only network calls are for downloading models and checking the model list (`models3.json`). Once models are downloaded, you can disconnect from the internet entirely.

**Q: What hardware do I need?**

At minimum, 8 GB of RAM for small models (1B–3B parameters). 16 GB is recommended for 7B–8B parameter models. A dedicated GPU (NVIDIA, AMD, or Intel) significantly speeds up inference but is not required — CPU inference works on any modern x86 or ARM processor.

**Q: Which GPU backends are supported?**

GPT4All supports Vulkan (cross-platform), CUDA (NVIDIA), and Kompute (experimental). The backend is selected at compile time and auto-detected at runtime.

**Q: Can I use models from Hugging Face?**

Yes, any GGUF-format model file can be used. Download the `.gguf` file, place it in your model directory, and load it by filename. Ensure the model's architecture is in the supported list.

**Q: What is the maximum GGUF version supported?**

GGUF version 3. Files created with newer GGUF versions may fail to load.

**Q: Can I use GPT4All as a drop-in replacement for OpenAI's API?**

Partially. The built-in API server implements `/v1/completions` and `/v1/chat/completions` with basic parameters (model, max_tokens, temperature, top_p, min_p, n). It does not support streaming, function calling, tools, logprobs, or response format constraints.

**Q: How do I use GPT4All with my own documents?**

Use the LocalDocs feature in the desktop application. Create a collection pointing to a folder of `.txt`, `.md`, `.rst`, or `.xlsx` files. GPT4All embeds and indexes them locally, then uses RAG to ground responses in your documents.

**Q: Can I run multiple models simultaneously?**

The desktop application is single-instance (enforced by SingleApplication). You can switch between models but not run two in parallel within the app. However, you can run the Python or Node.js SDK in separate processes to use multiple models concurrently, memory permitting.

**Q: Where are my chat histories stored?**

Chat histories are stored in the application data directory alongside the settings file. The serialization format uses a custom binary format (version 12, magic: `0xF5D553CC`).

**Q: How do I update GPT4All?**

Download the latest installer from [gpt4all.io](https://gpt4all.io). Your models and settings are preserved across updates. For the Python SDK, run `pip install --upgrade gpt4all`. For Node.js, run `npm update gpt4all`.

**Q: Does GPT4All support fine-tuning?**

The repository includes training scripts for fine-tuning with DeepSpeed, LoRA, PEFT, and Accelerate. These are separate from the inference engine and require a GPU with sufficient VRAM. See the `train/` directory in the source repository for scripts and configuration.

**Q: What file types can I attach in chat?**

The desktop application supports attaching `.txt`, `.md`, `.rst`, and `.xlsx` files directly to chat messages.

**Q: Is MCP Server support available?**

Not currently. This has been requested as a feature enhancement.

Related: [#3546](https://github.com/nomic-ai/gpt4all/issues/3546).

**Q: Are Qwen3 models supported?**

Qwen2 and Qwen2 MoE architectures are supported. Qwen3 support has been requested but is not yet available.

Related: [#3576](https://github.com/nomic-ai/gpt4all/issues/3576).

**Q: What is the minimum context size?**

The minimum context size is 8 tokens. In practice, you should use at least 512–2048 tokens for meaningful conversations.

**Q: Can I access GPT4All's API from another machine on my network?**

No. The built-in API server binds exclusively to localhost. If you need network access, place a reverse proxy (like nginx) in front of it.

> [!TIP]
> If you need to expose the API over a network, set up an SSH tunnel or use a reverse proxy with authentication. Never expose the raw API server to the internet.

**Q: What is the difference between `generate()` and `chat_completion()` in Python?**

`generate()` takes a single string prompt and returns a string. It supports streaming via `streaming=True`. `chat_completion()` takes a list of message dicts (with `role` and `content` keys) and returns a structured response object matching the OpenAI format. Use `generate()` for simple prompts and `chat_completion()` when you need multi-turn conversations or system prompts.

**Q: How do I enable verbose logging for debugging?**

Set the `GPT4ALL_VERBOSE_LLAMACPP` environment variable before launching GPT4All:

```bash
# Linux/macOS
export GPT4ALL_VERBOSE_LLAMACPP=1
./GPT4All

# Windows (PowerShell)
$env:GPT4ALL_VERBOSE_LLAMACPP = "1"
.\GPT4All.exe
```

> [!NOTE]
> Verbose logging produces substantial output. Use it for debugging only, not during normal operation.

---

_Generated May 2026 from nomic-ai/gpt4all at b666d16. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._