GPT4All User Guide

GPT4All is an open-source ecosystem by Nomic AI that lets you run large language models entirely on your own hardware — no cloud, no API keys, no data leaving your machine. It ships as a polished desktop application for Windows, macOS, and Linux, plus SDKs for Python and Node.js, and a terminal-based CLI. Whether you want a private ChatGPT replacement or need to embed local LLM inference into your application, GPT4All delivers production-ready tooling backed by the llama.cpp engine.

1. Quick Start

Get up and running in under five minutes with the desktop application.

Step 1 — Download the installer from gpt4all.io for your operating system.

Step 2 — Install and launch. Run the installer; GPT4All opens as a single-instance application (only one window at a time).

Step 3 — Download a model. On first launch, GPT4All presents the model gallery. Pick a model that fits your hardware:

Step 4 — Start chatting. Select the downloaded model, type a message, and press Enter.

Tip

If you have a dedicated GPU (NVIDIA, AMD, or Intel), GPT4All will automatically detect and use it for faster inference. You can verify GPU usage in the application settings.

2. Installation

2.1 Desktop Application

Windows

1. Download the .exe installer from gpt4all.io.
2. Run the installer and follow the prompts.
3. Launch GPT4All from the Start Menu or desktop shortcut.

macOS

1. Download the .dmg from gpt4all.io.
2. Open the .dmg and drag GPT4All to Applications.
3. On first launch, right-click and select Open to bypass Gatekeeper.

Linux

1. Download the .run installer from gpt4all.io.
2. Make it executable and run:

chmod +x gpt4all-installer-linux-vX.Y.Z.run
./gpt4all-installer-linux-vX.Y.Z.run

3. Alternatively, install via Flatpak:

flatpak install flathub io.gpt4all.gpt4all
flatpak run io.gpt4all.gpt4all

Note

The Linux .run installer may not create a desktop shortcut automatically on some distributions (see #3640, #3456). You can launch GPT4All directly from the installation directory or create a .desktop file manually.

2.2 Python SDK

Requires Python 3.8 or higher.

pip install gpt4all

For NVIDIA GPU acceleration:

pip install gpt4all[cuda]

This installs nvidia-cuda-runtime-cu11 and nvidia-cublas-cu11 as additional dependencies.

Important

The [cuda] extra targets CUDA 11. If your system has only CUDA 12 installed, you may encounter library loading errors (see #3033). Install the CUDA 11 compatibility libraries or use CPU inference.

Verify the installation:

from gpt4all import GPT4All
model = GPT4All("orca-mini-3b-gguf2-q4_0.gguf")
print(model.generate("Hello, world!", max_tokens=50))

2.3 Node.js SDK

Requires Node.js 18 or higher.

npm install gpt4all

The package includes native bindings built with node-addon-api and node-gyp-build. On first use, it downloads the llama.cpp backend library automatically.

Verify the installation:

const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("orca-mini-3b-gguf2-q4_0.gguf");
    const result = await createCompletion(model, [
        { role: "user", content: "Hello!" }
    ]);
    console.log(result.choices[0].message.content);
    model.dispose();
}
main();

2.4 CLI

The CLI is bundled with the Python SDK. After installing the Python package:

python -m gpt4all repl

Or check the version:

python -m gpt4all version

3. Configuration

3.1 Desktop Application Settings

GPT4All stores settings via Qt's QSettings system in INI format. The settings file location varies by OS:

Key configurable settings include:

Available interface languages: English (US), Spanish (Mexico), Italian (Italy), Portuguese (Brazil), Romanian (Romania), Chinese Simplified, Chinese Traditional.

3.2 Model Storage

All GGUF model files are stored in a single, user-configurable directory. Models are downloaded from https://gpt4all.io/models/gguf/ and model metadata is fetched from https://gpt4all.io/models/models3.json.

To change the model directory in the desktop app, go to Settings and update the model path. All SDKs also accept a custom model path.

3.3 Python SDK Configuration

The GPT4All constructor accepts configuration at instantiation:

from gpt4all import GPT4All

model = GPT4All(
    model_name="mistral-7b-instruct-v0.1.Q4_0.gguf",
    device="gpu"           # "cpu", "gpu", "amd", "nvidia", "intel"
)

Models are cached in ~/.cache/gpt4all/ by default. To use a different directory:

model = GPT4All(
    model_name="mistral-7b-instruct-v0.1.Q4_0.gguf",
    model_path="/path/to/your/models/"
)

3.4 Node.js SDK Configuration

const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf", {
    modelPath: "/path/to/models",    // Custom model directory
    allowDownload: true,              // Auto-download if missing (default: true)
    verbose: false,                   // Verbose logging
    device: "gpu",                    // "cpu", "gpu", "amd", "nvidia", "intel"
    nCtx: 2048,                       // Context window size (default: 2048)
    ngl: 100                          // Number of GPU layers (default: 100)
});

3.5 Environment Variables

Tip

Enable GPT4ALL_VERBOSE_LLAMACPP when debugging model loading issues — it prints detailed information about GGUF file parsing, tensor allocation, and GPU memory mapping.

3.6 Remote / API Model Configuration

GPT4All's desktop app supports connecting to external OpenAI-compatible endpoints (OpenAI, Groq, Mistral, etc.) via the ChatAPI class. Configure these in the application settings by providing the API endpoint URL and your API key. This lets you use cloud models alongside local ones in the same interface.

4. Core Usage

4.1 Desktop Application

Starting a Conversation

1. Select a model from the model dropdown at the top of the chat window.
2. Type your message and press Enter or click Send.
3. The model generates a response locally on your hardware.

GPT4All enforces a single-instance policy — attempting to launch a second instance activates the existing window instead.

Managing Models

The model gallery lists all available models with their sizes, descriptions, and hardware requirements. Models use the GGUF format and are sourced from the GPT4All model repository.

Popular models include:

To download a model, click the download icon next to it in the gallery. To remove a model, delete its .gguf file from the model directory.

Supported Model Architectures

The llama.cpp backend supports a wide range of architectures:

• General: LLaMA, Falcon, GPT-2, GPT-NeoX, MPT, Bloom, StableLM
• Instruction-tuned: Qwen, Qwen2, Qwen2 MoE, Phi-2, Phi-3, Gemma, Gemma 2, InternLM2, Command-R
• Code: StarCoder, StarCoder 2, CodeShell, Refact
• Specialized: Baichuan, Granite, GraniteMoE, JAIS, ChatGLM, Orion, Xverse, OLMo, OLMoE, OpenELM, DeepSeek2
• Embedding: BERT, Nomic-BERT

Maximum supported GGUF version: 3.

LocalDocs (RAG)

LocalDocs lets you ground model responses in your own documents. The model retrieves relevant snippets from your files and includes them as context.

Setting up a LocalDocs collection:

1. Go to LocalDocs in the settings or sidebar.
2. Click Add Collection and select a folder containing your documents.
3. GPT4All indexes the folder contents. Supported file types:
   • .txt — Plain text
   • .md — Markdown
   • .rst — reStructuredText
   • .xlsx — Excel spreadsheets
4. Enable the collection in your chat to start asking questions grounded in your documents.

Note

LocalDocs embedding runs locally. Large document collections may take significant time to index on first setup. There is currently no visual progress indicator for the embedding process (#2579). OpenDocument formats (.odt, .ods) are not yet supported (#3477).

File Attachments

You can attach files directly to a chat message for the model to read and respond to. Supported formats:

• .txt — Plain text files
• .md — Markdown files
• .rst — reStructuredText files
• .xlsx — Excel spreadsheets

Click the attachment icon in the chat input area to select a file.

Code Interpreter

GPT4All includes a built-in JavaScript code interpreter tool. When enabled, the model can write and execute JavaScript code to answer questions that require computation, data processing, or logic.

The interpreter enforces a maximum of 3 consecutive tool calls per turn to prevent runaway execution.

Built-in API Server

GPT4All includes an OpenAI-compatible API server that runs on localhost. Enable it in settings and configure the port.

Available endpoints:

Request parameters:

Example — curl request to the local API:

curl http://localhost:4891/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-7b-instruct-v0.1.Q4_0.gguf",
    "messages": [{"role": "user", "content": "Explain quantum computing briefly."}],
    "max_tokens": 200,
    "temperature": 0.7
  }'

Warning

The API server binds to localhost only — it is not accessible from other machines on your network. The following OpenAI API features are not supported: stream, frequency_penalty, presence_penalty, seed, stop, logit_bias, logprobs, tools, tool_choice, functions, and response_format. Requests using these parameters will be ignored or may produce errors.

Chat History

Conversations are automatically saved and can be resumed. The chat serialization format uses version 12 with magic number 0xF5D553CC. Chat history is stored locally alongside the application data.

4.2 Python SDK

Basic Text Generation

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")
response = model.generate("What is the capital of France?", max_tokens=100)
print(response)

Chat Completion (Messages Format)

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain photosynthesis in simple terms."}
]
response = model.chat_completion(messages)
print(response["choices"][0]["message"]["content"])

Chat Sessions (Multi-turn Conversations)

Use the chat session context manager to maintain conversation history across multiple exchanges:

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

with model.chat_session():
    response1 = model.generate("My name is Alice.")
    print(response1)

    response2 = model.generate("What is my name?")
    print(response2)  # The model remembers "Alice"

Streaming Responses

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

for token in model.generate("Write a haiku about coding.", streaming=True):
    print(token, end="", flush=True)
print()

Generation Parameters

GPU Inference

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

Valid device values: "cpu", "gpu", "amd", "nvidia", "intel".

4.3 Node.js SDK

Loading a Model

const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf", {
        device: "gpu",
        nCtx: 4096,
        ngl: 100
    });

    // Use the model...

    model.dispose();  // Always dispose when done
}

Chat Completion

const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const result = await createCompletion(model, [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "What is machine learning?" }
    ]);

    console.log(result.choices[0].message.content);
    model.dispose();
}
main();

Streaming Responses

const { loadModel, createCompletionStream } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const stream = createCompletionStream(model, [
        { role: "user", content: "Tell me a story." }
    ]);

    stream.tokens.on("data", (token) => {
        process.stdout.write(token);
    });

    await stream.result;
    console.log("\nDone.");
    model.dispose();
}
main();

Async Generator

const { loadModel, createCompletionGenerator } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const generator = createCompletionGenerator(model, [
        { role: "user", content: "List 5 programming languages." }
    ]);

    for await (const chunk of generator) {
        process.stdout.write(chunk);
    }

    model.dispose();
}
main();

Embeddings

const { loadModel, createEmbedding } = require("gpt4all");

async function main() {
    const model = await loadModel("nomic-embed-text-v1.5.f16.gguf");

    const embedding = await createEmbedding(model, "Hello, world!", {
        prefix: "search_document",
        dimensionality: 768
    });

    console.log(embedding.embeddings[0].length);  // 768
    model.dispose();
}
main();

Embedding prefix options (for Nomic Embed models):

Long text handling (longTextMode):

Chat Sessions (Conversation Memory)

const { loadModel, createCompletion } = require("gpt4all");

async function main() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");

    const session = await model.createChatSession({
        systemPrompt: "You are a Python expert."
    });

    const r1 = await createCompletion(session, [
        { role: "user", content: "What is a decorator?" }
    ]);
    console.log(r1.choices[0].message.content);

    const r2 = await createCompletion(session, [
        { role: "user", content: "Give me an example." }
    ]);
    console.log(r2.choices[0].message.content);

    model.dispose();
}
main();

Downloading Models Programmatically

const { downloadModel } = require("gpt4all");

const controller = downloadModel("mistral-7b-instruct-v0.1.Q4_0.gguf", {
    modelPath: "./models"
});

controller.promise.then(() => {
    console.log("Download complete!");
}).catch((err) => {
    console.error("Download failed:", err);
});

// Cancel if needed:
// controller.cancel();

Downloads support automatic resume — if a partial file exists, the download continues from where it left off.

Default Prompt Parameters

The Node.js SDK uses these defaults when no overrides are provided:

Default prompt template: ### Human:\n%1\n\n### Assistant:\n

4.4 CLI

Starting the REPL

python -m gpt4all repl

With options:

python -m gpt4all repl --model mistral-7b-instruct-v0.1.Q4_0.gguf --device gpu --n-threads 8

REPL Commands

The CLI uses streaming output by default, printing tokens as they are generated with these default parameters: max_tokens=200, temp=0.9, top_k=40, top_p=0.9, min_p=0, repeat_penalty=1.1, repeat_last_n=64, n_batch=9.

5. Common Tasks

Recipe 1: Private Document Q&A

Use LocalDocs in the desktop app to query your own documents without sending data to the cloud.

1. Open GPT4All and go to LocalDocs.
2. Create a collection pointing to a folder with your .txt, .md, .rst, or .xlsx files.
3. Wait for indexing to complete.
4. Start a new chat, enable the collection, and ask questions about your documents.

Recipe 2: Replace OpenAI API with Local Inference

Point any OpenAI-compatible client at GPT4All's built-in API server.

1. Enable the API server in GPT4All settings and note the port (e.g., 4891).
2. Configure your application:

import openai

client = openai.OpenAI(
    base_url="http://localhost:4891/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="mistral-7b-instruct-v0.1.Q4_0.gguf",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=100
)
print(response.choices[0].message.content)

Caution

Remember that streaming, function calling, and several other OpenAI API features are not implemented in GPT4All's server. Test your integration thoroughly before switching from a cloud API.

Recipe 3: Batch Processing with Python

Process multiple prompts efficiently using the Python SDK:

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

prompts = [
    "Summarize the benefits of exercise.",
    "Explain how a compiler works.",
    "Write a Python function to reverse a string."
]

for prompt in prompts:
    response = model.generate(prompt, max_tokens=200, temp=0.7)
    print(f"Q: {prompt}\nA: {response}\n{'='*60}\n")

Recipe 4: Building a Chatbot with Node.js

const { loadModel, createCompletion } = require("gpt4all");
const readline = require("readline");

async function chatbot() {
    const model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");
    const session = await model.createChatSession({
        systemPrompt: "You are a friendly assistant."
    });

    const rl = readline.createInterface({
        input: process.stdin,
        output: process.stdout
    });

    function ask() {
        rl.question("You: ", async (input) => {
            if (input.toLowerCase() === "exit") {
                model.dispose();
                rl.close();
                return;
            }
            const result = await createCompletion(session, [
                { role: "user", content: input }
            ]);
            console.log(`Bot: ${result.choices[0].message.content}\n`);
            ask();
        });
    }
    ask();
}
chatbot();

Recipe 5: Semantic Search with Embeddings

Generate embeddings for a document corpus using the Node.js SDK:

const { loadModel, createEmbedding } = require("gpt4all");

async function buildIndex(documents) {
    const model = await loadModel("nomic-embed-text-v1.5.f16.gguf");

    const embeddings = [];
    for (const doc of documents) {
        const result = await createEmbedding(model, doc, {
            prefix: "search_document",
            dimensionality: 768
        });
        embeddings.push(result.embeddings[0]);
    }

    model.dispose();
    return embeddings;
}

async function search(query, documents, index) {
    const model = await loadModel("nomic-embed-text-v1.5.f16.gguf");

    const queryEmb = await createEmbedding(model, query, {
        prefix: "search_query",
        dimensionality: 768
    });

    // Compute cosine similarity (simplified)
    const scores = index.map((docEmb, i) => {
        const dot = docEmb.reduce((sum, v, j) => sum + v * queryEmb.embeddings[0][j], 0);
        return { index: i, score: dot, text: documents[i] };
    });

    model.dispose();
    return scores.sort((a, b) => b.score - a.score);
}

Recipe 6: GPU Selection for Multi-GPU Systems

from gpt4all import GPT4All

# Use any available GPU
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

# Prefer NVIDIA
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="nvidia")

# Prefer AMD
model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="amd")

Recipe 7: Adjusting Generation Quality

Control creativity and consistency through temperature and sampling parameters:

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

# Creative writing — high temperature, broad sampling
creative = model.generate(
    "Write a poem about the ocean.",
    max_tokens=300, temp=1.2, top_k=80, top_p=0.95
)

# Factual answer — low temperature, narrow sampling
factual = model.generate(
    "What year was Python created?",
    max_tokens=50, temp=0.1, top_k=10, top_p=0.5
)

# Code generation — moderate settings with repeat penalty
code = model.generate(
    "Write a Python quicksort function.",
    max_tokens=400, temp=0.3, repeat_penalty=1.2
)

Recipe 8: Using Custom GGUF Models

Any GGUF model from Hugging Face or other sources can be used:

1. Download the .gguf file to your model directory (~/.cache/gpt4all/ by default).
2. Load it by filename:

from gpt4all import GPT4All

model = GPT4All("my-custom-model.Q4_K_M.gguf")
response = model.generate("Hello!", max_tokens=100)

Important

Only GGUF format (up to version 3) is supported. Older GGML format files will not load. Convert legacy models using the convert scripts in the llama.cpp repository.

Recipe 9: System Prompts for Specialized Behavior

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf")

messages = [
    {"role": "system", "content": "You are a SQL expert. Always respond with valid SQL queries and brief explanations."},
    {"role": "user", "content": "Find all users who signed up in the last 30 days."}
]

response = model.chat_completion(messages)
print(response["choices"][0]["message"]["content"])

Recipe 10: Resumable Model Downloads (Node.js)

If a download is interrupted, it resumes automatically on the next attempt:

const { downloadModel } = require("gpt4all");

const controller = downloadModel("llama-3-8b-instruct.Q4_0.gguf");

// Monitor progress
controller.promise
    .then(() => console.log("Download complete!"))
    .catch((err) => {
        if (err.message.includes("cancel")) {
            console.log("Download cancelled. Run again to resume.");
        } else {
            console.error("Error:", err);
        }
    });

// Cancel if needed (file is preserved for resume)
setTimeout(() => controller.cancel(), 30000);

Recipe 11: Integrating with Express.js

Wrap GPT4All in a simple REST API:

const express = require("express");
const { loadModel, createCompletion } = require("gpt4all");

const app = express();
app.use(express.json());

let model;

app.post("/chat", async (req, res) => {
    if (!model) {
        model = await loadModel("mistral-7b-instruct-v0.1.Q4_0.gguf");
    }

    const result = await createCompletion(model, req.body.messages || [
        { role: "user", content: req.body.prompt || "Hello" }
    ]);

    res.json(result);
});

app.listen(3000, () => console.log("Server on http://localhost:3000"));

Recipe 12: Multi-turn CLI Conversation Script

from gpt4all import GPT4All

model = GPT4All("mistral-7b-instruct-v0.1.Q4_0.gguf", device="gpu")

with model.chat_session():
    while True:
        user_input = input("\nYou: ").strip()
        if user_input.lower() in ("exit", "quit"):
            break
        response = model.generate(user_input, max_tokens=300, temp=0.7)
        print(f"\nAssistant: {response}")

6. Troubleshooting

6.1 Installation Issues

Problem: Linux installer does not run or create a desktop shortcut

The .run installer may fail silently on some Linux distributions, particularly those without certain GUI libraries.

• Ensure libxcb, libGL, and libfontconfig are installed.
• Run the installer from a terminal to see error output.
• If the installer succeeds but no shortcut appears, create a .desktop file manually pointing to the GPT4All binary.
• Consider using the Flatpak package as an alternative: flatpak install flathub io.gpt4all.gpt4all.

Related: #3640, #3456.

Problem: Application fails to start after installation

• On Windows, run as administrator or check that your antivirus is not blocking the executable.
• On macOS, right-click the app and choose Open to bypass Gatekeeper on first launch.
• On Linux, check for missing shared libraries with ldd on the GPT4All binary.

Related: #3456.

Problem: Python SDK installation fails on pip

pip install --upgrade pip setuptools wheel
pip install gpt4all

If building native extensions fails, ensure you have a C++ compiler installed (MSVC on Windows, gcc/clang on Linux/macOS).

Problem: Node.js SDK node-gyp build errors

Ensure you have the native build tools:

Windows

npm install -g windows-build-tools

macOS

xcode-select --install

Linux

sudo apt install build-essential python3

6.2 Model Loading Issues

Problem: Model download stalls or fails

• Check your internet connection and available disk space.
• Models are downloaded from https://gpt4all.io/models/gguf/ — ensure this domain is not blocked by firewall or proxy.
• Delete any partial .part file in the model directory and retry.

Problem: "Unsupported GGUF version" error

GPT4All supports GGUF versions up to 3. If you downloaded a model with a newer GGUF version, you need to either update GPT4All or find a compatible quantization of the model.

Problem: Model loads but produces garbage output

• Verify the model file is not corrupted (re-download it).
• Some models require specific prompt templates. Check the model card for the correct chat format.
• Reduce the context length if you are running low on RAM — an insufficient allocation can cause buffer corruption.

Problem: "Could not find model" error in Python SDK

The SDK looks for models in ~/.cache/gpt4all/ by default. Either:

1. Place the .gguf file in that directory, or
2. Specify model_path explicitly:

model = GPT4All("my-model.gguf", model_path="/path/to/models/")

6.3 GPU and Performance Issues

Problem: Vulkan error — vk::Device::createFence: ErrorInitializationFailed

This occurs when the Vulkan driver is outdated or incompatible.

• Update your GPU drivers to the latest version.
• If using an integrated GPU, try forcing CPU mode or switching to a discrete GPU.
• On Linux, ensure the Vulkan ICD loader is correctly installed (vulkan-tools package provides vulkaninfo for diagnosis).

Related: #3259.

Problem: CUDA 12 installed but GPT4All reports missing CUDA 11

The Python SDK's [cuda] extra installs CUDA 11 runtime libraries. If your system has only CUDA 12:

• Install the CUDA 11 compatibility libraries alongside CUDA 12.
• Or use CPU inference by not installing the [cuda] extra.
• Or use the desktop application, which handles CUDA detection automatically.

Related: #3033.

Problem: GPU not detected or inference falls back to CPU

• Verify GPU support with GPT4ALL_VERBOSE_LLAMACPP=1 to see backend selection logs.
• Ensure your GPU drivers are up to date.
• Check that the correct device string is used: "gpu", "nvidia", "amd", or "intel".
• On desktop, check Settings to confirm GPU acceleration is enabled.

Related: #1676 (Intel GPU support).

Problem: Out of memory (OOM) when loading large models

• Choose a smaller quantization (Q4_0 is smallest, Q8_0 is largest per parameter).
• Reduce nCtx (context window) — each token in context consumes memory.
• Close other applications to free RAM/VRAM.
• Consider a smaller model: 1B–3B parameter models run comfortably on 8 GB RAM.

Problem: Inference is very slow

• Enable GPU acceleration if not already active.
• Reduce max_tokens to limit generation length.
• Use a smaller model for faster inference.
• On CPU, increase the number of threads with --n-threads (CLI) or n_threads parameter.
• Ensure memory-mapped I/O is working (GPT4All uses mmap by default).

6.4 API Server Issues

Problem: Cannot connect to the built-in API server

• Verify the server is enabled in GPT4All settings.
• Check the port number — use curl http://localhost:<port>/v1/models to test.
• The server binds to localhost only; it is inaccessible from other machines by design.

Problem: TypeError: 'OpenAIResponse' object is not iterable with stream=True

Streaming is not implemented in the GPT4All API server. Remove stream=True from your request or set it to false.

Related: #704.

Problem: API returns empty or truncated responses

• Increase max_tokens in your request (default is only 16).
• Ensure the model is fully loaded before sending requests.

6.5 Desktop Application Issues

Problem: chat.exe process continues running after closing the application

On Windows, the chat process may persist in the background after the window is closed.

• Open Task Manager and manually end the chat.exe process.
• This is a known issue being tracked.

Related: #3656.

Problem: Cannot resize or maximize the window

This was reported in v3.0.0 and may affect certain window managers on Linux.

• Try toggling fullscreen mode.
• Update to the latest GPT4All release (v3.10.0+).

Related: #2712.

Problem: Cannot input Chinese characters (fcitx) on Linux

Input methods like fcitx may not work correctly with the Qt6 application.

• Try setting QT_IM_MODULE=fcitx before launching GPT4All:

QT_IM_MODULE=fcitx ./GPT4All

• Alternatively, try the ibus input method.

Related: #2967.

Problem: GUI Warning Issue on startup

If you see Qt warnings on application startup, these are typically non-fatal and can be ignored. If the application fails to render:

• Ensure your display drivers are current.
• Try running with QT_QPA_PLATFORM=xcb (Linux) to force X11 rendering.

Related: #3644.

6.6 LocalDocs Issues

Problem: LocalDocs indexing takes very long

• Embedding runs locally on your hardware. Large collections (thousands of documents) can take hours on CPU.
• Enable GPU acceleration to speed up embedding.
• Start with a smaller, focused collection.

Problem: LocalDocs retrieval returns irrelevant results

• Ensure your question is specific enough to match document content.
• Check that the documents were indexed correctly (re-index the collection).
• Use .txt or .md files for best results — the parser handles these most reliably.

Problem: File type not supported in LocalDocs

Currently supported: .txt, .md, .rst, .xlsx. OpenDocument formats (.odt, .ods, .odp) and PDF are not supported for LocalDocs indexing.

Related: #3477.

6.7 Python SDK Issues

Problem: PyPI package appears out of date

The PyPI release may lag behind the GitHub source. Check the latest version at pypi.org/project/gpt4all.

Related: #3650.

Problem: Chat session context not persisting

Ensure you are using the chat_session() context manager:

# WRONG — no session, each call is independent
model.generate("My name is Alice.")
model.generate("What is my name?")  # Won't remember

# RIGHT — session preserves context
with model.chat_session():
    model.generate("My name is Alice.")
    model.generate("What is my name?")  # Remembers "Alice"

7. FAQ

Q: Is GPT4All truly offline? Does it phone home?

Yes, all inference runs locally on your hardware. The only network calls are for downloading models and checking the model list (models3.json). Once models are downloaded, you can disconnect from the internet entirely.

Q: What hardware do I need?

At minimum, 8 GB of RAM for small models (1B–3B parameters). 16 GB is recommended for 7B–8B parameter models. A dedicated GPU (NVIDIA, AMD, or Intel) significantly speeds up inference but is not required — CPU inference works on any modern x86 or ARM processor.

Q: Which GPU backends are supported?

GPT4All supports Vulkan (cross-platform), CUDA (NVIDIA), and Kompute (experimental). The backend is selected at compile time and auto-detected at runtime.

Q: Can I use models from Hugging Face?

Yes, any GGUF-format model file can be used. Download the .gguf file, place it in your model directory, and load it by filename. Ensure the model's architecture is in the supported list.

Q: What is the maximum GGUF version supported?

GGUF version 3. Files created with newer GGUF versions may fail to load.

Q: Can I use GPT4All as a drop-in replacement for OpenAI's API?

Partially. The built-in API server implements /v1/completions and /v1/chat/completions with basic parameters (model, max_tokens, temperature, top_p, min_p, n). It does not support streaming, function calling, tools, logprobs, or response format constraints.

Q: How do I use GPT4All with my own documents?

Use the LocalDocs feature in the desktop application. Create a collection pointing to a folder of .txt, .md, .rst, or .xlsx files. GPT4All embeds and indexes them locally, then uses RAG to ground responses in your documents.

Q: Can I run multiple models simultaneously?

The desktop application is single-instance (enforced by SingleApplication). You can switch between models but not run two in parallel within the app. However, you can run the Python or Node.js SDK in separate processes to use multiple models concurrently, memory permitting.

Q: Where are my chat histories stored?

Chat histories are stored in the application data directory alongside the settings file. The serialization format uses a custom binary format (version 12, magic: 0xF5D553CC).

Q: How do I update GPT4All?

Download the latest installer from gpt4all.io. Your models and settings are preserved across updates. For the Python SDK, run pip install --upgrade gpt4all. For Node.js, run npm update gpt4all.

Q: Does GPT4All support fine-tuning?

The repository includes training scripts for fine-tuning with DeepSpeed, LoRA, PEFT, and Accelerate. These are separate from the inference engine and require a GPU with sufficient VRAM. See the train/ directory in the source repository for scripts and configuration.

Q: What file types can I attach in chat?

The desktop application supports attaching .txt, .md, .rst, and .xlsx files directly to chat messages.

Q: Is MCP Server support available?

Not currently. This has been requested as a feature enhancement.

Related: #3546.

Q: Are Qwen3 models supported?

Qwen2 and Qwen2 MoE architectures are supported. Qwen3 support has been requested but is not yet available.

Related: #3576.

Q: What is the minimum context size?

The minimum context size is 8 tokens. In practice, you should use at least 512–2048 tokens for meaningful conversations.

Q: Can I access GPT4All's API from another machine on my network?

No. The built-in API server binds exclusively to localhost. If you need network access, place a reverse proxy (like nginx) in front of it.

Tip

If you need to expose the API over a network, set up an SSH tunnel or use a reverse proxy with authentication. Never expose the raw API server to the internet.

Q: What is the difference between generate() and chat_completion() in Python?

generate() takes a single string prompt and returns a string. It supports streaming via streaming=True. chat_completion() takes a list of message dicts (with role and content keys) and returns a structured response object matching the OpenAI format. Use generate() for simple prompts and chat_completion() when you need multi-turn conversations or system prompts.

Q: How do I enable verbose logging for debugging?

Set the GPT4ALL_VERBOSE_LLAMACPP environment variable before launching GPT4All:

# Linux/macOS
export GPT4ALL_VERBOSE_LLAMACPP=1
./GPT4All

# Windows (PowerShell)
$env:GPT4ALL_VERBOSE_LLAMACPP = "1"
.\GPT4All.exe

Note

Verbose logging produces substantial output. Use it for debugging only, not during normal operation.