# GPT4All Administrator & Developer Guide

## 1. Architecture Overview

GPT4All is an open-source ecosystem by Nomic AI for running large language models entirely on local hardware. The project spans six major components — a desktop application, a C++ inference backend, Python and Node.js SDKs, a command-line interface, and a training pipeline — unified by a shared backend engine built on llama.cpp.

### 1.1 Component Map

| Component | Path | Language | Role |
|-----------|------|----------|------|
| Desktop App | `gpt4all-chat/` | C++ (Qt6/QML) | Cross-platform GUI chat application |
| Backend Engine | `gpt4all-backend/` | C++17 | LLM inference engine wrapping llama.cpp |
| Python SDK | `gpt4all-bindings/python/` | Python | Programmatic LLM access via ctypes FFI |
| Node.js SDK | `gpt4all-bindings/typescript/` | C++/JS (N-API) | Native Node.js addon for LLM inference |
| CLI | `gpt4all-bindings/cli/` | Python (Typer) | Terminal REPL for interactive use |
| Training | `gpt4all-training/` | Python | Fine-tuning pipeline (DeepSpeed, LoRA, PEFT) |

### 1.2 Technology Stack

- **Desktop**: Qt 6, QML for UI, SingleApplication for single-instance enforcement, CMake build system
- **Backend**: C++17, llama.cpp (GGUF model format), ggml tensor library for compute
- **GPU Acceleration**: Vulkan (via ggml-vulkan), CUDA (via ggml-cuda), Kompute (via ggml-kompute)
- **Python SDK**: ctypes FFI binding to the `llmodel` C API, Jinja2 for prompt templates
- **Node.js SDK**: N-API via `node-addon-api`, `node-gyp-build` for native compilation
- **Training**: PyTorch, Hugging Face Transformers, DeepSpeed, PEFT/LoRA, Accelerate, Weights & Biases

### 1.3 Runtime Data Flow

The inference pipeline follows a consistent path regardless of entry point (GUI, API, SDK):

```
User Prompt
    │
    ▼
Chat / Server / SDK
    │
    ├── [LocalDocs enabled?] ──▶ Database (SQLite FTS)
    │       │                       │
    │       │                   Retrieve chunks
    │       │                       │
    │       ◀── Prepend context ◄───┘
    │
    ▼
ChatLLM (worker thread)
    │
    ▼
LLModel interface
    │
    ▼
LLamaModel (llama.cpp)
    │
    ├── CPU inference (ggml)
    ├── Vulkan inference (ggml-vulkan)
    └── CUDA inference (ggml-cuda)
    │
    ▼
Response tokens (streamed)
    │
    ├── [Tool call detected?] ──▶ ToolCallParser
    │       │                       │
    │       │                   CodeInterpreter (JS)
    │       │                       │
    │       ◀── Result appended ◄───┘
    │       │
    │       ▼ (re-prompt, max 3 consecutive)
    │
    ▼
Final Response
```

> [!NOTE]
> Tool calls are limited to 3 consecutive invocations per response cycle. The CodeInterpreter executes JavaScript code and returns the result for re-prompting.

### 1.4 Key Classes — Desktop App (`gpt4all-chat/src/`)

| Class | Responsibility |
|-------|---------------|
| `Chat` | Main chat session: model lifecycle, prompt/response flow, tool calls, serialization |
| `ChatLLM` | LLM worker running in a dedicated `QThread`; handles model loading and inference |
| `ChatModel` | `QAbstractListModel` for rendering chat messages in QML |
| `ChatListModel` | Manages the collection of `Chat` instances and handles persistence |
| `Server` | Inherits `ChatLLM`; implements OpenAI-compatible HTTP API via `QHttpServer` |
| `ChatAPI` | Remote API model support (OpenAI, Groq, etc.) implementing the `LLModel` interface |
| `ChatAPIWorker` | Network worker thread for `ChatAPI` remote calls |
| `ModelList` | Global model registry: available, installed, and downloadable models |
| `MySettings` | Global settings singleton backed by `QSettings` |
| `Download` | Model download manager with progress tracking |
| `LocalDocs` | Local document collection manager for retrieval-augmented generation |
| `Database` | SQLite database powering LocalDocs full-text search |
| `EmbLLM` | Embedding model for LocalDocs document indexing |
| `ToolModel` | Tool registry (currently: CodeInterpreter only) |
| `ToolCallParser` | Parses tool call markup from model output tokens |
| `CodeInterpreter` | JavaScript code execution sandbox |
| `Logger` | Application-level logging |
| `Network` | Telemetry and analytics |

### 1.5 Key Classes — Backend (`gpt4all-backend/src/`)

| Class | Responsibility |
|-------|---------------|
| `LLModel` | Pure virtual base interface for all model backends |
| `LLamaModel` | llama.cpp implementation of `LLModel` |
| `Dlhandle` | Cross-platform dynamic library loader (`dlopen` / `LoadLibraryExW`) |

The backend uses a plugin architecture: model implementations are loaded as shared libraries at runtime via `Dlhandle`. The library search path is configured at startup through `LLModel::Implementation::setImplementationsSearchPath()`:

- **macOS**: `{appDir}/../Frameworks`
- **Linux / Windows**: `{appDir}` and `{appDir}/../lib`

### 1.6 Supported Model Architectures

The backend supports these GGUF model architectures (defined in `llamamodel.cpp` as `KNOWN_ARCHES`):

```
llama, falcon, gpt2, gptneox, granite, granitemoe, mpt, baichuan,
starcoder, refact, bert, nomic-bert, bloom, stablelm, qwen, qwen2,
qwen2moe, phi2, phi3, codeshell, orion, internlm2, gemma, gemma2,
starcoder2, xverse, command-r, olmo, olmoe, openelm, deepseek2,
chatglm, jais
```

Embedding-only architectures: `bert`, `nomic-bert`.

> [!WARNING]
> The model `open-orca_mistral-7b-openorca` with `n_vocab=32002` and a dummy token at position 32000 is explicitly blacklisted and will fail to load.

---

## 2. System Requirements

### 2.1 Desktop Application

#### Build Requirements

| Dependency | Minimum Version | Notes |
|------------|----------------|-------|
| CMake | 3.16+ | Build system generator |
| Qt | 6.x | Core, Quick, Qml, HttpServer, Svg, Sql modules |
| C++ Compiler | C++17 support | GCC 9+, Clang 10+, MSVC 2019+ |
| Vulkan SDK | 1.3+ | For GPU acceleration (optional) |
| CUDA Toolkit | 11.x or 12.x | For NVIDIA GPU acceleration (optional) |

#### Runtime Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| RAM | 8 GB | 16+ GB (dependent on model size) |
| Disk | 5 GB + model size | SSD strongly preferred |
| OS | Windows 10+, macOS 12+, Linux (glibc 2.31+) | — |
| GPU (optional) | Vulkan 1.1 capable | NVIDIA with CUDA, or any Vulkan 1.3 GPU |

> [!IMPORTANT]
> Model memory requirements vary dramatically. A 7B parameter model in Q4_0 quantization needs approximately 4 GB of RAM, while a 13B model needs approximately 8 GB. The `GetRequiredMemory` API call can be used to query exact requirements before loading.

### 2.2 Python SDK

- Python 3.8+
- Package: `gpt4all` (version 2.8.3.dev0)
- Ships prebuilt C libraries; no compilation needed for end users
- Platform-specific shared library extensions: `.dylib` (macOS), `.so` (Linux), `.dll` (Windows)

### 2.3 Node.js SDK

- Node.js 16+
- Requires native compilation toolchain:
  - **Linux/macOS**: `build_unix.sh` (GCC/Clang, make)
  - **Windows (MSVC)**: `build_msvc.bat`
  - **Windows (MinGW)**: `build_mingw.ps1`
- Build tool: `node-gyp`

### 2.4 Training Pipeline

- Python 3.8 (pinned in `env.yaml` Conda environment)
- PyTorch with CUDA support
- NVIDIA GPU(s) with sufficient VRAM for target model size
- Key dependencies: `accelerate`, `datasets`, `transformers>=4.28.0`, `wandb`, `peft`, `deepspeed`, `sentencepiece`

---

## 3. Configuration Reference

### 3.1 Desktop Application Settings

All settings are managed through the `MySettings` singleton (backed by `QSettings`). Settings persist to the platform-standard location:

- **Windows**: Registry or `%APPDATA%/nomic.ai/GPT4All/`
- **macOS**: `~/Library/Preferences/io.gpt4all.gpt4all.plist`
- **Linux**: `~/.config/nomic.ai/GPT4All.conf`

#### API Server Settings

| Setting | Method | Default | Description |
|---------|--------|---------|-------------|
| Server enabled | `serverChat()` | `false` | Enable/disable the local API server |
| Port | `networkPort()` | — | TCP port for the API server |

#### Model Settings

Models are downloaded to and loaded from the model path directory. Each model's parameters (temperature, top_p, etc.) are configured through the GUI and persisted in settings.

### 3.2 Environment Variables

| Variable | Scope | Purpose |
|----------|-------|---------|
| `GPT4ALL_VERBOSE_LLAMACPP` | Runtime | Set to any non-empty value to enable verbose llama.cpp debug logging |
| `CHAT_EXECUTABLE` | Testing | Path to the compiled chat executable for API server tests |
| `TEST_MODEL_PATH` | Testing | Path to model file for server integration tests |

> [!TIP]
> When debugging model loading or inference issues, set `GPT4ALL_VERBOSE_LLAMACPP=1` before launching. This enables llama.cpp's internal diagnostics which report tokenizer initialization, context allocation, and per-layer device placement.

### 3.3 Training Configuration

Training runs are configured via YAML files in `gpt4all-training/configs/train/`. Key parameters:

```yaml
# Model
model_name: "EleutherAI/gpt-j-6b"
tokenizer_name: "EleutherAI/gpt-j-6b"
revision: "main"

# Data
dataset_path: "path/to/data.jsonl"
max_length: 1024
streaming: false
num_proc: 8

# Training
batch_size: 8
lr: 2.0e-5
min_lr: 0.0
weight_decay: 0.0
num_epochs: 3
seed: 42
warmup_steps: 100
gradient_checkpointing: true

# LoRA
lora: true
lora_path: null  # path to existing LoRA weights for continued training

# Evaluation & Logging
eval_every: 500
save_every: 1000
log_lr_every: 10
log_grads_every: 100
wandb: true
wandb_project_name: "gpt4all"
wandb_entity: "your-entity"

# Output
output_dir: "./output"
save_name: "gpt4all-lora"
checkpoint: null  # path to resume from
```

#### DeepSpeed Configurations

Pre-built DeepSpeed configs live in `gpt4all-training/configs/deepspeed/` with variants for different model architectures (GPT-J, Pythia, MPT). These configure ZeRO stages, gradient accumulation, mixed precision, and optimizer settings.

#### Training Data Format

Training data must be JSONL with `prompt` and `response` fields:

```jsonl
{"prompt": "What is machine learning?", "response": "Machine learning is a subset of..."}
{"prompt": "Explain gradient descent.", "response": "Gradient descent is an optimization..."}
```

> [!NOTE]
> During tokenization, prompt tokens are masked with label value `-100` so the loss is computed only on response tokens. This ensures the model learns to generate responses, not to memorize prompts.

---

## 4. Deployment & Scaling

### 4.1 Building from Source

#### Backend

```bash
cd gpt4all-backend
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --parallel
```

To enable GPU acceleration, pass the appropriate flags:

```bash
# Vulkan
cmake .. -DCMAKE_BUILD_TYPE=Release -DLLAMA_VULKAN=ON

# CUDA
cmake .. -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
```

#### Desktop Application

```bash
cd gpt4all-chat
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/path/to/Qt/6.x/gcc_64
cmake --build . --parallel
```

#### Platform-Specific Packaging

CMake includes platform-specific deployment scripts in `gpt4all-chat/cmake/`:

| Platform | Script | Output |
|----------|--------|--------|
| Linux | `deploy-qt-linux` | AppImage or tarball |
| macOS | `deploy-qt-mac` + `sign_dmg.py` | Signed DMG |
| Windows | `deploy-qt-windows` | Installer via CPack (`cpack_config.cmake`) |

The macOS signing tool (`sign_dmg.py`) is a Click-based CLI that handles code signing and notarization.

#### Flatpak

A Flatpak manifest is provided as `io.gpt4all.gpt4all` for Linux distribution through Flathub.

### 4.2 Python SDK Build & Distribution

```bash
cd gpt4all-bindings/python

# The setup.py copies prebuilt C libraries from gpt4all-backend/build/
pip install -e .
```

The `setup.py` performs platform detection to select the correct shared library extension (`.dylib`, `.so`, or `.dll`) and copies the prebuilt backend libraries into the package.

### 4.3 Node.js SDK Build

```bash
cd gpt4all-bindings/typescript

# Linux/macOS
./build_unix.sh

# Windows (MSVC)
build_msvc.bat

# Windows (MinGW)
powershell -File build_mingw.ps1

# Install dependencies and build addon
npm install
```

The native addon is compiled with `node-gyp` and loaded at runtime via `node-gyp-build`.

> [!IMPORTANT]
> There is no Docker deployment provided in the project. The desktop application is designed for local installation, and the API server binds to localhost only. Containerized deployment would require modifications to the network binding configuration.

### 4.4 Training Environment Setup

```bash
cd gpt4all-training

# Create Conda environment from specification
conda env create -f env.yaml
conda activate gpt4all

# Launch distributed training
accelerate launch --config_file configs/deepspeed/your_config.yaml \
    train.py --config configs/train/your_training_config.yaml
```

Training supports multi-GPU setups through DeepSpeed and Accelerate. LoRA fine-tuning via PEFT enables training on consumer hardware by training only low-rank adapter matrices.

### 4.5 Scaling Considerations

GPT4All is fundamentally a single-machine, local-inference system. Scaling strategies:

| Strategy | Approach |
|----------|----------|
| **Vertical** | Larger GPU, more RAM, faster SSD for model loading |
| **Multiple instances** | Run multiple API server instances on different ports with different models |
| **Model selection** | Use smaller quantized models (Q4_0, Q4_1) for throughput vs. larger (Q8_0) for quality |
| **Embedding parallelism** | LocalDocs embedding can run concurrently with inference via `EmbLLM` |

---

## 5. Operations

### 5.1 Chat Persistence

Chat sessions are automatically serialized to disk with the following format:

| Field | Detail |
|-------|--------|
| Magic number | `0xF5D553CC` |
| Format version | `12` |
| File pattern | `gpt4all-{chat_id}.chat` |
| Location | Model path directory |
| Serialization | `QDataStream` (creation date, id, name, username, model info, collections, LLM state, chat model) |

Chats are saved automatically on application quit and restored on startup via a dedicated restore thread.

> [!CAUTION]
> Chat files are tied to the format version. If the format version is incremented in a future release, older chat files may not load correctly. Back up the model path directory before upgrading.

### 5.2 Model Management

Models are downloaded from `https://gpt4all.io/models/gguf/` and stored in the user-configured model path. The `ModelList` class maintains a registry of:

- **Available models**: fetched from the remote model list
- **Installed models**: detected on disk in the model path
- **Downloadable models**: available minus installed

The `Download` class handles downloads with progress tracking and resume support.

### 5.3 LocalDocs (RAG)

The LocalDocs feature enables retrieval-augmented generation from local documents:

1. **Indexing**: Documents in configured collections are chunked and embedded via `EmbLLM`
2. **Storage**: Embeddings and text chunks stored in SQLite with FTS (full-text search) via the `Database` class
3. **Retrieval**: At inference time, the prompt is used to query relevant chunks
4. **Augmentation**: Retrieved chunks are prepended to the prompt context before sending to the LLM

Collections are managed through the `LocalDocs` class and configured in the GUI.

### 5.4 Logging

The `Logger` class handles application logging. When `GPT4ALL_VERBOSE_LLAMACPP` is set, additional llama.cpp-level diagnostics are emitted to standard output covering tokenizer initialization, model layer loading, and memory allocation.

### 5.5 Telemetry

The `Network` class handles optional telemetry and analytics. This can be disabled through application settings.

---

## 6. Performance

### 6.1 Threading Model

The desktop application uses a multi-threaded architecture:

| Thread | Purpose |
|--------|---------|
| Main (GUI) thread | Qt event loop, QML rendering |
| `ChatLLM` worker thread | Model loading and inference (one per active chat) |
| `ChatAPIWorker` thread | Network calls for remote API models |
| `EmbLLM` thread | Document embedding for LocalDocs |
| Restore thread | Chat session loading on startup |

> [!TIP]
> The Node.js SDK uses `std::mutex inference_mutex` for thread-safe inference. Only one inference call can execute at a time per model instance. For concurrent inference in Node.js applications, instantiate multiple model objects.

### 6.2 GPU Acceleration

Three GPU backends are available, each with different trade-offs:

| Backend | API | Platforms | Notes |
|---------|-----|-----------|-------|
| ggml-vulkan | Vulkan | Windows, Linux | Broadest GPU support (AMD, NVIDIA, Intel) |
| ggml-cuda | CUDA | Windows, Linux | NVIDIA-only, typically highest throughput |
| ggml-kompute | Kompute/Vulkan | Windows, Linux | Alternative Vulkan path |

GPU device selection is available programmatically:

- **Node.js**: `HasGpuDevice`, `ListGpus`, `InitGpuByString`, `GetGpuDevices`
- **C API**: corresponding functions in `llmodel_c.h`
- **Desktop**: configured through GUI settings

### 6.3 Quantization and Memory

Model memory usage is determined by quantization level and parameter count. Use `GetRequiredMemory` (available in the Node.js SDK and C API) to query exact memory requirements before loading.

| Quantization | Bits/Weight | Relative Size | Quality Impact |
|-------------|-------------|---------------|----------------|
| Q4_0 | 4.0 | Smallest | Noticeable degradation |
| Q4_1 | 4.5 | Small | Slight degradation |
| Q5_0 | 5.0 | Medium | Minimal degradation |
| Q5_1 | 5.5 | Medium | Minimal degradation |
| Q8_0 | 8.0 | Large | Near-original quality |

### 6.4 Context Window

The `max_tokens` parameter in API requests controls generation length (default: 16, minimum: 1). The total context window is determined by the model's architecture and the `n_ctx` parameter at load time.

### 6.5 Inference Tuning Parameters

| Parameter | Range | Default | Effect |
|-----------|-------|---------|--------|
| `temperature` | 0.0–2.0 | 1.0 | Randomness of sampling; 0 = deterministic |
| `top_p` | 0.0–1.0 | 1.0 | Nucleus sampling threshold |
| `min_p` | 0.0–1.0 | 0.0 | Minimum probability threshold |
| `max_tokens` | 1+ | 16 | Maximum tokens to generate |
| `n` | 1+ | 1 | Number of completions to return |

---

## 7. Security

### 7.1 API Server Security Posture

The built-in API server has a minimal security model designed for local use:

| Aspect | Status | Detail |
|--------|--------|--------|
| Network binding | Localhost only | `QHostAddress::LocalHost` — not accessible from the network |
| Authentication | None | No API key, token, or auth mechanism |
| CORS | Open | `Access-Control-Allow-Origin: *` on all responses |
| TLS | None | HTTP only (acceptable for localhost) |
| Rate limiting | None | No built-in rate limiting |

> [!WARNING]
> The API server has no authentication and allows all CORS origins. While it binds to localhost only, any process on the same machine can access it. Do not expose the API server port through reverse proxies, SSH tunnels, or firewall rules without adding an authentication layer.

### 7.2 Remote API Key Handling

When using remote API models (OpenAI, Groq, etc.) through `ChatAPI`, API keys are:

- Set via `ChatAPI::setAPIKey()` — stored in memory only
- Not persisted to disk by the ChatAPI class itself
- Transmitted over HTTPS to the respective API endpoints

### 7.3 Model Download Integrity

- Models are downloaded from `https://gpt4all.io/models/gguf/` over HTTPS
- The Node.js SDK supports MD5 verification of downloaded models
- No GPG signature verification is implemented

### 7.4 Code Execution Risk

The `CodeInterpreter` tool executes JavaScript code generated by the LLM. This presents a code execution risk:

- Execution is limited to 3 consecutive tool calls per response
- The interpreter runs within the application's process
- No sandboxing or permission restrictions are enforced on the executed code

> [!CAUTION]
> The CodeInterpreter runs JavaScript within the application process with full access rights. Malicious or poorly-formed model output could potentially execute harmful code. Monitor tool call activity when using models that support tool use.

### 7.5 Data Privacy

- All inference runs locally — no data leaves the machine (unless using remote API models)
- LocalDocs indexes are stored in a local SQLite database
- Chat history is stored locally in the model path directory
- Telemetry (via `Network` class) is optional and can be disabled

---

## 8. Developer Internals

### 8.1 Build System Deep Dive

The project uses CMake with a two-level structure:

```
gpt4all-backend/CMakeLists.txt    ← Backend shared libraries
gpt4all-chat/CMakeLists.txt       ← Desktop application (depends on backend)
```

Key CMake features:
- `FetchContent` for GoogleTest integration in the test suite
- CPack configuration (`cpack_config.cmake`) for installer generation
- Platform-specific deployment scripts in `gpt4all-chat/cmake/`
- Flatpak manifest (`io.gpt4all.gpt4all`) for Linux packaging

### 8.2 Backend Plugin System

The `LLModel` abstract class defines the inference interface:

```cpp
class LLModel {
public:
    virtual bool loadModel(const std::string &modelPath, int n_ctx, int ngl) = 0;
    virtual void prompt(const std::string &prompt,
                       const std::string &promptTemplate,
                       std::function<bool(int32_t)> promptCallback,
                       std::function<bool(int32_t, const std::string&)> responseCallback,
                       bool special = false,
                       std::string *fakeReply = nullptr) = 0;
    virtual size_t stateSize() const = 0;
    virtual size_t saveState(uint8_t *dest) const = 0;
    virtual size_t restoreState(const uint8_t *src) = 0;
    virtual std::vector<float> embed(const std::string &text) = 0;
    // ... tokenize, setThreadCount, etc.
};
```

To add a new backend:
1. Create a new class inheriting from `LLModel`
2. Implement all pure virtual methods
3. Build as a shared library
4. Place in the implementation search path

### 8.3 C API (`llmodel_c.h`)

The C API is the FFI boundary used by the Python and Node.js SDKs. It wraps the C++ `LLModel` interface with C-linkage functions operating on opaque `llmodel_model` handles. This API is defined in `gpt4all-backend/include/gpt4all-backend/llmodel_c.h`.

### 8.4 Python SDK Internals

Architecture:
- `_pyllmodel.py` — Low-level ctypes bindings to `llmodel_c.h`
- `gpt4all.py` — High-level `GPT4All` class providing the public API

The `setup.py` handles platform detection and copies prebuilt backend shared libraries from `gpt4all-backend/build/` into the Python package at install time.

### 8.5 Node.js SDK Internals

The native addon is built with `node-gyp` and uses N-API via `node-addon-api`:

**Core class**: `NodeModelWrapper` (defined in `index.cc` / `index.h`)

| Method | Purpose |
|--------|---------|
| `GetType` | Return model type string |
| `IsModelLoaded` | Check if model is ready |
| `StateSize` | Get serialized state size |
| `Infer` | Run inference (mutex-protected) |
| `SetThreadCount` | Configure inference threads |
| `Dispose` | Release model resources |
| `GetName` | Get model name |
| `ThreadCount` | Get current thread count |
| `GenerateEmbedding` | Generate text embeddings |
| `HasGpuDevice` | Check GPU availability |
| `ListGpus` | Enumerate available GPUs |
| `InitGpuByString` | Initialize specific GPU |
| `GetRequiredMemory` | Query model memory requirements |
| `GetGpuDevices` | Get GPU device details |
| `GetLibraryPath` | Get backend library path |

### 8.6 API Server Implementation

The `Server` class inherits from `ChatLLM` and exposes an OpenAI-compatible HTTP API:

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/v1/models` | List installed models |
| `GET` | `/v1/models/<model>` | Get specific model info |
| `POST` | `/v1/completions` | Text completion |
| `POST` | `/v1/chat/completions` | Chat completion |

#### Request Schema

**Base parameters** (shared by both endpoints):

| Parameter | Type | Default | Constraints |
|-----------|------|---------|-------------|
| `model` | string | required | Must match an installed model |
| `max_tokens` | integer | 16 | min: 1 |
| `n` | integer | 1 | min: 1 |
| `temperature` | number | 1.0 | range: 0–2 |
| `top_p` | number | 1.0 | range: 0–1 |
| `min_p` | number | 0.0 | range: 0–1 |

**Completion-specific**: `prompt` (string, required), `echo` (boolean, default false)

**Chat-specific**: `messages` (array of `{role, content}`, required). Valid roles: `system`, `user`, `assistant`.

#### Unsupported Parameters

The following OpenAI API parameters raise `InvalidRequestError` if provided: `stream`, `frequency_penalty`, `presence_penalty`, `seed`, `stop`, `stream_options`, `logit_bias`, `logprobs`, `tools`, `tool_choice`, `functions`, `response_format`, `best_of`, `suffix`.

#### Response Format

Responses follow the OpenAI API structure with `id`, `object`, `created`, `model`, `choices`, and `usage` (containing `prompt_tokens`, `completion_tokens`, `total_tokens`).

Chat completions can optionally include a `references` array when LocalDocs "show references" is enabled.

> [!NOTE]
> The server returns HTTP 401 Unauthorized for all requests when the server is not enabled via `MySettings::serverChat()`. This is a global kill switch, not an authentication mechanism.

### 8.7 Test Infrastructure

| Suite | Framework | Location |
|-------|-----------|----------|
| C++ unit tests | GoogleTest | `gpt4all-chat/tests/cpp/` |
| Python API tests | pytest | `gpt4all-chat/tests/python/test_server_api.py` |
| Node.js SDK tests | Jest | `gpt4all-bindings/typescript/test/` |
| Python SDK tests | pytest | `gpt4all-bindings/python/gpt4all/tests/` |

#### Server Integration Tests

The Python server tests (`test_server_api.py`) use a `Requestor` class that:
1. Creates a temporary directory for test state
2. Optionally copies a model file from `TEST_MODEL_PATH`
3. Starts the chat executable (from `CHAT_EXECUTABLE`) as a subprocess
4. Sends HTTP requests to the API server
5. Validates responses against expected structure

```bash
# Running server tests
export CHAT_EXECUTABLE=/path/to/chat
export TEST_MODEL_PATH=/path/to/model.gguf
pytest gpt4all-chat/tests/python/test_server_api.py
```

### 8.8 Adding a New Tool

The tool system is extensible through the `ToolModel` registry. Currently only `CodeInterpreter` is implemented. To add a new tool:

1. Create a new class following the `CodeInterpreter` pattern
2. Register it in `ToolModel`
3. Add parsing support in `ToolCallParser` for the tool's markup format
4. Handle tool results in the `Chat` response cycle

---

## 9. Integration Points

### 9.1 OpenAI-Compatible API

The built-in API server provides drop-in compatibility with OpenAI's API format, enabling integration with any tool that supports the OpenAI API:

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:{port}/v1",
    api_key="not-needed"  # no auth required
)

response = client.chat.completions.create(
    model="your-installed-model",
    messages=[
        {"role": "user", "content": "Hello, world!"}
    ]
)
print(response.choices[0].message.content)
```

> [!TIP]
> Although no API key is required, many OpenAI client libraries require the parameter to be set. Pass any non-empty string.

### 9.2 Python SDK Integration

```python
from gpt4all import GPT4All

model = GPT4All("model-name")
response = model.generate("Your prompt here")
```

The Python SDK provides direct access to the inference engine without the desktop application. It uses ctypes FFI to call the same C backend.

### 9.3 Node.js SDK Integration

```javascript
const { createCompletion, loadModel } = require("gpt4all");

const model = await loadModel("model-name");
const result = await createCompletion(model, [
    { role: "user", content: "Your prompt here" }
]);
```

### 9.4 Training Pipeline Integration

The training pipeline integrates with:

- **Hugging Face Hub**: for base model and tokenizer downloads
- **Weights & Biases**: for experiment tracking and visualization
- **Nomic Atlas**: for embedding visualization and map creation
- **DeepSpeed**: for distributed training and memory optimization

Training output (LoRA adapters or full model weights) can be converted to GGUF format using llama.cpp's conversion tools for use with the inference engine.

### 9.5 Embedding API

Both the Python and Node.js SDKs expose embedding generation:

```python
# Python
model = GPT4All("nomic-embed-text-v1.5")
embeddings = model.embed("Text to embed")
```

Embedding models must use `bert` or `nomic-bert` architecture. These models are identified by the backend and routed to embedding-specific inference paths.

### 9.6 Troubleshooting

#### Common Issues

| Symptom | Likely Cause | Resolution |
|---------|-------------|------------|
| Application process persists after close | Known issue ([#3656](https://github.com/nomic-ai/gpt4all/issues/3656)) | Force-kill the `chat.exe` / `chat` process |
| Vulkan `ErrorInitializationFailed` | Driver or SDK mismatch ([#3259](https://github.com/nomic-ai/gpt4all/issues/3259)) | Update GPU drivers; verify Vulkan SDK installation |
| Ubuntu installer fails | Package conflicts ([#3640](https://github.com/nomic-ai/gpt4all/issues/3640)) | Install dependencies manually; try Flatpak instead |
| Fails to run after installation | Missing runtime dependencies ([#3456](https://github.com/nomic-ai/gpt4all/issues/3456)) | Install Qt6 runtime; check `ldd` for missing shared libs |
| Window resize issues (v3.0.0) | Qt/QML rendering bug ([#2712](https://github.com/nomic-ai/gpt4all/issues/2712)) | Update to latest release |
| fcitx Chinese input on Linux | Qt input method issue ([#2967](https://github.com/nomic-ai/gpt4all/issues/2967)) | Set `QT_IM_MODULE=fcitx` environment variable |
| Docker CUDA version conflict | CUDA 11 vs 12 mismatch ([#3033](https://github.com/nomic-ai/gpt4all/issues/3033)) | Match CUDA toolkit version to driver capabilities |
| PyPI package out of date | Release lag ([#3650](https://github.com/nomic-ai/gpt4all/issues/3650)) | Install from source or use nightly builds |

#### Diagnostic Steps

1. **Enable verbose logging**: `export GPT4ALL_VERBOSE_LLAMACPP=1`
2. **Check model loading**: Verbose output will show layer-by-layer loading, device placement, and any quantization issues
3. **Verify GPU availability**: Use `ListGpus` / `GetGpuDevices` from the SDK to confirm GPU detection
4. **Test API server**: `curl http://localhost:{port}/v1/models` should return a model list when the server is enabled
5. **Check memory**: Monitor RAM/VRAM during model loading — out-of-memory failures may manifest as cryptic crashes

---

_Generated May 2026 from nomic-ai/gpt4all at b666d16. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._