PrivateGPT User Guide

1. Quick Start

PrivateGPT is a production-ready AI project that lets you interact with your documents using large language models (LLMs), completely privately. No data leaves your execution environment. It exposes an OpenAI-compatible API and a built-in Gradio web UI with RAG (Retrieval-Augmented Generation), search, chat, and summarization modes.

Key features:

• Fully private document Q&A — your data never leaves your machine
• OpenAI-compatible REST API for programmatic access
• Built-in Gradio web UI with four interaction modes
• Support for multiple LLM backends (Ollama, llama.cpp, OpenAI, Azure OpenAI, Gemini, SageMaker)
• Multiple vector store options (Qdrant, Chroma, PostgreSQL, ClickHouse, Milvus)
• Document ingestion with file upload, text input, and bulk/watch modes
• Streaming responses with source attribution
• Optional HTTP basic authentication

Prerequisites: Python 3.11+, Poetry, and an LLM backend (Ollama is the easiest to start with).

Fastest start with Ollama:

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"
PGPT_PROFILES=ollama python -m private_gpt

12:00:00.000 [INFO   ]      private_gpt.launcher - Mounting the gradio UI, at path=/
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

Open http://localhost:8001 in your browser to access the web UI. Upload a document, select RAG mode, and ask a question about it.

Tip

If you already have Ollama running with llama3.1 and nomic-embed-text models pulled, PrivateGPT will work immediately with the ollama profile — no additional model downloads needed.

What to try next:

• Upload documents via the web UI sidebar
• Switch between RAG, Search, Basic Chat, and Summarize modes
• Use the API directly at http://localhost:8001/v1/chat/completions
• See the Configuration section for customizing your setup

2. Installation

System Requirements

Installation Methods

Ollama (Recommended)

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"

Local llama.cpp

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
poetry install --extras "ui llms-llama-cpp vector-stores-qdrant embeddings-huggingface"

OpenAI

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
poetry install --extras "ui llms-openai embeddings-openai vector-stores-qdrant"

Docker (llama.cpp CPU)

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
docker compose up -d

Docker (Ollama)

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
docker build -t private-gpt -f Dockerfile.ollama .
docker run -p 8001:8080 private-gpt

Important

You must install the correct Poetry extras for your chosen LLM and embedding backend. Mismatched extras cause ImportError at startup (e.g., UI dependencies not found, install with poetry install --extras ui).

Post-Install Verification

PGPT_PROFILES=ollama python -m private_gpt

12:00:00.000 [INFO   ]      private_gpt.launcher - Mounting the gradio UI, at path=/
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

Verify the health endpoint:

curl http://localhost:8001/health

{"status": "ok"}

Poetry Extras Reference

Install Troubleshooting

Error: Preparing metadata (pyproject.toml) did not run successfully

Cause: Missing system dependencies or incompatible Python version. This is the most commonly reported installation issue (#942, #1314).

Solution:

1. Verify Python version is 3.11+:

   python --version
   

2. Ensure Poetry is up to date:

   pip install --upgrade poetry
   

3. On Ubuntu/Debian, install build dependencies:

   sudo apt-get install build-essential python3-dev
   

Prevention: Always use Python 3.11+ and the latest Poetry version.

Error: Python was not found on Windows

Cause: Python is not installed or not in PATH (#539).

Solution:

1. Install Python 3.11+ from python.org
2. Ensure "Add Python to PATH" is checked during installation
3. Restart your terminal

Error: AttributeError: type object 'hnswlib.Index' has no attribute 'file_handle_count'

Cause: Version incompatibility with the hnswlib package (#1012).

Solution:

pip install --force-reinstall hnswlib

Error: Dependency conflict on Apple Silicon (Mac M1/M2)

Cause: Some native dependencies lack pre-built wheels for ARM macOS (#2076).

Solution:

brew install cmake
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"

Error: Building Docker image fails on Ubuntu

Cause: Missing Docker build dependencies or outdated Docker version (#1244).

Solution: Ensure Docker and Docker Compose are installed and up to date:

docker compose version

3. Configuration

PrivateGPT uses a YAML-based profile system. The base configuration is in settings.yaml, and profiles override specific values.

Profile System

Activate profiles by setting the PGPT_PROFILES environment variable:

PGPT_PROFILES=ollama python -m private_gpt

Multiple profiles can be comma-separated and are merged in order:

PGPT_PROFILES=ollama,local python -m private_gpt

Note

The default profile (settings.yaml) is always loaded first. Additional profiles are merged on top, with later profiles taking priority.

Available Profiles

Environment Variable Substitution

YAML values support ${VAR} and ${VAR:default} syntax for environment variables:

openai:
  api_key: ${OPENAI_API_KEY:}
  model: gpt-3.5-turbo

Warning

If an environment variable is referenced with ${VAR} (no default) and is not set, PrivateGPT will fail to start with: ValueError: Environment variable VAR is not set and not default was provided.

Common Configuration Options

Settings Folder Override

To load settings from a custom directory:

PGPT_SETTINGS_FOLDER=/path/to/settings PGPT_PROFILES=ollama python -m private_gpt

For the full configuration reference with all settings, see the Admin Guide.

4. Core Usage

Web UI

The Gradio-based web UI is available at the server root (default http://localhost:8001/). It provides four interaction modes:

RAG Mode (Default)

Get answers grounded in your uploaded documents. The LLM uses ingested document context to generate responses with source citations.

1. Upload one or more documents using the Upload File(s) button
2. Ensure RAG mode is selected
3. Type your question in the chat input
4. The response includes source file names and page numbers

Tip

Select a specific file from the ingested files list to restrict RAG queries to that document only. Click De-select selected file to query across all documents again.

Search Mode

Find relevant text chunks across your documents without LLM generation. Returns up to 4 matching chunks with file name, page number, and the matching text. Useful for quickly locating specific information.

Basic Chat Mode

Chat directly with the LLM using its training data. Uploaded documents are ignored in this mode.

Summarize Mode

Generate summaries of ingested documents. Optionally select a specific file and provide custom summarization instructions in the chat input.

System Prompts

Each mode has a configurable system prompt that influences how the LLM responds. The defaults are set in settings.yaml under ui.default_chat_system_prompt, ui.default_query_system_prompt, and ui.default_summarization_system_prompt. You can override them in the UI's System Prompt field during a session.

Note

The chat history is capped at 20 messages to avoid exceeding the LLM's context window. Older messages are silently dropped.

File Management

• Upload: Click Upload File(s) — supports multiple files. Re-uploading a file with the same name replaces the previous version automatically.
• Select: Click a file in the ingested files list to scope queries and deletion to that file.
• Delete selected: Select a file in the list, then click Delete selected file.
• Delete all: Click Delete ALL files to clear the entire ingestion store.

Warning

File deletion is permanent. There is no undo. Deleted documents are removed from the vector store immediately.

API Usage

PrivateGPT exposes an OpenAI-compatible REST API. All versioned endpoints are under /v1/.

Chat Completions

curl -X POST http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is PrivateGPT?"}
    ],
    "use_context": false,
    "stream": false
  }'

{
  "id": "c202d5e6-7b69-4869-81cc-dd574ee8ee11",
  "object": "completion",
  "created": 1623340000,
  "model": "private-gpt",
  "choices": [
    {
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "PrivateGPT is a privacy-focused AI assistant..."
      },
      "sources": null,
      "index": 0
    }
  ]
}

RAG Chat (with Document Context)

curl -X POST http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Summarize the key findings"}
    ],
    "use_context": true,
    "include_sources": true,
    "stream": false
  }'

When use_context is true, the response includes sources in each choice — an array of chunks with document IDs, file names, page labels, relevance scores, and the matching text.

Streaming Responses

Set "stream": true to receive Server-Sent Events (SSE). Each event follows the OpenAI streaming format:

data: {"id":"12345","object":"completion.chunk","created":1694268190,"model":"private-gpt","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"12345","object":"completion.chunk","created":1694268190,"model":"private-gpt","choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop"}]}

data: [DONE]

Text Completions

curl -X POST http://localhost:8001/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "How do you fry an egg?",
    "system_prompt": "You are a helpful chef.",
    "use_context": false,
    "stream": false
  }'

Tip

The /v1/chat/completions endpoint is recommended over /v1/completions for most use cases.

Ingest a File

curl -X POST http://localhost:8001/v1/ingest/file \
  -F "file=@/path/to/document.pdf"

{
  "object": "list",
  "model": "private-gpt",
  "data": [
    {
      "object": "ingest.document",
      "doc_id": "c202d5e6-7b69-4869-81cc-dd574ee8ee11",
      "doc_metadata": {"file_name": "document.pdf", "page_label": "1"}
    }
  ]
}

Ingest Text

curl -X POST http://localhost:8001/v1/ingest/text \
  -H "Content-Type: application/json" \
  -d '{
    "file_name": "Company Policy",
    "text": "All employees must complete security training annually."
  }'

List Ingested Documents

curl http://localhost:8001/v1/ingest/list

Delete a Document

curl -X DELETE http://localhost:8001/v1/ingest/c202d5e6-7b69-4869-81cc-dd574ee8ee11

Context Filtering

Restrict queries to specific documents by passing context_filter with document IDs (obtained from /v1/ingest/list):

{
  "messages": [{"role": "user", "content": "What does this say?"}],
  "use_context": true,
  "context_filter": {
    "docs_ids": ["c202d5e6-7b69-4869-81cc-dd574ee8ee11"]
  }
}

Retrieve Chunks

curl -X POST http://localhost:8001/v1/chunks \
  -H "Content-Type: application/json" \
  -d '{
    "text": "quarterly revenue",
    "limit": 5,
    "prev_next_chunks": 2
  }'

Summarize

curl -X POST http://localhost:8001/v1/summarize \
  -H "Content-Type: application/json" \
  -d '{
    "use_context": true,
    "instructions": "Provide a concise executive summary",
    "stream": false
  }'

Generate Embeddings

curl -X POST http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello world"}'

5. Common Tasks

Task: Set Up PrivateGPT with Ollama

Goal: Run PrivateGPT locally with Ollama as the LLM backend.

Prerequisites: Ollama installed and running.

Steps:

1. Pull the required models:

   ollama pull llama3.1
   ollama pull nomic-embed-text
   

2. Install PrivateGPT with Ollama extras:

   poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"
   

3. Start with the Ollama profile:

   PGPT_PROFILES=ollama python -m private_gpt
   

4. Open http://localhost:8001 in your browser.

Verify: The health endpoint returns {"status": "ok"}.

If it fails: Check that Ollama is running (ollama list) and the models are pulled. See Troubleshooting for connection issues.

Task: Use OpenAI as the Backend

Goal: Use OpenAI's API instead of a local model.

Steps:

1. Install with OpenAI extras:

   poetry install --extras "ui llms-openai embeddings-openai vector-stores-qdrant"
   

2. Set your API key:

   export OPENAI_API_KEY=sk-your-key-here
   

3. Start with the OpenAI profile:

   PGPT_PROFILES=openai python -m private_gpt
   

Task: Ingest a Folder of Documents

Goal: Bulk-ingest an entire folder of documents.

Prerequisites: Enable local ingestion in settings.

Steps:

1. In your settings YAML, enable local ingestion:

   data:
     local_ingestion:
       enabled: true
       allow_ingest_from: ["*"]
   

2. Use the provided script:

   python scripts/ingest_folder.py /path/to/documents
   

Warning

Local ingestion is not secure by default and should only be used in controlled environments. Set allow_ingest_from to restrict which folders can be ingested.

Task: Use PostgreSQL as Vector and Node Store

Goal: Use PostgreSQL for persistent storage instead of Qdrant.

Steps:

1. Install with PostgreSQL extras:

   poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-postgres storage-nodestore-postgres"
   

2. Start with the Ollama + PostgreSQL profile:

   PGPT_PROFILES=ollama-pg python -m private_gpt
   

3. Default PostgreSQL connection: localhost:5432, database postgres, user postgres, schema private_gpt.

Task: Enable Authentication

Goal: Protect the API with HTTP basic authentication.

Steps:

1. Generate your auth secret:

   python -c 'import base64; print("Basic " + base64.b64encode("myuser:mypass".encode()).decode())'
   

   Basic bXl1c2VyOm15cGFzcw==
   

2. Add to your settings YAML:

   server:
     auth:
       enabled: true
       secret: "Basic bXl1c2VyOm15cGFzcw=="
   

3. Include the Authorization header with every API request:

   curl -H "Authorization: Basic bXl1c2VyOm15cGFzcw==" http://localhost:8001/v1/ingest/list
   

Caution

The default configuration ships with a hardcoded auth secret (Basic c2VjcmV0OmtleQ==, which decodes to secret:key). Always change this before exposing PrivateGPT to any network (#2204).

Task: Deploy with Docker Compose

Goal: Run PrivateGPT in a container.

Steps:

1. Configure environment variables in your shell or a .env file:

   export PGPT_MODE=ollama
   export PGPT_EMBED_MODE=ollama
   export PGPT_OLLAMA_API_BASE=http://ollama:11434
   

2. Start the stack:

   docker compose up -d
   

3. Access the UI at http://localhost:8080 (Docker profile uses port 8080).

Task: Use Azure OpenAI

Goal: Connect to Azure OpenAI instead of direct OpenAI.

Steps:

1. Set environment variables:

   export AZ_OPENAI_API_KEY=your-key
   export AZ_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
   export AZ_OPENAI_EMBEDDING_DEPLOYMENT_NAME=your-embedding-deployment
   export AZ_OPENAI_LLM_DEPLOYMENT_NAME=your-llm-deployment
   

2. Start with the Azure profile:

   PGPT_PROFILES=azopenai python -m private_gpt
   

Task: Speed Up Document Ingestion

Goal: Use parallel processing for faster ingestion.

Update your settings YAML:

embedding:
  mode: huggingface
  ingest_mode: parallel
  count_workers: 4

Tip

For local setups, parallel mode is fastest. Do not set count_workers higher than your CPU thread count — it may cause memory issues.

Task: Use Google Gemini

Goal: Use Google's Gemini models.

Steps:

1. Set your API key:

   export GOOGLE_API_KEY=your-gemini-api-key
   

2. Start with the Gemini profile:

   PGPT_PROFILES=gemini python -m private_gpt
   

Task: Enable Reranking for Better RAG Results

Goal: Improve retrieval quality with a cross-encoder reranker.

Add to your settings YAML:

rag:
  similarity_top_k: 5
  rerank:
    enabled: true
    model: cross-encoder/ms-marco-MiniLM-L-2-v2
    top_n: 2

This retrieves the top 5 candidates, then reranks to return the top 2 most relevant.

6. Troubleshooting

Diagnostic Decision Tree

PrivateGPT won't start
├── ImportError?
│   └── Install missing extras: poetry install --extras "..."
├── ValueError about environment variable?
│   └── Set the required env var or add a default in YAML
├── Connection refused on port 8001?
│   └── Check if another process uses the port
└── Model download hangs?
    └── Check internet connection / Ollama service status

Error: ImportError: UI dependencies not found, install with poetry install --extras ui

Cause: The ui extra was not installed, but ui.enabled is true in settings.

Solution:

poetry install --extras "ui"

Prevention: Always include ui in your extras if you want the web interface.

Error: ImportError: Ollama dependencies not found, install with poetry install --extras llms-ollama or embeddings-ollama

Cause: Ollama Python package not installed but an Ollama LLM or embedding mode is configured.

Solution:

poetry install --extras "llms-ollama embeddings-ollama"

Error: Failed to connect to Ollama

Cause: Ollama service is not running or unreachable at the configured api_base (#2071).

Solution:

1. Verify Ollama is running:

   ollama list
   

2. Check the ollama.api_base setting matches your Ollama instance (default: http://localhost:11434).
3. If running in Docker, use the Docker service name (e.g., http://ollama:11434).

Prevention: Set ollama.autopull_models: true to automatically pull missing models on startup.

Error: HNSW index loading error when ingesting multiple documents

Cause: Concurrent writes to the HNSW index in local-only mode (#2180).

Solution:

1. Reduce embedding.count_workers to 1
2. Use simple ingest mode
3. Consider switching to a client-server vector store (Qdrant server, PostgreSQL)

Prevention: For multi-document ingestion, use a vector store that handles concurrent writes natively.

Error: TypeError: Config file has no top-level mapping

Cause: A settings YAML file is empty or malformed.

Solution: Verify the profile YAML file has valid YAML with a top-level mapping:

python -c "import yaml; print(yaml.safe_load(open('settings-ollama.yaml')))"

Error: ValueError: Environment variable ... is not set and not default was provided

Cause: A ${VAR} reference in YAML has no matching environment variable and no default.

Solution: Either set the environment variable or provide a default in the YAML:

# Before (fails if OPENAI_API_KEY not set):
api_key: ${OPENAI_API_KEY}

# After (empty string default):
api_key: ${OPENAI_API_KEY:}

Error: HTTP 401 Not authenticated

Cause: Authentication is enabled and the request lacks a valid Authorization header.

Solution: Include the configured auth secret in the header:

curl -H "Authorization: Basic c2VjcmV0OmtleQ==" http://localhost:8001/v1/ingest/list

Warning

The secret must match the exact value in server.auth.secret, including the Basic prefix. The comparison uses constant-time comparison to prevent timing attacks.

Error: ValueError: Settings are not configured

Cause: Settings object is None, typically from a corrupted or missing settings file.

Solution:

1. Ensure settings.yaml exists in the project root
2. Verify PGPT_SETTINGS_FOLDER points to the correct directory
3. Check that the YAML is valid

Error: Preparing metadata (pyproject.toml) did not run successfully

Cause: Missing system dependencies or incompatible Python version (#942, #1314).

Solution:

1. Ensure Python 3.11+
2. Install build tools: sudo apt-get install build-essential python3-dev
3. Update Poetry: pip install --upgrade poetry

UI shows no model label

Cause: The llm.mode value doesn't match any known mode.

Solution: Ensure llm.mode is one of: llamacpp, openai, openailike, azopenai, sagemaker, mock, ollama, gemini.

7. FAQ

Q: Which LLM backend should I choose?

For the easiest local setup, use Ollama — it manages model downloads and inference automatically. For maximum privacy with no external dependencies, use llama.cpp. For best quality without privacy constraints, use OpenAI or Azure OpenAI.

Q: What document formats are supported?

PrivateGPT uses LlamaIndex for document parsing, which supports PDF, DOCX, TXT, CSV, HTML, Markdown, and many other formats. You may need to install an extra dependency for specific file types.

Q: Can I use PrivateGPT without the web UI?

Yes. Set ui.enabled: false in your settings and interact entirely through the REST API. This also removes the need for the ui Poetry extra.

Q: How do I change the default LLM model?

Update the model setting in your profile YAML. For Ollama: ollama.llm_model. For llama.cpp: llamacpp.llm_hf_repo_id and llamacpp.llm_hf_model_file. For OpenAI: openai.model.

Q: What vector stores are supported?

Qdrant (local file or server), Chroma, PostgreSQL (with pgvector), ClickHouse, and Milvus. Set via vectorstore.database.

Q: How do I reset all ingested documents?

Delete all documents via the API — list with GET /v1/ingest/list then delete each with DELETE /v1/ingest/{doc_id}. Or use the Delete ALL files button in the web UI.

Q: Can multiple users share the same instance?

All ingested documents are shared across all sessions. There is no per-user document isolation (#1796). Enable authentication to control API access.

Q: What is the prompt_style setting?

It controls how messages are formatted for the LLM. Options: default, llama2, llama3, tag, mistral, chatml. Use llama3 for Llama 3.x models, mistral for Mistral models.

Q: How does the tiktoken cache work?

PrivateGPT bundles a tiktoken_cache/ directory so tokenization works in fully offline environments. The TIKTOKEN_CACHE_DIR environment variable is set automatically at import time.

Q: Is Gradio analytics enabled?

No. PrivateGPT explicitly disables Gradio analytics by setting GRADIO_ANALYTICS_ENABLED=False at import time.