# TextGen (text-generation-webui) Admin Guide

This guide is for operators, developers, and architects who deploy, secure, extend, or
maintain TextGen. It covers the system architecture, the complete configuration surface, the
loader and sampler backends, the API server, security model, operations, and internals deep
enough to reason about — or contribute to — the codebase.

For end-user, install, and day-to-day usage, see the companion **User Guide**.

---

## 1. Executive Summary

TextGen is a Python application that wraps multiple local LLM inference runtimes behind a
single **Gradio** web UI and an optional **FastAPI** OpenAI/Anthropic-compatible HTTP server.
It is a desktop/self-hosted product, not a multi-tenant service: there is no database, no
user accounts beyond optional HTTP Basic auth, and the unit of state is the `user_data/`
directory on disk.

**Architecturally it is three layers:**

1. **A launcher/installer layer** (`one_click.py`, `start_*.sh/.bat`, `desktop/main.js`) that
   provisions an isolated Python 3.13 + PyTorch environment per hardware target, or ships as a
   self-contained portable build.
2. **An orchestration layer** (`server.py` + `modules/`) that parses configuration, loads a
   model through one of five swappable loaders, builds the Gradio interface, and manages
   generation.
3. **A serving layer** (`modules/api/`) that re-exposes the loaded model as OpenAI- and
   Anthropic-compatible HTTP endpoints.

**Key design facts an operator must know:**

- The active model is global, singleton state (`modules/shared.py:model`). One model is
  loaded at a time (image models load separately and concurrently).
- The llama.cpp loader is **out-of-process**: it spawns and supervises a `llama-server`
  subprocess. Clean shutdown explicitly stops it.
- Configuration precedence is **command line → `CMD_FLAGS.txt` → `settings.yaml` → built-in
  defaults**.
- The API is **localhost-locked by default** and only opens to the network on explicit opt-in.

---

## 2. Architecture

### 2.1 Component overview

```
                         ┌──────────────────────────────────────────────┐
                         │  Launcher / Installer                        │
   start_windows.bat ───►│  one_click.py: conda env + PyTorch per-GPU   │
   start_linux.sh        │  desktop/main.js: Electron window (portable) │
                         └───────────────────┬──────────────────────────┘
                                             │ python server.py [flags]
                                             ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  server.py  (entry point)                                                 │
   │   • modules/shared.py   → argparse CLI, settings dict, global state       │
   │   • modules/paths.py    → resolve user_data dir                           │
   │   • loads settings.yaml, CMD_FLAGS.txt, mcp.json                          │
   └───────────────┬───────────────────────────────────┬──────────────────────┘
                   │ create_interface()                 │ if --api / --nowebui
                   ▼                                     ▼
   ┌───────────────────────────────────┐   ┌────────────────────────────────────┐
   │  Gradio UI (modules/ui_*.py)      │   │  FastAPI server (modules/api/)     │
   │  Tabs: Chat, Notebook, Parameters,│   │  uvicorn on :5000                  │
   │  Character, Model, Image, Train,  │   │  /v1/* OpenAI + /v1/messages       │
   │  Session                          │   │  Anthropic                         │
   └───────────────┬───────────────────┘   └────────────────┬───────────────────┘
                   │                                         │
                   └──────────────┬──────────────────────────┘
                                  ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  Generation core                                                          │
   │   modules/text_generation.py · modules/chat.py · modules/sampler_hijack  │
   └───────────────┬──────────────────────────────────────────────────────────┘
                   ▼  modules/models.py dispatches to one loader
   ┌──────────────┬──────────────┬───────────────┬─────────────┬──────────────┐
   │  llama.cpp   │ Transformers │ ExLlamav3_HF  │ ExLlamav3   │ TensorRT-LLM │
   │ (subprocess) │ (in-process) │ (in-process)  │ (in-process)│ (in-process) │
   └──────────────┴──────────────┴───────────────┴─────────────┴──────────────┘
                   ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  user_data/  (the entire persistent state surface)                        │
   │  models · loras · characters · presets · instruction-templates · grammars │
   │  logs · cache · settings.yaml · CMD_FLAGS.txt · mcp.json                   │
   └──────────────────────────────────────────────────────────────────────────┘
```

### 2.2 Repository structure

| Path | Responsibility |
|---|---|
| `server.py` | Entry point: parse config, load model, build Gradio UI, start API, restart loop |
| `download-model.py` | Standalone Hugging Face model downloader (`ModelDownloader` class) |
| `one_click.py` | Installer/updater: conda env, PyTorch per-GPU, requirements, update wizard |
| `modules/` | Core engine — config, loaders, generation, UI, samplers, tools |
| `modules/api/` | FastAPI OpenAI/Anthropic-compatible server |
| `modules/grammar/` | GBNF grammar parsing + constrained-logits processor |
| `extensions/` | Optional bundled extensions (TTS, STT, RAG, translation, …) |
| `user_data/` | Per-user state (models, presets, characters, logs, config files) |
| `requirements/` | `full/` (per-GPU) and `portable/` dependency sets |
| `css/`, `js/` | Front-end assets (fonts, KaTeX, highlight.js, morphdom, UI logic) |
| `desktop/` | Electron desktop wrapper (`main.js`, `preload.js`) |
| `docker/` | Per-vendor Dockerfiles (`nvidia`, `amd`, `intel`, `cpu`, `TensorRT-LLM`) |

### 2.3 Key modules

| Module | Role |
|---|---|
| `modules/shared.py` | Single source of truth for the argparse CLI, the `settings` dict, and all global runtime state |
| `modules/paths.py` | `resolve_user_data_dir()` — portable/electron/in-tree precedence |
| `modules/models.py` | `load_model()` / `unload_model()` / `unload_model_if_idle()` dispatch |
| `modules/models_settings.py` | Reads model metadata, auto-detects loader and parameters |
| `modules/loaders.py` | `loaders_and_params` + `loaders_samplers` — the per-loader capability matrix |
| `modules/llama_cpp_server.py` | Spawns and supervises the `llama-server` subprocess |
| `modules/text_generation.py` | The generation entry points and stop handling |
| `modules/chat.py` | Chat history, templates, character handling, attachments |
| `modules/sampler_hijack.py` | Custom sampler implementations patched into Transformers |
| `modules/presets.py` | Default sampler values and preset YAML load/save |
| `modules/reasoning.py` | Detects model-family-specific "thinking" blocks |
| `modules/tool_use.py` | Tool calling + MCP server integration |
| `modules/web_search.py` | DuckDuckGo search with SSRF protection |
| `modules/extensions.py` | Extension discovery, setup, and hook application |
| `modules/ui_*.py` | One module per Gradio tab |

### 2.4 The startup sequence (`server.py`)

1. Install SIGINT/SIGTERM handlers (second Ctrl-C forces exit; explicitly stops a running
   `LlamaServer`).
2. `do_cmd_flags_warnings()` — emit security warnings (see [§6](#6-security)).
3. Load `settings.yaml` (CLI `--settings` path, else `user_data/settings.yaml` if present).
4. Detect MCP config (`user_data/mcp.json`) and image-model CLI overrides.
5. Activate extensions from `settings['default_extensions']` (the legacy `openai` extension is
   aliased to `--api`).
6. Resolve and optionally load the model (`--model`, or `--model-menu` for a terminal picker).
7. If `--idle-timeout > 0`, start the idle-unload watchdog thread.
8. If `--nowebui`, start only the API; otherwise `create_interface()` builds Gradio, starts
   the API if requested, and enters a restart loop watching `shared.need_restart`.

> [!NOTE]
> The model is loaded **before** the UI is built. A bad `--model` or out-of-memory condition
> surfaces at startup, not on first chat.

---

## 3. Requirements & Environment

| Component | Requirement |
|---|---|
| Python | **3.13** exactly (the installer refuses to update across Python versions) |
| PyTorch | **2.9.0**, installed from a per-GPU index (CUDA 12.8 / ROCm 7.2 / CPU / XPU) |
| NVIDIA | CUDA **12.8** (`NVIDIA_CUDA128`). The older CUDA 12.4 path was removed — needs a clean reinstall |
| AMD | ROCm **7.2**, **Linux only** (no AMD-on-Windows install path) |
| Apple | M-series via the CPU/MPS PyTorch wheel |
| Intel | Arc (beta) via the XPU wheel |
| libstdc++ (Linux) | `libstdcxx-ng >= 12.1.0` (installed via conda-forge) |

The installer writes its decisions to `.installer_state.json` (`gpu_choice`,
`last_installed_commit`, `installed_wheels`) so updates only re-download changed wheels.

### 3.1 Requirements layout

- `requirements/full/requirements.txt` — NVIDIA CUDA 12.8 (the default full install).
- `requirements/full/requirements_amd.txt`, `_apple_silicon.txt`, `_apple_intel.txt`,
  `_cpu_only.txt` — per-hardware variants selected by `get_requirements_file()`.
- `requirements/portable/requirements.txt` — the slim, llama.cpp-only portable set.

The full set pins a **custom Gradio fork** (`oobabooga/gradio` 4.37.2+custom.x, installed from
a wheel URL, not PyPI), plus prebuilt CUDA wheels for `llama_cpp_binaries`,
`ik_llama_cpp_binaries`, `exllamav3`, and `flash_attn`.

> [!IMPORTANT]
> Because Gradio is a pinned fork, do **not** `pip install gradio` over it — you'll break the
> UI. Treat the requirements files as authoritative and let the update wizard manage them.

---

## 4. Configuration Mastery

### 4.1 Precedence model

```
command-line flags  ►  user_data/CMD_FLAGS.txt  ►  user_data/settings.yaml  ►  built-in defaults
   (highest)                                                                      (lowest)
```

- **`CMD_FLAGS.txt`** — persistent CLI flags, one per line, `#` comments allowed. Prepended to
  `sys.argv` so real command-line flags still win.
- **`settings.yaml`** — persisted interface settings (the `settings` dict in `shared.py`).
  Auto-loaded from `user_data/settings.yaml` without needing `--settings`.
- **`config-user.yaml`** (in the model dir) — per-model setting overrides loaded by
  `load_user_config()`.

### 4.2 The `user_data/` directory

`resolve_user_data_dir()` picks the directory in this order: `--user-data-dir` flag →
(portable+electron) `../../user_data` → (portable) `../user_data` → in-tree `user_data`.

| Subfolder / file | Contents |
|---|---|
| `models/` | Text model weights (GGUF, safetensors, …) |
| `loras/` | LoRA adapters |
| `characters/` | Character cards (YAML/JSON + optional images) |
| `presets/` | Sampler presets (`<name>.yaml`) |
| `instruction-templates/` | Jinja2 chat/instruction templates |
| `grammars/` | GBNF grammar files |
| `image_models/`, `mmproj/` | Diffusion image models; multimodal projectors |
| `logs/` | Chat logs and notebook prompts |
| `cache/` | Gradio temp, Electron data, profile-picture cache, disk KV cache |
| `settings.yaml`, `CMD_FLAGS.txt`, `mcp.json` | Config files |

> [!TIP]
> `user_data/` is the **only** thing you need to back up. It is never modified by the
> updater. To migrate an install, copy this folder.

### 4.3 Full CLI reference

Flags are defined in `modules/shared.py`. Grouped by argparse section:

**Basic**

| Flag | Default | Purpose |
|---|---|---|
| `--user-data-dir` | auto | Override the user-data directory |
| `--model` | — | Model to load on startup |
| `--lora` | — | LoRA(s) to apply (space-separated) |
| `--model-dir`, `--lora-dir` | `user_data/…` | Model/LoRA directories |
| `--model-menu` | off | Terminal model picker at launch |
| `--settings` | — | Path to a settings YAML |
| `--extensions` | — | Extensions to load (space-separated) |
| `--verbose` | off | Print prompts to the terminal |
| `--idle-timeout` | `0` | Minutes of inactivity before auto-unload (0 = never) |
| `--multi-user` | off | Stateless multi-user mode (small trusted teams only) |

**Model loader & context**

| Flag | Default | Purpose |
|---|---|---|
| `--loader` | auto | Force a loader: `llama.cpp`, `Transformers`, `ExLlamav3_HF`, `ExLlamav3`, `TensorRT-LLM` |
| `--ctx-size` / `--n_ctx` / `--max_seq_len` | `0` | Context tokens (0 = auto for llama.cpp, else 8192) |
| `--cache-type` | `fp16` | KV cache type (`fp16`, `q8_0`, `q4_0`; ExLlamaV3 `q2`–`q8`) |

**llama.cpp**

| Flag | Default | Purpose |
|---|---|---|
| `--gpu-layers` / `--n-gpu-layers` | `-1` | Layers on GPU (-1 = auto-fit) |
| `--cpu-moe` | off | Keep MoE experts on CPU |
| `--mmproj` | — | Multimodal projector for vision models |
| `--tensor-split` | — | Multi-GPU split proportions (e.g. `60,40`) |
| `--split-mode` | `layer` | `layer`/`row`/`tensor`/`none` |
| `--batch-size`, `--ubatch-size` | `1024` | App-level / device-level batch sizes |
| `--threads`, `--threads-batch` | `0` | CPU threads (0 = auto) |
| `--parallel` | `1` | Parallel request slots (context divided among them) |
| `--fit-target` | `512` | Target VRAM margin (MiB) for auto GPU layers |
| `--extra-flags` | — | Raw flags passed through to `llama-server` |
| `--ik` | off | Use `ik_llama.cpp` instead of upstream (needs its wheel) |
| `--streaming-llm`, `--no-mmap`, `--mlock`, `--no-kv-offload`, `--numa` | off | Memory/eval tuning |

**Speculative decoding**

| Flag | Default | Purpose |
|---|---|---|
| `--model-draft` | — | Draft model for speculative decoding |
| `--draft-max` | `3` | Tokens to draft per step |
| `--spec-type` | `none` | `draft-mtp`, `ngram-mod`, `ngram-simple`, `ngram-map-k`, `ngram-map-k4v` |
| `--gpu-layers-draft` | `256` | Draft-model GPU layers |
| `--ctx-size-draft` | `0` | Draft context (0 = same as main) |

**Transformers / Accelerate / bitsandbytes**

| Flag | Default | Purpose |
|---|---|---|
| `--cpu` | off | CPU-only generation |
| `--cpu-memory` | `0` | Max CPU memory (GiB) for offload |
| `--disk` | off | Offload overflow layers to disk |
| `--load-in-8bit`, `--load-in-4bit` | off | bitsandbytes quantization |
| `--use_double_quant`, `--compute_dtype`, `--quant_type` | — | 4-bit tuning (`nf4`/`fp4`) |
| `--bf16` | off | bfloat16 (Ampere+) |
| `--attn-implementation` | `sdpa` | `sdpa`, `eager`, `flash_attention_2` |
| `--trust-remote-code` | off | **Allows model code execution** — see §6 |
| `--force-safetensors` | off | Refuse non-safetensors (prevents arbitrary code) |
| `--no_use_fast` | off | Disable the fast tokenizer |

**ExLlamaV3**

| Flag | Default | Purpose |
|---|---|---|
| `--gpu-split` | — | Per-GPU VRAM (GB) for layers (e.g. `20,7,7`) |
| `--enable-tp` / `--enable_tp` | off | Tensor parallelism across GPUs |
| `--tp-backend` | `native` | `native` or `nccl` |
| `--cfg-cache` | off | Extra cache for CFG negative prompts |

**Gradio / network**

| Flag | Default | Purpose |
|---|---|---|
| `--listen` | off | Bind to the LAN (`0.0.0.0` or `--listen-host`) |
| `--listen-port` | — | UI port |
| `--share` | off | Public Gradio share tunnel (proprietary binary) |
| `--auto-launch` | off | Open a browser on launch |
| `--gradio-auth` / `--gradio-auth-path` | — | HTTP Basic auth (`user:pass[,user:pass…]`) |
| `--ssl-keyfile`, `--ssl-certfile` | — | Serve the UI over HTTPS |
| `--subpath` | — | Mount under a subpath (reverse-proxy) |
| `--portable` | off | Hide non-portable features (training, image gen, `ik`) |

**API**

| Flag | Default | Purpose |
|---|---|---|
| `--api` | off | Enable the HTTP API |
| `--api-port` | `5000` | API port |
| `--api-key` | — | Bearer key for all endpoints |
| `--admin-key` | — | Separate key for model load/unload (defaults to `--api-key`) |
| `--public-api` / `--public-api-id` | off | Cloudflare tunnel for the API |
| `--api-enable-ipv6` / `--api-disable-ipv4` | off | Address family control |
| `--nowebui` | off | Run the API without the Gradio UI |

**Image model**

`--image-model`, `--image-model-dir`, `--image-dtype` (`bfloat16`/`float16`),
`--image-attn-backend` (`sdpa`/`flash_attention_2`), `--image-cpu-offload`, `--image-compile`,
`--image-quant` (`none`/`bnb-8bit`/`bnb-4bit`/`torchao-*`).

**API generation defaults** — every sampler (`--temperature`, `--top-p`, `--top-k`,
`--min-p`, `--repetition-penalty`, `--dry-multiplier`, `--mirostat-mode`, `--enable-thinking`,
`--reasoning-effort`, `--chat-template-file`, …) can be set as a CLI default; see §5.2.

> [!NOTE]
> This is the complete top-level flag surface. Sampler flags map 1:1 to the `settings` dict
> keys documented in §5.2. Flags not listed here **do not exist** — don't assume undocumented
> options.

### 4.4 Environment variables

| Variable | Used by | Effect |
|---|---|---|
| `HF_TOKEN` | `download-model.py` | Hugging Face auth token for gated/private models |
| `HF_USER` / `HF_PASS` | `download-model.py` | HTTP Basic auth alternative |
| `HF_ENDPOINT` | `download-model.py` | Override the Hugging Face base URL (mirror) |
| `GPU_CHOICE` | `one_click.py` | Non-interactive GPU selection (`A`–`D`, `N`) |
| `INSTALL_EXTENSIONS` | `one_click.py` | If truthy, install all extension requirements |
| `LAUNCH_AFTER_INSTALL` | `one_click.py` | If falsey, exit after install instead of launching |
| `OPENEDAI_EMBEDDING_MODEL` | API | Embedding model (default `sentence-transformers/all-mpnet-base-v2`) |
| `OPENEDAI_EMBEDDING_DEVICE` | API | Embedding device (default `cpu`) |
| `OPENEDAI_PORT` | API | Override the API port |
| `OPENEDAI_CERT_PATH` / `OPENEDAI_KEY_PATH` | API | API TLS cert/key |
| `OPENEDAI_ENABLE_IPV6` / `OPENEDAI_DISABLE_IPV4` | API | API address family |
| `OPENEDAI_DEBUG` | API | Verbose API debug logging (default `0`) |
| `TEXTGEN_ELECTRON` | UI | Set to `1` by the desktop wrapper to enable Electron-specific behavior |

---

## 5. Loaders & Samplers (the pluggable backends)

### 5.1 Loader capability matrix

Each loader supports a different set of load-time parameters, defined in
`loaders_and_params` (`modules/loaders.py`). `modules/models_settings.py` auto-detects the
loader from the model's files; `--loader` overrides it (`fix_loader_name()` normalizes
aliases like `hf`, `llamacpp`, `exllama3`).

| Loader | Process model | Quantization support | Multi-GPU | Speculative decoding | Vision |
|---|---|---|---|---|---|
| **llama.cpp** | Subprocess (`llama-server`) | GGUF (built-in) | `--tensor-split`/`--split-mode` | ✅ (draft + ngram) | ✅ (`--mmproj`) |
| **Transformers** | In-process | 8-bit/4-bit (bitsandbytes), bf16 | `--gpu-split`, disk/CPU offload | ❌ | via model |
| **ExLlamav3_HF** | In-process | EXL3 + cache quant | `--gpu-split`, `--enable-tp` | ❌ | ❌ |
| **ExLlamav3** | In-process | EXL3 + cache quant | `--gpu-split`, `--enable-tp` | ✅ (draft) | ❌ |
| **TensorRT-LLM** | In-process | engine-baked | engine-baked | ❌ | ❌ |

> [!IMPORTANT]
> The llama.cpp loader's out-of-process design means crashes show up as
> *"Server process terminated unexpectedly with exit code: N"* and a lingering
> `llama-server` can hold VRAM after an unclean exit. Supervise/kill that process in
> production deployments.

### 5.2 Sampler availability per loader

This matrix has no equivalent in the project README and is the highest-value reference for
anyone tuning generation. Defined in `loaders_samplers` (`modules/loaders.py`):

| Sampler family | Transformers | ExLlamav3_HF | ExLlamav3 | llama.cpp | TensorRT-LLM |
|---|:--:|:--:|:--:|:--:|:--:|
| temperature, top_p, top_k, min_p | ✅ | ✅ | ✅ | ✅ | ✅ |
| dynatemp / smoothing | ✅ | ✅ | ❌ | partial | ❌ |
| typical_p, top_a, top_n_sigma | ✅ | ✅ | ❌ | partial | ❌ |
| XTC (xtc_threshold/probability) | ✅ | ✅ | ❌ | ✅ | ❌ |
| epsilon/eta cutoff, TFS | ✅ | ✅ | ❌ | ❌ | ❌ |
| DRY (dry_multiplier/base/length) | ✅ | ✅ | ❌ | ✅ | ❌ |
| repetition/frequency/presence penalty | ✅ | ✅ | ✅ | ✅ | ✅ |
| mirostat | ✅ | ✅ | ❌ | ✅ | ❌ |
| guidance_scale (CFG), penalty_alpha | ✅ | ✅ | ❌ | ❌ | ❌ |
| grammar (GBNF) | ✅ | ✅ | ❌ | ✅ | ❌ |
| prompt_lookup_num_tokens | ✅ | ❌ | ❌ | ❌ | ❌ |

The UI calls `blacklist_samplers()` / `make_loader_params_visible()` to hide unsupported
controls. **Setting an unsupported sampler over the API is silently ignored** — it is not an
error, but it has no effect. Treat the table above as the negative-knowledge boundary.

### 5.3 Default sampler values

Neutral defaults come from `default_preset_values` (`modules/presets.py`): `temperature=1`,
`top_p=1`, `top_k=0`, `min_p=0`, `repetition_penalty=1`, `dry_multiplier=0`,
`mirostat_mode=0`, `repetition_penalty_range=1024`, and the `sampler_priority` ordering that
determines which samplers apply first. Presets in `user_data/presets/*.yaml` override these;
`generate_preset_yaml()` saves only the non-default deltas.

### 5.4 Reasoning-model parsing

`modules/reasoning.py` strips "thinking" blocks before display using a `THINKING_FORMATS`
table of model-family-specific tag pairs: `<think>…</think>`, GPT-OSS
`<|channel|>analysis<|message|>…<|end|>`, Gemma 4 `<|channel>thought`, Seed `<seed:think>`,
Solar `<|think|>`, and an end-only `</think>` variant. Parsing is **heuristic per family** —
a model that uses an unrecognized marker will show its reasoning inline. There is no universal
thinking format.

---

## 6. Security

> [!CAUTION]
> TextGen is designed to run on a trusted machine for a trusted user. It is **not** a
> hardened multi-tenant service. Every "exposure" flag widens the trust boundary — understand
> each before enabling it.

### 6.1 Network exposure model

- **UI** is local by default. `--listen` binds it to the LAN; `--share` creates a public
  tunnel using a proprietary binary. With either, set `--gradio-auth user:pass` — the startup
  code warns loudly if you don't.
- **API** is **localhost-locked** by two mechanisms in `modules/api/script.py`:
  1. A host-header validation middleware that rejects any `Host` other than
     `localhost`/`127.0.0.1` unless `--listen`/`--public-api` is set
     (returns `{"detail": "Invalid host header"}`).
  2. A CORS policy that, when not exposed, only allows `localhost`/`127.0.0.1` origins.
- Exposing the API requires `--listen` or `--public-api`. When exposed, **always** set
  `--api-key`.

### 6.2 Authentication

| Surface | Mechanism |
|---|---|
| Web UI | Gradio HTTP Basic auth via `--gradio-auth` / `--gradio-auth-path` |
| API (general) | `Authorization: Bearer <api-key>` (`verify_api_key`) |
| API (admin: model load/unload) | `Authorization: Bearer <admin-key>` (`verify_admin_key`) |
| API (Anthropic `/v1/messages`) | `x-api-key: <api-key>` (`verify_anthropic_key`) |

> [!IMPORTANT]
> Set `--admin-key` to something **different** from `--api-key` in any shared deployment.
> Otherwise every API client that can generate text can also unload/swap your model
> (the model/load endpoint controls what the whole instance serves). Issue #7577 documents
> that `/v1/internal/model/load` honors loader args — an admin-only capability by design.

### 6.3 Code-execution risks

- **`--trust-remote-code`** runs arbitrary Python shipped with a model. The startup warning is
  explicit: only use it with fully trusted models, and gate access with `--gradio-auth`.
  Prefer `--force-safetensors` to refuse non-safetensors weights entirely.
- **Tools / MCP** (`modules/tool_use.py`, `user_data/mcp.json`) let the model invoke external
  commands/servers. Enable `confirm_tool_calls` so calls require user approval.

### 6.4 SSRF protection in web search

`modules/web_search.py` validates every URL (and every redirect hop) before fetching:
rejects backslashes (the GHSA-27xf-58m5-vxmc parser-discrepancy bypass), non-`http(s)`
schemes, embedded credentials, and any hostname that resolves to a **non-global** IP. This
prevents the web-search feature from being steered at internal/metadata endpoints.

### 6.5 Multi-user caveats

`--multi-user` is for **small trusted teams only**. The startup warning enumerates the
limits: the Stop button stops generation for everyone, chat history isn't saved, only one user
generates at a time unless the backend is parallel-capable (llama.cpp `--parallel N`,
ExLlamaV3), and grayed-out UI actions can be bypassed client-side. **Do not expose publicly.**

---

## 7. The API Server (internals)

`modules/api/script.py` builds a FastAPI app served by uvicorn (default port 5000, overridable
via `--api-port`/`OPENEDAI_PORT`). It is started in-process from `server.py` when `--api` is
set, or standalone with `--nowebui`.

### 7.1 Endpoint inventory

| Method | Path | Auth | Purpose |
|---|---|---|---|
| POST | `/v1/completions` | api-key | OpenAI text completion (streaming via SSE) |
| POST | `/v1/chat/completions` | api-key | OpenAI chat completion |
| POST | `/v1/messages` | x-api-key | Anthropic Messages API |
| GET | `/v1/models`, `/v1/models/{model}` | api-key | List/describe models |
| POST | `/v1/audio/transcriptions` | api-key | Whisper speech-to-text |
| POST | `/v1/images/generations` | api-key | Diffusion image generation |
| POST | `/v1/embeddings` | api-key | Sentence-embedding vectors |
| POST | `/v1/moderations` | api-key | Moderation scoring |
| GET | `/v1/internal/health` | api-key | Health check |
| POST | `/v1/internal/encode`, `/decode`, `/token-count` | api-key | Tokenizer utilities |
| POST | `/v1/internal/logits` | api-key | Raw next-token logits |
| POST | `/v1/internal/chat-prompt` | api-key | Render the chat prompt without generating |
| POST | `/v1/internal/stop-generation` | api-key | Cancel an in-flight generation |
| GET | `/v1/internal/model/info`, `/list` | api-key | Loaded-model info / available models |
| POST | `/v1/internal/model/load`, `/unload` | **admin-key** | Load/unload a model |
| GET/POST | `/v1/internal/lora/list`, `/load`, `/unload` | api-key / admin | LoRA management |

### 7.2 Streaming, cancellation, and errors

- Streaming responses use `sse-starlette`'s `EventSourceResponse`; the server watches for
  client disconnects and sets a stop event so generation halts when a client goes away.
- `n > 1` is rejected with streaming enabled.
- Error handlers translate internal exceptions into OpenAI-shaped
  (`{"error": {"message", "type", …}}`) or Anthropic-shaped
  (`{"type": "error", "error": {…}}`) bodies depending on the route.

### 7.3 Embeddings & TLS

The embedding endpoint loads a sentence-transformers model
(`OPENEDAI_EMBEDDING_MODEL`, default `all-mpnet-base-v2`) on `OPENEDAI_EMBEDDING_DEVICE`. The
API can serve TLS directly via `OPENEDAI_CERT_PATH`/`OPENEDAI_KEY_PATH`. If neither IPv4 nor
IPv6 is enabled it refuses to start: *"you MUST enable IPv6 or IPv4 for the API to work."*

---

## 8. Extensions

Extensions are folders under `extensions/<name>/` with a `script.py` exposing hook functions
(`setup`, `ui`, plus generation hooks like `input_modifier`, `output_modifier`,
`custom_generate_chat_prompt`). They are discovered by `modules/extensions.py`, activated via
`--extensions` or `settings['default_extensions']`, and their CSS/JS is merged into the UI.

| Extension | Purpose |
|---|---|
| `coqui_tts`, `silero_tts` | Text-to-speech |
| `whisper_stt` | Speech-to-text input |
| `sd_api_pictures` | Generate images via an external Stable Diffusion API |
| `send_pictures` | Send images into multimodal chats |
| `superbooga`, `superboogav2` | ChromaDB-backed retrieval/RAG long-term memory |
| `google_translate` | Translate input/output |
| `ngrok` | Expose the UI via an ngrok tunnel |
| `gallery` | Character gallery picker |
| `character_bias`, `long_replies`, `perplexity_colors` | Generation tweaks/visualizations |
| `example` | Template for writing your own extension |

Each extension declares its own `requirements.txt`; install them via the update wizard
(option **B**), `INSTALL_EXTENSIONS=1` at install time, or directly with pip. Failures are
non-fatal — a missing requirement logs *"Failed to load the extension …"* and the rest of the
app continues.

> [!WARNING]
> `superboogav2` pulls heavy dependencies (chromadb, spacy, nltk, PyMuPDF, optuna). Install it
> only if you need RAG memory; on Windows some extension requirements are expected to fail
> harmlessly.

---

## 9. Operations

### 9.1 Running headless / as a service

For a server deployment, run the API without the UI:

```bash
python server.py --nowebui --api --api-key "$API_KEY" --admin-key "$ADMIN_KEY" \
  --model my-model --listen --idle-timeout 30
```

Wrap it in a systemd unit (or the appropriate Docker image under `docker/<vendor>/`). The
process supervises its own `llama-server` child; ensure your service manager sends SIGTERM
(handled gracefully) rather than SIGKILL so the subprocess is stopped cleanly.

### 9.2 Docker

Per-vendor Dockerfiles live under `docker/` (`nvidia`, `amd`, `intel`, `cpu`,
`TensorRT-LLM`), each with a `docker-compose.yml`. Mount `user_data/` as a volume so models
and config persist across container rebuilds.

### 9.3 Reverse proxy

Behind nginx/Traefik, set `--subpath /textgen` so Gradio generates correct asset URLs, and
forward WebSocket upgrades (Gradio's queue uses them). Note issue #7545: heavy proxying
(e.g. Cloudflare in front of a remote host) can degrade UI responsiveness — prefer a direct
`--listen` path on a trusted network when latency matters.

### 9.4 Health, logs, and restarts

- Health: `GET /v1/internal/health` (when the API is enabled).
- Logs: Rich-formatted to the console; a 100-entry in-memory ring buffer is also kept
  (`modules/logging_colors.py`). Third-party loggers (urllib3, httpx, diffusers, torch) are
  quieted to ERROR.
- Hot restart: setting `shared.need_restart` rebuilds the Gradio interface without exiting the
  process (used by settings changes that require a UI reload).
- Idle unload: `--idle-timeout N` runs a daemon thread that unloads the model after N minutes
  idle and reloads on demand — useful for shared GPUs.

### 9.5 Backup & migration

Back up `user_data/`. To migrate, copy that directory to the new host and reinstall the
application around it. Model checksums can be re-validated with
`python download-model.py <repo> --check`.

---

## 10. Code Quality & Conventions

- **Global singleton state.** `modules/shared.py` holds the model, tokenizer, settings, and
  Gradio component registry as module globals. This is deliberate (single-user desktop app)
  but means there is no isolation between requests — relevant when reasoning about
  `--multi-user`.
- **Sampler hijacking.** `modules/sampler_hijack.py` patches custom samplers (TFS, top-a,
  mirostat variants, DRY, adaptive) into the Transformers generation path, validating ranges
  and raising `ValueError` on bad values. This is the in-process analogue of the
  monkey-patching pattern common to Gradio ML UIs — upgrading `transformers` can silently
  break these patches; smoke-test generation after a Transformers bump.
- **Loader normalization.** `fix_loader_name()` accepts many spellings of each loader name;
  prefer the canonical forms in scripts.
- **Custom Gradio fork.** The UI depends on `oobabooga/gradio` (a pinned fork with custom
  patches), not upstream Gradio. Front-end behavior won't match mainline Gradio docs.
- **Front-end is imperative DOM manipulation.** `js/main.js` wires keyboard shortcuts,
  sidebars, hover menus, syntax highlighting (highlight.js), and math (KaTeX) directly against
  the rendered Gradio DOM via `MutationObserver`s — fragile across Gradio upgrades, another
  reason the fork is pinned.

---

## 11. Potential Improvements

These are observations for maintainers, derived from the source — not defects.

- **Single global model.** Serving more than one model simultaneously (beyond the separate
  image model) would require refactoring `shared.model` into a registry. Today, `model/load`
  swaps the one global model for all clients.
- **DOM-coupled front-end.** The reliance on `MutationObserver`-driven DOM patching against a
  forked Gradio is the main maintenance tax. A move toward Gradio-native components or a
  custom front-end would reduce upgrade breakage.
- **Heuristic reasoning parser.** `THINKING_FORMATS` must be extended by hand for each new
  model family's thinking markup; a model-declared format (via metadata) would be more robust.
- **Per-loader sampler drift.** The `loaders_samplers` sets are maintained manually; a model
  could request a sampler the loader silently ignores. Surfacing "ignored sampler" warnings on
  the API would aid debugging.
- **Extension requirement fragility.** Extensions fail open (logged, skipped). A dependency
  health endpoint listing which extensions loaded vs. failed would help operators.

---

## 12. Quick Reference

**Start (full, NVIDIA):** `./start_linux.sh` → pick **A** → Model tab → download → load.

**Start (API only):**
`python server.py --nowebui --api --api-key KEY --admin-key ADMIN --model NAME`

**Persistent flags:** `user_data/CMD_FLAGS.txt` (one per line).

**Persistent UI settings:** `user_data/settings.yaml`.

**Config precedence:** CLI → CMD_FLAGS.txt → settings.yaml → defaults.

**Loaders:** `llama.cpp` (GGUF, subprocess) · `Transformers` · `ExLlamav3_HF` · `ExLlamav3` ·
`TensorRT-LLM`.

**Default API port:** 5000. **Localhost-locked** unless `--listen`/`--public-api`.

**Auth headers:** `Authorization: Bearer <key>` (OpenAI) · `x-api-key: <key>` (Anthropic) ·
admin-key for `model/load|unload`.

**State to back up:** `user_data/` (never touched by updates).

**Update:** `update_wizard_*` → **A**. Major Python/CUDA bumps require deleting
`installer_files/` and reinstalling.

**Key env vars:** `HF_TOKEN`, `GPU_CHOICE`, `OPENEDAI_*`, `TEXTGEN_ELECTRON`.

---

_Generated June 2026 from oobabooga/textgen at ed888c7. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._