TextGen (text-generation-webui) Admin Guide

This guide is for operators, developers, and architects who deploy, secure, extend, or maintain TextGen. It covers the system architecture, the complete configuration surface, the loader and sampler backends, the API server, security model, operations, and internals deep enough to reason about — or contribute to — the codebase.

For end-user, install, and day-to-day usage, see the companion User Guide.

1. Executive Summary

TextGen is a Python application that wraps multiple local LLM inference runtimes behind a single Gradio web UI and an optional FastAPI OpenAI/Anthropic-compatible HTTP server. It is a desktop/self-hosted product, not a multi-tenant service: there is no database, no user accounts beyond optional HTTP Basic auth, and the unit of state is the user_data/ directory on disk.

Architecturally it is three layers:

1. A launcher/installer layer (one_click.py, start_*.sh/.bat, desktop/main.js) that provisions an isolated Python 3.13 + PyTorch environment per hardware target, or ships as a self-contained portable build.
2. An orchestration layer (server.py + modules/) that parses configuration, loads a model through one of five swappable loaders, builds the Gradio interface, and manages generation.
3. A serving layer (modules/api/) that re-exposes the loaded model as OpenAI- and Anthropic-compatible HTTP endpoints.

Key design facts an operator must know:

• The active model is global, singleton state (modules/shared.py:model). One model is loaded at a time (image models load separately and concurrently).
• The llama.cpp loader is out-of-process: it spawns and supervises a llama-server subprocess. Clean shutdown explicitly stops it.
• Configuration precedence is command line → CMD_FLAGS.txt → settings.yaml → built-in defaults.
• The API is localhost-locked by default and only opens to the network on explicit opt-in.

2. Architecture

2.1 Component overview

                         ┌──────────────────────────────────────────────┐
                         │  Launcher / Installer                        │
   start_windows.bat ───►│  one_click.py: conda env + PyTorch per-GPU   │
   start_linux.sh        │  desktop/main.js: Electron window (portable) │
                         └───────────────────┬──────────────────────────┘
                                             │ python server.py [flags]
                                             ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  server.py  (entry point)                                                 │
   │   • modules/shared.py   → argparse CLI, settings dict, global state       │
   │   • modules/paths.py    → resolve user_data dir                           │
   │   • loads settings.yaml, CMD_FLAGS.txt, mcp.json                          │
   └───────────────┬───────────────────────────────────┬──────────────────────┘
                   │ create_interface()                 │ if --api / --nowebui
                   ▼                                     ▼
   ┌───────────────────────────────────┐   ┌────────────────────────────────────┐
   │  Gradio UI (modules/ui_*.py)      │   │  FastAPI server (modules/api/)     │
   │  Tabs: Chat, Notebook, Parameters,│   │  uvicorn on :5000                  │
   │  Character, Model, Image, Train,  │   │  /v1/* OpenAI + /v1/messages       │
   │  Session                          │   │  Anthropic                         │
   └───────────────┬───────────────────┘   └────────────────┬───────────────────┘
                   │                                         │
                   └──────────────┬──────────────────────────┘
                                  ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  Generation core                                                          │
   │   modules/text_generation.py · modules/chat.py · modules/sampler_hijack  │
   └───────────────┬──────────────────────────────────────────────────────────┘
                   ▼  modules/models.py dispatches to one loader
   ┌──────────────┬──────────────┬───────────────┬─────────────┬──────────────┐
   │  llama.cpp   │ Transformers │ ExLlamav3_HF  │ ExLlamav3   │ TensorRT-LLM │
   │ (subprocess) │ (in-process) │ (in-process)  │ (in-process)│ (in-process) │
   └──────────────┴──────────────┴───────────────┴─────────────┴──────────────┘
                   ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  user_data/  (the entire persistent state surface)                        │
   │  models · loras · characters · presets · instruction-templates · grammars │
   │  logs · cache · settings.yaml · CMD_FLAGS.txt · mcp.json                   │
   └──────────────────────────────────────────────────────────────────────────┘

2.2 Repository structure

2.3 Key modules

2.4 The startup sequence (server.py)

1. Install SIGINT/SIGTERM handlers (second Ctrl-C forces exit; explicitly stops a running LlamaServer).
2. do_cmd_flags_warnings() — emit security warnings (see §6).
3. Load settings.yaml (CLI --settings path, else user_data/settings.yaml if present).
4. Detect MCP config (user_data/mcp.json) and image-model CLI overrides.
5. Activate extensions from settings['default_extensions'] (the legacy openai extension is aliased to --api).
6. Resolve and optionally load the model (--model, or --model-menu for a terminal picker).
7. If --idle-timeout > 0, start the idle-unload watchdog thread.
8. If --nowebui, start only the API; otherwise create_interface() builds Gradio, starts the API if requested, and enters a restart loop watching shared.need_restart.

Note

The model is loaded before the UI is built. A bad --model or out-of-memory condition surfaces at startup, not on first chat.

3. Requirements & Environment

The installer writes its decisions to .installer_state.json (gpu_choice, last_installed_commit, installed_wheels) so updates only re-download changed wheels.

3.1 Requirements layout

• requirements/full/requirements.txt — NVIDIA CUDA 12.8 (the default full install).
• requirements/full/requirements_amd.txt, _apple_silicon.txt, _apple_intel.txt, _cpu_only.txt — per-hardware variants selected by get_requirements_file().
• requirements/portable/requirements.txt — the slim, llama.cpp-only portable set.

The full set pins a custom Gradio fork (oobabooga/gradio 4.37.2+custom.x, installed from a wheel URL, not PyPI), plus prebuilt CUDA wheels for llama_cpp_binaries, ik_llama_cpp_binaries, exllamav3, and flash_attn.

Important

Because Gradio is a pinned fork, do not pip install gradio over it — you'll break the UI. Treat the requirements files as authoritative and let the update wizard manage them.

4. Configuration Mastery

4.1 Precedence model

command-line flags  ►  user_data/CMD_FLAGS.txt  ►  user_data/settings.yaml  ►  built-in defaults
   (highest)                                                                      (lowest)

• CMD_FLAGS.txt — persistent CLI flags, one per line, # comments allowed. Prepended to sys.argv so real command-line flags still win.
• settings.yaml — persisted interface settings (the settings dict in shared.py). Auto-loaded from user_data/settings.yaml without needing --settings.
• config-user.yaml (in the model dir) — per-model setting overrides loaded by load_user_config().

4.2 The user_data/ directory

resolve_user_data_dir() picks the directory in this order: --user-data-dir flag → (portable+electron) ../../user_data → (portable) ../user_data → in-tree user_data.

Tip

user_data/ is the only thing you need to back up. It is never modified by the updater. To migrate an install, copy this folder.

4.3 Full CLI reference

Flags are defined in modules/shared.py. Grouped by argparse section:

Basic

Model loader & context

llama.cpp

Speculative decoding

Transformers / Accelerate / bitsandbytes

ExLlamaV3

Gradio / network

API

Image model

--image-model, --image-model-dir, --image-dtype (bfloat16/float16), --image-attn-backend (sdpa/flash_attention_2), --image-cpu-offload, --image-compile, --image-quant (none/bnb-8bit/bnb-4bit/torchao-*).

API generation defaults — every sampler (--temperature, --top-p, --top-k, --min-p, --repetition-penalty, --dry-multiplier, --mirostat-mode, --enable-thinking, --reasoning-effort, --chat-template-file, …) can be set as a CLI default; see §5.2.

Note

This is the complete top-level flag surface. Sampler flags map 1:1 to the settings dict keys documented in §5.2. Flags not listed here do not exist — don't assume undocumented options.

4.4 Environment variables

5. Loaders & Samplers (the pluggable backends)

5.1 Loader capability matrix

Each loader supports a different set of load-time parameters, defined in loaders_and_params (modules/loaders.py). modules/models_settings.py auto-detects the loader from the model's files; --loader overrides it (fix_loader_name() normalizes aliases like hf, llamacpp, exllama3).

Important

The llama.cpp loader's out-of-process design means crashes show up as "Server process terminated unexpectedly with exit code: N" and a lingering llama-server can hold VRAM after an unclean exit. Supervise/kill that process in production deployments.

5.2 Sampler availability per loader

This matrix has no equivalent in the project README and is the highest-value reference for anyone tuning generation. Defined in loaders_samplers (modules/loaders.py):

The UI calls blacklist_samplers() / make_loader_params_visible() to hide unsupported controls. Setting an unsupported sampler over the API is silently ignored — it is not an error, but it has no effect. Treat the table above as the negative-knowledge boundary.

5.3 Default sampler values

Neutral defaults come from default_preset_values (modules/presets.py): temperature=1, top_p=1, top_k=0, min_p=0, repetition_penalty=1, dry_multiplier=0, mirostat_mode=0, repetition_penalty_range=1024, and the sampler_priority ordering that determines which samplers apply first. Presets in user_data/presets/*.yaml override these; generate_preset_yaml() saves only the non-default deltas.

5.4 Reasoning-model parsing

modules/reasoning.py strips "thinking" blocks before display using a THINKING_FORMATS table of model-family-specific tag pairs: <think>…</think>, GPT-OSS <|channel|>analysis<|message|>…<|end|>, Gemma 4 <|channel>thought, Seed <seed:think>, Solar <|think|>, and an end-only </think> variant. Parsing is heuristic per family — a model that uses an unrecognized marker will show its reasoning inline. There is no universal thinking format.

6. Security

Caution

TextGen is designed to run on a trusted machine for a trusted user. It is not a hardened multi-tenant service. Every "exposure" flag widens the trust boundary — understand each before enabling it.

6.1 Network exposure model

• UI is local by default. --listen binds it to the LAN; --share creates a public tunnel using a proprietary binary. With either, set --gradio-auth user:pass — the startup code warns loudly if you don't.
• API is localhost-locked by two mechanisms in modules/api/script.py:
  1. A host-header validation middleware that rejects any Host other than localhost/127.0.0.1 unless --listen/--public-api is set (returns {"detail": "Invalid host header"}).
  2. A CORS policy that, when not exposed, only allows localhost/127.0.0.1 origins.
• Exposing the API requires --listen or --public-api. When exposed, always set --api-key.

6.2 Authentication

Important

Set --admin-key to something different from --api-key in any shared deployment. Otherwise every API client that can generate text can also unload/swap your model (the model/load endpoint controls what the whole instance serves). Issue #7577 documents that /v1/internal/model/load honors loader args — an admin-only capability by design.

6.3 Code-execution risks

• --trust-remote-code runs arbitrary Python shipped with a model. The startup warning is explicit: only use it with fully trusted models, and gate access with --gradio-auth. Prefer --force-safetensors to refuse non-safetensors weights entirely.
• Tools / MCP (modules/tool_use.py, user_data/mcp.json) let the model invoke external commands/servers. Enable confirm_tool_calls so calls require user approval.

6.4 SSRF protection in web search

modules/web_search.py validates every URL (and every redirect hop) before fetching: rejects backslashes (the GHSA-27xf-58m5-vxmc parser-discrepancy bypass), non-http(s) schemes, embedded credentials, and any hostname that resolves to a non-global IP. This prevents the web-search feature from being steered at internal/metadata endpoints.

6.5 Multi-user caveats

--multi-user is for small trusted teams only. The startup warning enumerates the limits: the Stop button stops generation for everyone, chat history isn't saved, only one user generates at a time unless the backend is parallel-capable (llama.cpp --parallel N, ExLlamaV3), and grayed-out UI actions can be bypassed client-side. Do not expose publicly.

7. The API Server (internals)

modules/api/script.py builds a FastAPI app served by uvicorn (default port 5000, overridable via --api-port/OPENEDAI_PORT). It is started in-process from server.py when --api is set, or standalone with --nowebui.

7.1 Endpoint inventory

7.2 Streaming, cancellation, and errors

• Streaming responses use sse-starlette's EventSourceResponse; the server watches for client disconnects and sets a stop event so generation halts when a client goes away.
• n > 1 is rejected with streaming enabled.
• Error handlers translate internal exceptions into OpenAI-shaped ({"error": {"message", "type", …}}) or Anthropic-shaped ({"type": "error", "error": {…}}) bodies depending on the route.

7.3 Embeddings & TLS

The embedding endpoint loads a sentence-transformers model (OPENEDAI_EMBEDDING_MODEL, default all-mpnet-base-v2) on OPENEDAI_EMBEDDING_DEVICE. The API can serve TLS directly via OPENEDAI_CERT_PATH/OPENEDAI_KEY_PATH. If neither IPv4 nor IPv6 is enabled it refuses to start: "you MUST enable IPv6 or IPv4 for the API to work."

8. Extensions

Extensions are folders under extensions/<name>/ with a script.py exposing hook functions (setup, ui, plus generation hooks like input_modifier, output_modifier, custom_generate_chat_prompt). They are discovered by modules/extensions.py, activated via --extensions or settings['default_extensions'], and their CSS/JS is merged into the UI.

Each extension declares its own requirements.txt; install them via the update wizard (option B), INSTALL_EXTENSIONS=1 at install time, or directly with pip. Failures are non-fatal — a missing requirement logs "Failed to load the extension …" and the rest of the app continues.

Warning

superboogav2 pulls heavy dependencies (chromadb, spacy, nltk, PyMuPDF, optuna). Install it only if you need RAG memory; on Windows some extension requirements are expected to fail harmlessly.

9. Operations

9.1 Running headless / as a service

For a server deployment, run the API without the UI:

python server.py --nowebui --api --api-key "$API_KEY" --admin-key "$ADMIN_KEY" \
  --model my-model --listen --idle-timeout 30

Wrap it in a systemd unit (or the appropriate Docker image under docker/<vendor>/). The process supervises its own llama-server child; ensure your service manager sends SIGTERM (handled gracefully) rather than SIGKILL so the subprocess is stopped cleanly.

9.2 Docker

Per-vendor Dockerfiles live under docker/ (nvidia, amd, intel, cpu, TensorRT-LLM), each with a docker-compose.yml. Mount user_data/ as a volume so models and config persist across container rebuilds.

9.3 Reverse proxy

Behind nginx/Traefik, set --subpath /textgen so Gradio generates correct asset URLs, and forward WebSocket upgrades (Gradio's queue uses them). Note issue #7545: heavy proxying (e.g. Cloudflare in front of a remote host) can degrade UI responsiveness — prefer a direct --listen path on a trusted network when latency matters.

9.4 Health, logs, and restarts

• Health: GET /v1/internal/health (when the API is enabled).
• Logs: Rich-formatted to the console; a 100-entry in-memory ring buffer is also kept (modules/logging_colors.py). Third-party loggers (urllib3, httpx, diffusers, torch) are quieted to ERROR.
• Hot restart: setting shared.need_restart rebuilds the Gradio interface without exiting the process (used by settings changes that require a UI reload).
• Idle unload: --idle-timeout N runs a daemon thread that unloads the model after N minutes idle and reloads on demand — useful for shared GPUs.

9.5 Backup & migration

Back up user_data/. To migrate, copy that directory to the new host and reinstall the application around it. Model checksums can be re-validated with python download-model.py <repo> --check.

10. Code Quality & Conventions

• Global singleton state. modules/shared.py holds the model, tokenizer, settings, and Gradio component registry as module globals. This is deliberate (single-user desktop app) but means there is no isolation between requests — relevant when reasoning about --multi-user.
• Sampler hijacking. modules/sampler_hijack.py patches custom samplers (TFS, top-a, mirostat variants, DRY, adaptive) into the Transformers generation path, validating ranges and raising ValueError on bad values. This is the in-process analogue of the monkey-patching pattern common to Gradio ML UIs — upgrading transformers can silently break these patches; smoke-test generation after a Transformers bump.
• Loader normalization. fix_loader_name() accepts many spellings of each loader name; prefer the canonical forms in scripts.
• Custom Gradio fork. The UI depends on oobabooga/gradio (a pinned fork with custom patches), not upstream Gradio. Front-end behavior won't match mainline Gradio docs.
• Front-end is imperative DOM manipulation. js/main.js wires keyboard shortcuts, sidebars, hover menus, syntax highlighting (highlight.js), and math (KaTeX) directly against the rendered Gradio DOM via MutationObservers — fragile across Gradio upgrades, another reason the fork is pinned.

11. Potential Improvements

These are observations for maintainers, derived from the source — not defects.

• Single global model. Serving more than one model simultaneously (beyond the separate image model) would require refactoring shared.model into a registry. Today, model/load swaps the one global model for all clients.
• DOM-coupled front-end. The reliance on MutationObserver-driven DOM patching against a forked Gradio is the main maintenance tax. A move toward Gradio-native components or a custom front-end would reduce upgrade breakage.
• Heuristic reasoning parser. THINKING_FORMATS must be extended by hand for each new model family's thinking markup; a model-declared format (via metadata) would be more robust.
• Per-loader sampler drift. The loaders_samplers sets are maintained manually; a model could request a sampler the loader silently ignores. Surfacing "ignored sampler" warnings on the API would aid debugging.
• Extension requirement fragility. Extensions fail open (logged, skipped). A dependency health endpoint listing which extensions loaded vs. failed would help operators.

12. Quick Reference

Start (full, NVIDIA): ./start_linux.sh → pick A → Model tab → download → load.

Start (API only): python server.py --nowebui --api --api-key KEY --admin-key ADMIN --model NAME

Persistent flags: user_data/CMD_FLAGS.txt (one per line).

Persistent UI settings: user_data/settings.yaml.

Config precedence: CLI → CMD_FLAGS.txt → settings.yaml → defaults.

Loaders: llama.cpp (GGUF, subprocess) · Transformers · ExLlamav3_HF · ExLlamav3 · TensorRT-LLM.

Default API port: 5000. Localhost-locked unless --listen/--public-api.

Auth headers: Authorization: Bearer <key> (OpenAI) · x-api-key: <key> (Anthropic) · admin-key for model/load|unload.

State to back up: user_data/ (never touched by updates).

Update: update_wizard_* → A. Major Python/CUDA bumps require deleting installer_files/ and reinstalling.

Key env vars: HF_TOKEN, GPU_CHOICE, OPENEDAI_*, TEXTGEN_ELECTRON.