# GPT Academic Administrator & Developer Guide

## 1. Architecture Overview

GPT Academic is a Gradio-based web application designed for academic and programming productivity through LLM integration. The system provides a multi-model chat interface with a plugin architecture supporting 40+ specialized functions for code analysis, paper review, LaTeX processing, and more.

### High-Level Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        User Browser                              │
└──────────────────────────────┬──────────────────────────────────┘
                               │ HTTP/WebSocket
┌──────────────────────────────▼──────────────────────────────────┐
│                    FastAPI + Gradio Server                        │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────────┐ │
│  │ main.py     │  │ FastAPI      │  │ Gradio 3.32.15         │ │
│  │ (entry)     │  │ Server       │  │ (custom fork)          │ │
│  └──────┬──────┘  └──────┬───────┘  └────────────┬───────────┘ │
│         │                │                        │              │
│  ┌──────▼──────────────────────────────────────────▼───────────┐ │
│  │              shared_utils/ (17 modules)                      │ │
│  │  config_loader · cookie_manager · fastapi_server             │ │
│  │  fastapi_stream_server · handle_upload · key_pattern_manager │ │
│  └──────────────────────────┬──────────────────────────────────┘ │
│                             │                                    │
│  ┌──────────────────────────▼──────────────────────────────────┐ │
│  │           crazy_functions/ (49 plugin files)                  │ │
│  │  agent_fns · rag_fns · pdf_fns · latex_fns · review_fns     │ │
│  └──────────────────────────┬──────────────────────────────────┘ │
│                             │                                    │
│  ┌──────────────────────────▼──────────────────────────────────┐ │
│  │           request_llms/ (LLM Bridge Layer)                   │ │
│  │  bridge_all.py → 30+ model bridges                          │ │
│  │  key_manager.py · local_llm_class.py                         │ │
│  └──────────────────────────┬──────────────────────────────────┘ │
└──────────────────────────────┼──────────────────────────────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
   ┌─────────────┐    ┌─────────────┐    ┌─────────────────┐
   │ OpenAI/Azure│    │ Claude/     │    │ Local Models    │
   │ GPT-4o      │    │ Gemini/etc  │    │ Ollama/vLLM     │
   └─────────────┘    └─────────────┘    └─────────────────┘
```

### Entry Point Flow

```
main.py
  ├── config.py (defaults) → config_private.py (overrides) → env vars (highest priority)
  ├── shared_utils/config_loader.py → get_conf() resolves final values
  ├── themes/theme.py → CSS/JS generation, dark mode
  ├── core_functional.py → basic function buttons (polish, translate, grammar)
  ├── crazy_functional.py → plugin registry
  └── Gradio Blocks UI construction → FastAPI mount → server launch
```

### Configuration Priority Chain

```
Environment Variables (highest)
       ↓
config_private.py (user overrides, gitignored)
       ↓
config.py (defaults, committed)
```

The `get_conf()` function in `shared_utils/config_loader.py` resolves this chain at runtime for every configuration key access.

### Component Responsibilities

| Component | Path | Responsibility |
|---|---|---|
| `main.py` | root | Gradio UI construction, app bootstrap |
| `config.py` | root | 80+ default settings |
| `core_functional.py` | root | Basic function button definitions |
| `crazy_functional.py` | root | Plugin registry (40+ plugins) |
| `toolbox.py` | root | Shared decorators, helpers, utilities |
| `check_proxy.py` | root | Proxy verification, auto-update check |
| `multi_language.py` | root | i18n support |
| `request_llms/bridge_all.py` | request_llms/ | Model routing dispatch |
| `request_llms/key_manager.py` | request_llms/ | API key rotation and blacklisting |
| `shared_utils/fastapi_server.py` | shared_utils/ | FastAPI app, auth, file serving |
| `shared_utils/fastapi_stream_server.py` | shared_utils/ | WebSocket streaming |
| `shared_utils/cookie_manager.py` | shared_utils/ | Per-session state |
| `themes/theme.py` | themes/ | Dynamic theming system |

---

## 2. System Requirements

### Hardware Requirements

| Deployment Type | CPU | RAM | Storage | GPU |
|---|---|---|---|---|
| API-only (remote LLMs) | 2 cores | 4 GB | 10 GB | None |
| With PDF/LaTeX plugins | 4 cores | 8 GB | 20 GB | None |
| With local LLMs (ChatGLM, etc.) | 8+ cores | 32 GB | 100 GB | NVIDIA 8+ GB VRAM |
| Production multi-user | 8+ cores | 16 GB | 50 GB | Optional |

### Software Dependencies

| Software | Version | Required |
|---|---|---|
| Python | 3.9+ | Yes |
| Gradio | 3.32.15 (bundled custom fork) | Yes |
| FastAPI | 0.110 | Yes |
| tiktoken | >=0.3.3 | Yes |
| Docker | 20.10+ | Optional |
| Docker Compose | 2.x | Optional |
| GROBID | latest | Optional (PDF parsing) |
| Milvus | latest | Optional (RAG vector store) |
| LaTeX (texlive) | latest | Optional (LaTeX plugins) |

> [!IMPORTANT]
> The project uses a **custom fork of Gradio 3.32.15**. Using standard Gradio from PyPI will cause `ModuleNotFoundError` or `AttributeError: module 'gradio' has no attribute 'Floating'` errors. Always use the bundled version.

### Python Package Requirements

Core dependencies from the project:

```
tiktoken>=0.3.3
openai
anthropic>=0.18.1
zhipuai==2.0.1
dashscope
gradio==3.32.15  # custom fork
fastapi==0.110
transformers>=4.27.1,<4.42
edge-tts>=7.0.0
spacy==3.7.4
pymupdf
nltk
llamaindex==0.10.68
```

> [!WARNING]
> The `transformers` library is pinned to `>=4.27.1,<4.42`. Versions 4.42+ introduce breaking changes for local model loading (ChatGLM, InternLM, etc.).

---

## 3. Configuration Reference

All configuration lives in `config.py` with overrides in `config_private.py` (gitignored) or environment variables. The `get_conf()` function resolves the final value at runtime.

### API Keys and Model Configuration

| Setting | Default | Description |
|---|---|---|
| `API_KEY` | `""` | OpenAI API key. Supports comma-separated list for rotation. |
| `USE_PROXY` | `False` | Enable HTTP proxy |
| `proxies` | `{"http": "socks5h://...", "https": "socks5h://..."}` | Proxy configuration dict |
| `LLM_MODEL` | `"gpt-3.5-turbo"` | Default model selection |
| `AVAIL_LLM_MODELS` | (list of models) | Models shown in UI dropdown |
| `API_URL_REDIRECT` | `{}` | URL redirect map for API endpoints |

### Azure OpenAI Configuration

| Setting | Default | Description |
|---|---|---|
| `AZURE_ENDPOINT` | `""` | Azure OpenAI endpoint URL |
| `AZURE_API_KEY` | `""` | Azure API key |
| `AZURE_ENGINE` | `""` | Azure deployment name |
| `AZURE_CFG_ARRAY` | `{}` | Multi-deployment configuration array |

### Server Configuration

| Setting | Default | Description |
|---|---|---|
| `WEB_PORT` | `None` (auto) | Server port; -1 for random |
| `AUTHENTICATION` | `[]` | List of `[username, password]` pairs |
| `CUSTOM_PATH` | `"/"` | URL path prefix |
| `SSL_KEYFILE` | `""` | Path to SSL private key |
| `SSL_CERTFILE` | `""` | Path to SSL certificate |
| `SERVER_IP_FOR_GRADIO` | `"0.0.0.0"` | Bind address |
| `AUTO_OPEN_BROWSER` | `True` | Open browser on start |
| `MAX_RETRY` | `2` | LLM call retry count |

### File and Path Configuration

| Setting | Default | Description |
|---|---|---|
| `PATH_PRIVATE_UPLOAD` | `"private_upload"` | User upload directory |
| `PATH_LOGGING` | `"gpt_log"` | Log output directory |
| `WHEN_TO_USE_PROXY` | `["Download LLM"]` | Operations that use proxy |

### Feature Flags

| Setting | Default | Description |
|---|---|---|
| `ALLOW_RESET_CONFIG` | `False` | Allow natural language config modification |
| `PLUGIN_HOT_RELOAD` | `True` | Enable plugin hot-reloading |
| `MULTI_QUERY_LLM_MODELS` | `""` | Models for multi-query comparison |
| `NUM_CUSTOM_BASIC_BTN` | `4` | Number of custom function buttons |
| `LAYOUT` | `"LEFT-RIGHT"` | UI layout: LEFT-RIGHT or TOP-DOWN |
| `DARK_MODE` | `True` | Enable dark mode by default |
| `ADD_WAIFU` | `False` | Enable animated assistant character |
| `CHATBOT_HEIGHT` | `1115` | Chat window height in pixels |
| `INIT_SYS_PROMPT` | (Chinese prompt) | System prompt for LLM |

### Token and Rate Limiting

| Setting | Default | Description |
|---|---|---|
| `MAX_TOKEN_OUTPUT` | `512` | Max tokens per completion (overridable per model) |
| `MAX_TOKEN_OUTPUT_REASONING` | `12800` | Max tokens for reasoning models (o1, deepseek-reasoner) |
| `TOKEN_LIMIT_PER_FRAGMENT` | `None` | Per-fragment token cap |
| `INPUT_LIMIT_PER_FRAGMENT` | `None` | Input token limit |
| `TIMEOUT_SECONDS` | `25` | Request timeout |
| `DEFAULT_WORKER_NUM` | `3` | Parallel worker threads |

### Plugin-Specific Configuration

| Setting | Default | Description |
|---|---|---|
| `GROBID_URL` | `"https://qingxu98-grobid.hf.space"` | GROBID service endpoint |
| `DOC2X_API_KEY` | `""` | Doc2X PDF parsing API key |
| `ARXIV_CACHE_DIR` | `None` | arXiv paper cache directory |
| `HUGGINGFACE_ACCESS_TOKEN` | `""` | HuggingFace token for models |

> [!NOTE]
> The `ALLOW_RESET_CONFIG` setting defaults to `False`. When enabled, users can modify configuration via natural language commands in the chat interface — this is a significant security consideration in multi-user deployments.

### Model-Specific Settings

Settings are often model-prefixed for per-model override:

```python
# Example: per-model token limits
MAX_TOKEN_OUTPUT = 512                    # global default
# Override in config_private.py:
# GPT4_MAX_TOKEN_OUTPUT = 4096
```

### Environment Variable Override

Any setting from `config.py` can be overridden via environment variable of the same name:

```bash
export API_KEY="sk-xxxxx,sk-yyyyy"
export LLM_MODEL="gpt-4o"
export WEB_PORT="8080"
export AUTHENTICATION="[['admin','password123']]"
```

> [!CAUTION]
> The `AUTHENTICATION` setting via environment variable must be a valid Python literal string representation of a list of lists. Malformed values will cause startup failures.

---

## 4. Deployment & Scaling

### Docker Deployment (Recommended)

The project includes `Dockerfile`, `docker-compose.yml`, and `.dockerignore` at the repository root.

```bash
# Build and run
docker-compose up -d

# Or build manually
docker build -t gpt-academic .
docker run -d -p 8080:8080 \
  -e API_KEY="sk-your-key" \
  -e LLM_MODEL="gpt-4o" \
  -v ./gpt_log:/app/gpt_log \
  -v ./private_upload:/app/private_upload \
  gpt-academic
```

### Docker Compose Configuration

```yaml
version: '3'
services:
  gpt-academic:
    build: .
    ports:
      - "8080:8080"
    environment:
      - API_KEY=${API_KEY}
      - LLM_MODEL=${LLM_MODEL}
    volumes:
      - ./config_private.py:/app/config_private.py
      - ./gpt_log:/app/gpt_log
      - ./private_upload:/app/private_upload
    restart: unless-stopped
```

### Bare Metal Deployment

```bash
# Clone repository
git clone https://github.com/binary-husky/gpt_academic.git
cd gpt_academic

# Install dependencies
pip install -r requirements.txt

# Create config_private.py
cat > config_private.py << 'EOF'
API_KEY = "sk-your-key-here"
LLM_MODEL = "gpt-4o"
WEB_PORT = 8080
AUTHENTICATION = [["admin", "secure_password"]]
EOF

# Start server
python main.py
```

### Reverse Proxy (Nginx)

```nginx
server {
    listen 443 ssl;
    server_name gpt-academic.example.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}
```

> [!IMPORTANT]
> WebSocket support (`Upgrade` and `Connection` headers) is mandatory for the streaming chat interface. Without WebSocket passthrough, responses will not stream in real-time.

### SSL/TLS Direct Configuration

The application supports native SSL without a reverse proxy:

```python
# config_private.py
SSL_KEYFILE = "/path/to/privkey.pem"
SSL_CERTFILE = "/path/to/fullchain.pem"
```

> [!WARNING]
> If `SSL_KEYFILE` is set, `SSL_CERTFILE` must also be set. Setting only one raises a `ValueError` at startup.

### Scaling Considerations

| Strategy | Approach | Limitation |
|---|---|---|
| Vertical | Increase RAM/CPU | Single process, GIL-bound for local models |
| Horizontal (stateless) | Multiple containers behind LB | Session state in cookies — works if sticky sessions configured |
| Worker threads | `DEFAULT_WORKER_NUM` | Controls parallel LLM calls per user session |
| Multi-model | `MULTI_QUERY_LLM_MODELS` | Compare outputs from multiple models simultaneously |

The application uses `DEFAULT_WORKER_NUM = 3` for parallel processing within plugin executions. This controls how many LLM calls can run concurrently for a single user's operation (e.g., batch file analysis).

### GROBID Service Deployment (for PDF plugins)

```bash
# Run GROBID for academic PDF parsing
docker run -d --name grobid \
  -p 8070:8070 \
  lfoppiano/grobid:latest

# Point GPT Academic to local GROBID
# config_private.py:
GROBID_URL = "http://localhost:8070"
```

### Milvus Deployment (for RAG plugins)

```bash
# docker-compose for Milvus
docker-compose -f docker-compose-milvus.yml up -d

# The RAG plugins (crazy_functions/rag_fns/) connect to Milvus
# for vector storage and retrieval
```

---

## 5. Operations

### Starting and Stopping

```bash
# Start (foreground)
python main.py

# Start (background with logging)
nohup python main.py > gpt_log/server.log 2>&1 &

# Docker
docker-compose up -d
docker-compose down
```

### Log Management

Logs are written to `gpt_log/` by default (configurable via `PATH_LOGGING`):

| Log Type | Location | Content |
|---|---|---|
| Chat history | `gpt_log/chat_history/` | Per-session conversation logs |
| Plugin output | `gpt_log/` | Plugin execution results, generated files |
| Server logs | stdout/stderr | FastAPI/Gradio server messages |

### File Upload Management

User uploads are stored in `private_upload/` (configurable via `PATH_PRIVATE_UPLOAD`):

```
private_upload/
  └── {session_hash}/
      └── {timestamp}/
          └── uploaded_files...
```

> [!TIP]
> Implement periodic cleanup of `private_upload/` in production. The application does not automatically purge old upload directories. A cron job removing directories older than 24-48 hours is recommended.

### API Key Rotation

The `key_manager.py` module implements automatic key rotation:

1. Multiple keys are provided comma-separated in `API_KEY`
2. Keys are distributed round-robin across requests
3. Failed keys (auth errors, rate limits) are temporarily blacklisted
4. Blacklisted keys are retried after a cooldown period

```python
# config_private.py — multiple keys for rotation
API_KEY = "sk-key1,sk-key2,sk-key3"
```

### Health Monitoring

The application does not expose a dedicated health endpoint. Monitor via:

```bash
# Check if Gradio is responsive
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/

# Check WebSocket connectivity
curl -s -o /dev/null -w "%{http_code}" \
  -H "Upgrade: websocket" \
  -H "Connection: Upgrade" \
  http://localhost:8080/queue/join
```

### Auto-Update Mechanism

`check_proxy.py` includes an auto-update check that compares the local version against the remote repository. This runs at startup by default. In air-gapped or production environments, this behavior should be noted — it makes outbound HTTPS calls to GitHub.

### Troubleshooting

#### Error: `ModuleNotFoundError: No module named 'pkg_resources'`

**Cause:** Python 3.12+ removed `pkg_resources` from the standard library. Some dependencies still reference it.

**Solution:**
```bash
pip install setuptools
```

**Prevention:** Pin Python to 3.11 or ensure `setuptools` is in requirements.

---

#### Error: `AttributeError: module 'gradio' has no attribute 'Floating'`

**Cause:** Standard Gradio installed instead of the project's custom 3.32.15 fork.

**Solution:**
```bash
pip uninstall gradio
# Install the bundled version from the project's requirements
pip install -r requirements.txt
```

**Prevention:** Always install from the project's pinned requirements, never `pip install gradio` directly.

---

#### Error: `AssertionError` on API_KEY format

**Cause:** API key does not match expected format patterns. The `key_pattern_manager.py` validates key prefixes: `sk-` (OpenAI), `fk` (api2d), `azure-` (Azure), etc.

**Solution:** Verify the key format matches the model being used. Check `shared_utils/key_pattern_manager.py` for the full pattern list.

**Prevention:** Use the correct key format for your provider. Multiple keys must be comma-separated with no spaces.

---

#### Error: `ConnectionAbortedError` during generation

**Cause:** Token limit exceeded — the response exceeded the model's context window.

**Solution:** Reduce `MAX_TOKEN_OUTPUT` or break input into smaller fragments via `TOKEN_LIMIT_PER_FRAGMENT`.

---

#### Error: `GROBID_OFFLINE_EXCEPTION`

**Cause:** GROBID service is unreachable at the configured `GROBID_URL`.

**Solution:**
```bash
# Check GROBID availability
curl -s http://localhost:8070/api/isalive

# If using the default HuggingFace Space, it may be sleeping
# Deploy a local instance for reliability
```

---

#### Error: `RuntimeError` about missing API keys

**Cause:** Attempting to use a model bridge without configuring its API key.

**Solution:** Set the appropriate key in `config_private.py` or environment variables. Each model family requires its specific key (e.g., `ANTHROPIC_API_KEY` for Claude, `GOOGLE_API_KEY` for Gemini).

---

## 6. Performance

### Token Counting

The application uses `tiktoken` for accurate token counting with OpenAI models. For non-OpenAI models, approximate token counting is applied.

### Timeout Configuration

```python
TIMEOUT_SECONDS = 25        # Per-request timeout
MAX_RETRY = 2               # Retry count on failure
DEFAULT_WORKER_NUM = 3      # Parallel workers per operation
```

> [!TIP]
> For large document analysis (papers, codebases), increase `TIMEOUT_SECONDS` to 120+ and `DEFAULT_WORKER_NUM` to 5-8 depending on available API rate limits.

### Fragment Processing

Large inputs are automatically split into fragments based on:
- `TOKEN_LIMIT_PER_FRAGMENT` — max tokens per chunk
- `INPUT_LIMIT_PER_FRAGMENT` — max input tokens per call

This enables processing of documents that exceed a single model's context window by splitting, processing in parallel (up to `DEFAULT_WORKER_NUM`), and reassembling results.

### Local Model Performance

Local models (ChatGLM, LLaMA, etc.) use the `local_llm_class.py` base class which implements:
- Multiprocess architecture to avoid GIL contention
- Separate process for model inference
- IPC via pipes for streaming tokens back to the main process

```
Main Process (Gradio/FastAPI)
    │
    ├── Pipe (IPC)
    │
    └── Child Process (GPU inference)
        └── Model loaded in VRAM
```

### Caching

The application does not implement response caching by default. Each request hits the LLM provider. For production deployments handling repeated queries, consider adding a caching layer at the reverse proxy level.

---

## 7. Security

### Authentication

Authentication is configured via the `AUTHENTICATION` setting:

```python
# config_private.py
AUTHENTICATION = [
    ["admin", "strong_password_here"],
    ["user1", "another_password"],
]
```

When `AUTHENTICATION` is an empty list `[]`, the application runs without authentication (default).

The logout endpoint is available at `/academic_logout`.

### File Upload Security

`shared_utils/handle_upload.py` implements multiple protections:

1. **Path traversal prevention** — Uploaded filenames are sanitized; `../` sequences are rejected
2. **Symlink detection** — Symbolic links in upload paths are detected and rejected
3. **Directory confinement** — All file operations are validated to stay within the user's upload directory (`private_upload/{session_hash}/`)
4. **FriendlyException** — Validation failures raise user-facing errors without exposing internal paths

> [!CAUTION]
> The file serving endpoint (`/file/{path:path}`) serves files from the application directory. In production, ensure the application runs with minimal filesystem permissions and that sensitive files (config_private.py, .env) are not accessible via this endpoint.

### API Key Security

- Keys in `config_private.py` are gitignored by default
- The key rotation system in `key_manager.py` blacklists compromised keys automatically
- `ALLOW_RESET_CONFIG = False` prevents users from extracting or modifying keys via the chat interface

> [!WARNING]
> Never commit API keys to the repository. GitHub issue #2270 documents a case where a HuggingFace API key was accidentally exposed. Always use `config_private.py` or environment variables.

### Network Security

| Setting | Security Impact |
|---|---|
| `SERVER_IP_FOR_GRADIO = "0.0.0.0"` | Binds to all interfaces — restrict in production |
| `SSL_KEYFILE` / `SSL_CERTFILE` | Enable TLS termination at the application level |
| `USE_PROXY` / `proxies` | Route LLM traffic through proxy (privacy, compliance) |
| `WHEN_TO_USE_PROXY` | Fine-grained proxy application |

### Configuration Security

The `ALLOW_RESET_CONFIG` flag (default `False`) controls whether users can modify runtime configuration via natural language commands in the chat. When enabled:
- Users can change `API_KEY`, `LLM_MODEL`, and other settings via chat
- Changes persist for the session only
- This is intended for single-user/development scenarios

For multi-user production deployments, this **must** remain `False`.

### Session Isolation

Each user session gets a unique hash (managed by `shared_utils/cookie_manager.py`). File uploads, chat history, and plugin state are isolated per session via this hash.

---

## 8. Developer Internals

### Plugin Architecture

#### Plugin Types

1. **Legacy plugins** — Standalone functions decorated with `@CatchException`:
```python
from toolbox import CatchException

@CatchException
def my_plugin(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
    # Plugin logic here
    pass
```

2. **New-generation plugins** — Class-based using `GptAcademicPluginTemplate`:
```python
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate

class MyPlugin(GptAcademicPluginTemplate):
    def define_arg_selection_menu(self):
        # Return up to 8 argument definitions
        pass

    def execute(self, txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
        # Plugin logic
        pass
```

#### Plugin Registration

Plugins are registered in `crazy_functional.py`:

```python
from crazy_functions.my_plugin import my_plugin

function_plugins = {
    "My Plugin Name": {
        "Group": "编程",           # Plugin group
        "Color": "stop",          # Button color
        "AsButton": True,         # Show as button vs dropdown
        "Function": HotReload(my_plugin),  # Hot-reload wrapper
    }
}
```

#### Hot Reload

The `HotReload()` wrapper enables plugin modification without server restart. When `PLUGIN_HOT_RELOAD = True`, the wrapper re-imports the plugin module on each invocation.

#### Plugin Groups

| Group | Chinese | Content |
|---|---|---|
| Dialog | 对话 | Conversation enhancement |
| Programming | 编程 | Code analysis, generation |
| Academic | 学术 | Paper review, LaTeX, citations |
| Agent | 智能体 | AutoGen agents, autonomous tasks |

#### Plugin Access to System Resources

Plugins receive these parameters:
- `txt` — User input text
- `llm_kwargs` — Model configuration (model name, temperature, etc.)
- `plugin_kwargs` — Plugin-specific arguments (up to 8 from arg selection menu)
- `chatbot` — Chat history (mutable, for streaming updates)
- `history` — Full conversation history
- `system_prompt` — Current system prompt
- `user_request` — Request metadata (including file upload paths)

### LLM Bridge System

`request_llms/bridge_all.py` implements a routing layer that dispatches to model-specific bridges:

```python
# Model prefix → Bridge mapping (simplified)
model_info = {
    "gpt-3.5-turbo":    bridge_chatgpt,
    "gpt-4":            bridge_chatgpt,
    "gpt-4o":           bridge_chatgpt,
    "claude-3-":        bridge_claude,
    "gemini-":          bridge_google_gemini,
    "glm-":             bridge_zhipu,
    "qwen-":            bridge_qwen,
    "deepseek-":        bridge_deepseek,
    "moonshot-":        bridge_moonshot,
    "ollama-":          bridge_ollama,
    "one-api-":         bridge_one_api,
    "openrouter-":      bridge_openrouter,
    "vllm-":            bridge_vllm,
}
```

Each bridge implements:
- `predict()` — Synchronous completion
- `predict_no_ui_long_connection()` — Long-running without UI updates
- Streaming response handling
- Error handling and retry logic

### Key Pattern Manager

`shared_utils/key_pattern_manager.py` validates API key formats and routes to the correct provider:

| Pattern | Provider |
|---|---|
| `sk-` | OpenAI |
| `fk` | api2d |
| `azure-` | Azure OpenAI |
| (other patterns) | Respective providers |

This enables a single `API_KEY` field to auto-detect the provider and route accordingly.

### Local LLM Architecture

`request_llms/local_llm_class.py` provides the base class for running local models:

```
┌─────────────────────────────────────┐
│         Main Process (Gradio)        │
│  ┌────────────────────────────────┐ │
│  │     LocalLLMClass instance      │ │
│  │  - Spawns child process         │ │
│  │  - Manages pipe communication   │ │
│  │  - Handles token streaming      │ │
│  └──────────────┬─────────────────┘ │
└─────────────────┼───────────────────┘
                  │ Pipe (IPC)
┌─────────────────▼───────────────────┐
│        Child Process (GPU)           │
│  - Loads model into VRAM             │
│  - Runs inference                    │
│  - Streams tokens back via pipe      │
└─────────────────────────────────────┘
```

This architecture avoids GIL contention between the web server and model inference.

### Theme System

`themes/theme.py` provides dynamic theming:
- CSS generation based on configuration
- Dark mode toggle support
- Custom JavaScript injection via `themes/*.js`
- Gradio component styling overrides

### FastAPI Endpoints

| Method | Path | Module | Purpose |
|---|---|---|---|
| GET | `/` | `toolbox.py:717` | Redirect to Gradio UI |
| GET | `/file/{path:path}` | `fastapi_server.py` | Serve generated/uploaded files |
| HEAD/GET | `/file={path_or_url:path}` | `fastapi_server.py` | Alternative file serving |
| GET | `/academic_logout` | `fastapi_server.py:171` | Session logout |
| POST | `/vits` | `fastapi_server.py:239` | TTS forward proxy |
| GET | `/favicon.ico` | `fastapi_server.py:264` | Favicon serving |
| POST | `/predict_user_input` | `fastapi_stream_server.py:493` | Chat prediction (non-Gradio clients) |
| POST | `/core_functional` | `fastapi_stream_server.py:533` | Execute core functions |
| POST | `/upload` | `fastapi_stream_server.py:547` | File upload endpoint |

### Cookie/Session Management

`shared_utils/cookie_manager.py` manages per-user session state:
- Session hash generation and validation
- State persistence across page reloads
- Plugin state isolation between users
- Upload directory assignment per session

### Void Terminal

`crazy_functions/vt_fns/` implements the Void Terminal — a stateful plugin execution environment that maintains context across multiple plugin invocations within a session.

---

## 9. Integration Points

### OpenAI-Compatible APIs

Any OpenAI-compatible API can be integrated via `API_URL_REDIRECT`:

```python
# config_private.py
API_URL_REDIRECT = {
    "https://api.openai.com/v1/chat/completions": "https://your-proxy.example.com/v1/chat/completions"
}
```

This enables integration with:
- Local OpenAI-compatible servers (LiteLLM, LocalAI, text-generation-webui)
- Corporate API gateways
- Rate-limiting proxies

### Ollama Integration

```python
# config_private.py
LLM_MODEL = "ollama-llama3"
AVAIL_LLM_MODELS = ["ollama-llama3", "ollama-codellama", "ollama-mistral"]
OLLAMA_HOST = "http://localhost:11434"  # if not default
```

Models prefixed with `ollama-` are automatically routed to the Ollama bridge.

### vLLM Integration

```python
# config_private.py
LLM_MODEL = "vllm-your-model"
# Requires vLLM server running with OpenAI-compatible API
```

### One-API / OpenRouter

```python
# One-API (unified gateway)
LLM_MODEL = "one-api-gpt4"
API_KEY = "your-one-api-key"

# OpenRouter
LLM_MODEL = "openrouter-anthropic/claude-3-opus"
API_KEY = "your-openrouter-key"
```

### RAG Pipeline (LlamaIndex + Milvus)

The `crazy_functions/rag_fns/` module implements retrieval-augmented generation:

```
Documents → LlamaIndex (chunking + embedding) → Milvus (vector store)
                                                       ↓
User Query → Embedding → Milvus (similarity search) → Context → LLM → Response
```

Configuration for RAG requires:
- Running Milvus instance
- LlamaIndex 0.10.68
- Embedding model access (OpenAI or local)

### GROBID Integration (Academic PDF)

```python
# config_private.py
GROBID_URL = "http://localhost:8070"  # Local GROBID instance
# Or use the public HuggingFace Space (unreliable):
# GROBID_URL = "https://qingxu98-grobid.hf.space"
```

GROBID provides structured extraction of academic papers (title, authors, abstract, sections, references).

### Doc2X Integration (PDF)

```python
# config_private.py
DOC2X_API_KEY = "your-doc2x-key"
```

Alternative PDF parsing service with better handling of complex layouts, tables, and equations.

### Text-to-Speech (Edge TTS)

```python
# Requires edge-tts>=7.0.0
# The /vits endpoint forwards TTS requests
# No additional configuration needed beyond package installation
```

### AutoGen Agent System

`crazy_functions/agent_fns/` integrates Microsoft AutoGen for multi-agent conversations:
- Agent-to-agent communication
- Task decomposition
- Tool use within agent chains

> [!NOTE]
> The AutoGen integration uses the older AutoGen API. Compatibility with AutoGen 0.4+ (the AG2 rewrite) has not been verified.

### Webhook / External Notification

The application does not include built-in webhook support for completion notifications. For integration with external systems (Slack, Discord, etc.), implement a wrapper around the `/predict_user_input` endpoint or add custom logic in a plugin.

### Multi-Language Support

`multi_language.py` provides i18n support. The primary interface is Chinese with English as secondary. Translations are managed via the multi-language module.

---

_Generated May 2026 from binary-husky/gpt_academic at d6bde0f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._