# ChatGLM-6B Admin Guide

This guide is for the engineers, operators, and architects who deploy, harden, fine-tune, and maintain ChatGLM-6B in production or research environments. It covers the repository's architecture, the full fine-tuning configuration surface, the security implications of how the model is loaded and served, deployment patterns the repository itself does not prescribe, and the internals you need to operate it reliably.

For installation and day-to-day usage, see the companion **User Guide**.

---

## 1. Executive Summary

ChatGLM-6B is an open-source, bilingual (Chinese–English) conversational large language model with **6.2 billion parameters**, built on the GLM (General Language Model) architecture. It was released by the Knowledge Engineering Group (KEG) of Tsinghua University / Zhipu AI (the `THUDM` / `zai-org` organization).

**What this repository actually contains.** This is a **model-release repository**, not a self-contained application. The model weights and the model's architecture code live on the Hugging Face Hub (`THUDM/chatglm-6b`); this repository ships:

- A handful of **demo entry points** — a CLI chat (`cli_demo.py`), three web UIs (`web_demo.py`, `web_demo2.py`, `web_demo_old.py`), a REST API (`api.py`), and two multimodal VisualGLM demos.
- A **multi-GPU helper** (`utils.py`).
- A **P-Tuning v2 fine-tuning harness** (`ptuning/`) — the largest body of code in the repo.

Every demo is a thin wrapper that calls `AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)` and then `model.chat(...)` / `model.stream_chat(...)`. The substantive model logic is downloaded and executed from the Hub at runtime.

> [!IMPORTANT]
> **Two security facts dominate the threat model and recur throughout this guide.** (1) Loading the model executes **arbitrary remote Python** via `trust_remote_code=True`, and deserializes pickled checkpoints via `torch.load`. (2) The bundled REST API (`api.py`) binds to `0.0.0.0:8000` with **no authentication**. Treat both as load-bearing operational constraints, not footnotes. See [Section 5: Security](#5-security-analysis).

**Lifecycle status.** ChatGLM-6B is the **first-generation** GLM chat model. The maintainers have since shipped ChatGLM2-6B, ChatGLM3, and GLM-4, and the README routes new users to them. The repository is effectively in maintenance: stable and runnable, but not actively developed. Plan to **pin your dependency versions** (especially `transformers==4.27.1`) and carry any patches internally rather than expecting upstream fixes.

| Attribute | Value |
|---|---|
| Parameters | 6.2 billion (62亿) |
| Architecture | GLM (prefix-LM, relative position encoding, 28 transformer layers) |
| Languages | Chinese (primary), English (secondary) |
| Training | ~1T bilingual tokens + SFT + feedback bootstrapping + RLHF |
| Context length (effective) | 2048 tokens (training length) |
| Code license | Apache-2.0 |
| Weights license | Separate Model License (academic-open; commercial-allowed after registration) |
| Quantization | INT4 / INT8 via `cpm_kernels` |
| Source commit | `401bf3a` (2024-06-27) |

---

## 2. Architecture

### 2.1 System overview

```
                        Hugging Face Hub
                 ┌──────────────────────────────┐
                 │  THUDM/chatglm-6b             │
                 │   ├── modeling_chatglm.py ────┼──► executed locally
                 │   ├── tokenization_chatglm.py │    (trust_remote_code=True)
                 │   ├── quantization.py         │
                 │   └── pytorch_model-*.bin ────┼──► torch.load (pickle)
                 └──────────────┬───────────────┘
                                │  AutoModel.from_pretrained(...)
                                ▼
  ┌─────────────────────────────────────────────────────────────────┐
  │                      THIS REPOSITORY (glue)                       │
  │                                                                   │
  │   Inference demos            Fine-tuning harness (ptuning/)       │
  │   ┌──────────────┐           ┌─────────────────────────────────┐ │
  │   │ cli_demo.py  │           │ main.py  (HfArgumentParser)      │ │
  │   │ web_demo*.py │           │   ├── arguments.py (dataclasses) │ │
  │   │ api.py       │           │   ├── trainer.py        (vendored│ │
  │   │ *_vision.py  │           │   ├── trainer_seq2seq.py  HF code)│ │
  │   └──────┬───────┘           │   └── deepspeed.json (ZeRO-2)    │ │
  │          │ model.chat /      └─────────────────────────────────┘ │
  │          │ stream_chat        produces: PrefixEncoder checkpoint  │
  │   ┌──────▼───────┐                                                │
  │   │ utils.py     │  load_model_on_gpus() → accelerate dispatch    │
  │   └──────────────┘                                                │
  └─────────────────────────────────────────────────────────────────┘
```

### 2.2 Component inventory

| Component | File(s) | Responsibility |
|---|---|---|
| CLI demo | `cli_demo.py` | Terminal streaming chat; `clear`/`stop`/SIGINT handling. |
| Gradio demo | `web_demo.py`, `web_demo_old.py` | Browser chat UI; sliders for `max_length`/`top_p`/`temperature`. |
| Streamlit demo | `web_demo2.py` | Alternative browser UI; caches model via `@st.cache_resource`. |
| REST API | `api.py` | FastAPI app, single `POST /` endpoint, Uvicorn server on `:8000`. |
| Vision demos | `cli_demo_vision.py`, `web_demo_vision.py` | VisualGLM-6B (`THUDM/visualglm-6b`) multimodal chat. |
| Multi-GPU loader | `utils.py` | `auto_configure_device_map()`, `load_model_on_gpus()`. |
| Fine-tune driver | `ptuning/main.py` | Seq2seq training/eval entry point. |
| Fine-tune arguments | `ptuning/arguments.py` | `ModelArguments`, `DataTrainingArguments` dataclasses. |
| Trainer (vendored) | `ptuning/trainer.py`, `ptuning/trainer_seq2seq.py` | Copies of Hugging Face's `Trainer`/`Seq2SeqTrainer`, pinned for `transformers==4.27.1`. |
| DeepSpeed config | `ptuning/deepspeed.json` | ZeRO stage-2 config for full fine-tuning. |
| Launch scripts | `ptuning/*.sh` | Train/evaluate/web-demo wrappers. |

> [!NOTE]
> `ptuning/trainer.py` and `ptuning/trainer_seq2seq.py` are **vendored copies of Hugging Face internals** (~234 KB of the repo's ~253 KB of source). They are not project-authored logic — they are pinned to guarantee compatibility with `transformers==4.27.1`. Do not treat their classes (`Trainer`, `Seq2SeqTrainer`) or their SageMaker-specific environment-variable reads as part of the project's own configuration surface.

### 2.3 The model-loading path

All demos converge on the same sequence:

```python
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()
```

- `trust_remote_code=True` causes `transformers` to download and **execute** `modeling_chatglm.py`, `tokenization_chatglm.py`, and `quantization.py` from the Hub.
- `.half()` casts to FP16; `.quantize(4|8)` (when chained) builds INT4/INT8 kernels via `cpm_kernels`.
- `.cuda()` / `.float()` / `.to('mps')` selects the device.

### 2.4 Multi-GPU device map (`utils.py`)

`load_model_on_gpus()` splits the model across cards. The model has a fixed layer budget that `auto_configure_device_map()` distributes:

- `transformer.word_embeddings`, `transformer.final_layernorm`, and `lm_head` are **always pinned to GPU 0** (counted as 2 layers).
- The 28 transformer layers (`transformer.layers.{0..27}`) are spread evenly across the remaining capacity (`30 / num_gpus` layers per card).

> [!TIP]
> The embeddings, final layernorm, and head are forced onto card 0 deliberately. On Linux, `model.device` resolves to `lm_head.device`; if `word_embeddings` lived on a different card, `model.chat`/`stream_chat` would move `input_ids` to the wrong device and raise a `RuntimeError`. This is a deliberate workaround, not an arbitrary choice — preserve it if you write a custom `device_map`.

---

## 3. System Requirements

### 3.1 Hardware (inference)

| Quantization | GPU VRAM (inference) | GPU VRAM (fine-tuning) | System RAM to load |
|---|---|---|---|
| FP16 | 13 GB | 14 GB | ~13 GB |
| INT8 | 8 GB | 9 GB | ~13 GB (or load pre-quantized) |
| INT4 | 6 GB | 7 GB | ~13 GB (or ~5.2 GB pre-quantized) |
| CPU (FP16) | — | — | ~32 GB |

> [!IMPORTANT]
> On-the-fly quantization (`.quantize(n)`) loads the FP16 model into RAM (~13 GB) **before** compressing. On RAM-constrained hosts, load a pre-quantized checkpoint (`THUDM/chatglm-6b-int4`) instead, which needs only ~5.2 GB.

### 3.2 Software

| Requirement | Constraint |
|---|---|
| Python | 3.7+ |
| `torch` | >= 1.10 |
| `transformers` | **4.27.1 recommended**; >= 4.23.1 minimum |
| `cpm_kernels` | Required for INT4/INT8 GPU quantization |
| `gcc` + OpenMP | Required for **CPU** quantized inference (Linux `gcc 11.3.0`, Windows TDM-GCC 10.3.0 with `openmp`, macOS via manual OpenMP install) |
| CUDA | Required for any GPU path; not used on CPU/MPS |

Fine-tuning adds `rouge_chinese`, `nltk`, `jieba`, `datasets`, and — for full fine-tuning — `deepspeed`.

> [!WARNING]
> The `transformers` version is a hard compatibility gate, not a recommendation. The model's remote code is written against 4.27.1; newer releases surface as runtime errors (`ChatGLMTokenizer has no attribute vocab_size`, `KeyError: '<|endoftext|>'`, `.generate()` incompatibilities). Pin it in your deployment image.

---

## 4. Configuration Reference

This repository has **no application config files** (no YAML/TOML/`.env`). The two configuration surfaces are: (1) the generation parameters passed to `model.chat`, and (2) the fine-tuning arguments in `ptuning/`.

### 4.1 Generation parameters

| Parameter | Type | Default | Notes |
|---|---|---|---|
| `max_length` | int | 2048 | Total tokens (prompt + response). Quality degrades past 2048. |
| `top_p` | float | 0.7 | Nucleus sampling threshold. |
| `temperature` | float | 0.95 | Sampling randomness. |
| `history` | list | `[]` | List of `[query, response]` pairs for multi-turn context. |

### 4.2 Fine-tuning: `ModelArguments` (`ptuning/arguments.py`)

| Argument | Default | Purpose |
|---|---|---|
| `model_name_or_path` | (required) | Hub ID or local path to the base model. |
| `ptuning_checkpoint` | `None` | Path to a P-Tuning v2 prefix checkpoint (inference/eval). |
| `config_name` | `None` | Override config path. |
| `tokenizer_name` | `None` | Override tokenizer path. |
| `cache_dir` | `None` | Hub download cache location. |
| `use_fast_tokenizer` | `True` | Use the fast (Rust) tokenizer. |
| `model_revision` | `"main"` | Hub branch/tag/commit. |
| `use_auth_token` | `False` | Use the cached `huggingface-cli login` token. |
| `resize_position_embeddings` | `None` | Auto-resize position embeddings if needed. |
| `quantization_bit` | `None` | `4` or `8` to quantize; unset = FP16. |
| `pre_seq_len` | `None` | Soft-prompt (prefix) length. Must match between train and inference. |
| `prefix_projection` | `False` | Whether to add an MLP projection to the prefix encoder. |

### 4.3 Fine-tuning: `DataTrainingArguments` (`ptuning/arguments.py`)

| Argument | Default | Purpose |
|---|---|---|
| `train_file` / `validation_file` / `test_file` | `None` | Paths to JSON/CSV data splits. |
| `prompt_column` | `None` | JSON key holding the input text. |
| `response_column` | `None` | JSON key holding the target text. |
| `history_column` | `None` | JSON key holding multi-turn history. |
| `max_source_length` | 1024 | Max input tokens (scripts override to 64 or 256). |
| `max_target_length` | 128 | Max output tokens (scripts override to 64 or 256). |
| `val_max_target_length` | = `max_target_length` | Eval generation length. |
| `overwrite_cache` | `False` | Rebuild the tokenized dataset cache. |
| `preprocessing_num_workers` | `None` | Tokenization parallelism. |
| `pad_to_max_length` | `False` | Static vs dynamic padding. |
| `max_train_samples` / `max_eval_samples` / `max_predict_samples` | `None` | Truncate splits (debugging). |
| `num_beams` | `None` | Beam count for eval generation. |
| `ignore_pad_token_for_loss` | `True` | Mask pad tokens in the loss. |
| `source_prefix` | `""` | Prefix prepended to every input. |

`main.py` parses these three dataclasses (`ModelArguments`, `DataTrainingArguments`, `Seq2SeqTrainingArguments`) via Hugging Face's `HfArgumentParser`. You can also pass a single `.json` file argument and it will be parsed as the full config.

### 4.4 The launch scripts (`ptuning/*.sh`)

| Script | Effective batch | Steps | LR | Quant | Lengths |
|---|---|---|---|---|---|
| `train.sh` (P-Tuning v2) | 1 × 16 accum = 16 | 3000 | 2e-2 | INT4 | 64 / 64 |
| `train_chat.sh` (multi-turn) | 1 × 16 = 16 | 3000 | 1e-2 | INT4 | 256 / 256 |
| `ds_train_finetune.sh` (full) | 4 × 1 = 4 (×4 GPU) | 5000 | 1e-4 | FP16 | 64 / 64 |
| `evaluate.sh` | eval bs 1 | — | — | INT4 | 64 / 64 |
| `evaluate_finetune.sh` | eval bs 1 (`fp16_full_eval`) | — | — | FP16 | 256 / 256 |

`PRE_SEQ_LEN=128` is the soft-prompt length used across the P-Tuning scripts.

### 4.5 DeepSpeed config (`ptuning/deepspeed.json`)

```json
{
  "train_micro_batch_size_per_gpu": "auto",
  "zero_allow_untested_optimizer": true,
  "fp16": { "enabled": "auto", "loss_scale": 0, "initial_scale_power": 16,
            "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 },
  "zero_optimization": {
    "stage": 2, "allgather_partitions": true, "allgather_bucket_size": 5e8,
    "overlap_comm": false, "reduce_scatter": true, "reduce_bucket_size": 5e8,
    "contiguous_gradients": true
  }
}
```

ZeRO **stage 2** partitions optimizer state and gradients (but not parameters) across the four GPUs. This is what makes full-parameter fine-tuning fit on 4× consumer/datacenter cards.

---

## 5. Security Analysis

> [!CAUTION]
> ChatGLM-6B's loading and serving model carries three concrete, exploitable risks. Review all three before any networked deployment. A community security report (issue #1517, "Insecure Deserialization and Remote Code Execution via `trust_remote_code` and `torch.load`") documents the first two.

### 5.1 Remote code execution via `trust_remote_code=True`

Every demo loads the model with `trust_remote_code=True`, which downloads and **executes** Python from the Hugging Face Hub (`modeling_chatglm.py`, `tokenization_chatglm.py`, `quantization.py`). If the Hub repository, your network path to it, or your local model cache is compromised, that code runs with your process's full privileges.

**Mitigations:**

- **Pin a known-good revision.** Use `revision="<commit-sha>"` (not a moving branch) so an upstream change cannot silently alter the executed code.
- **Vendor the weights and code.** Download once to a local, integrity-checked folder and load from that path. Audit the model's `.py` files before first use.
- **Run unprivileged and sandboxed.** Use a dedicated non-root user, a container with no host mounts, and no outbound network beyond what's required.

### 5.2 Pickle deserialization via `torch.load`

The P-Tuning inference path calls `torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))`. `torch.load` uses Python `pickle`, which can execute arbitrary code during deserialization. Loading a checkpoint from an untrusted source is equivalent to running an untrusted program.

**Mitigations:** only load checkpoints you produced or trust; store them with restrictive permissions; prefer `safetensors` where a conversion is available; never load a user-uploaded `.bin` directly.

### 5.3 Unauthenticated network service (`api.py`)

```python
uvicorn.run(app, host='0.0.0.0', port=8000, workers=1)
```

`api.py` binds to **all interfaces** with **no authentication, no rate limiting, and no TLS**. The single `POST /` endpoint runs inference for anyone who can reach the port, and it **prints every prompt to stdout** (the server logs user input in cleartext).

**Mitigations:**

- Bind to `127.0.0.1` and front the service with a reverse proxy (nginx/Caddy/Traefik) that enforces TLS, authentication, and rate limits.
- Place the service on a private network or behind a firewall; never expose port 8000 to the internet directly.
- Review the stdout logging if prompts may contain sensitive data — it logs `prompt` and `response` verbatim.
- `workers=1` means the API serves one request at a time on a single GPU; do not assume concurrency.

### 5.4 Model-level safety limitations

The model itself can produce **harmful, biased, or factually wrong** content, is **easily misled** (including about its own identity), and is weak at math/logic. These are documented limitations, not bugs. Any user-facing deployment needs its own content moderation and output validation — do not rely on the base model for safety.

---

## 6. Deployment Patterns

The repository ships demos, not a production server — it is deliberately unopinionated about deployment. The patterns below are operator guidance for taking it to production.

### 6.1 Single-GPU inference service

The simplest production shape: harden `api.py` and run it as a managed service.

- Change `host='0.0.0.0'` to `host='127.0.0.1'`; add a reverse proxy for TLS + auth.
- Load at the quantization level that fits your card with headroom (INT8 on an 8 GB card, INT4 on 6 GB).
- Run under a process manager (systemd, supervisor) that restarts on crash.

Example systemd unit (sketch):

```ini
[Unit]
Description=ChatGLM-6B API
After=network.target

[Service]
User=chatglm
WorkingDirectory=/opt/chatglm-6b
ExecStart=/opt/chatglm-6b/venv/bin/python api.py
Restart=on-failure
Environment=CUDA_VISIBLE_DEVICES=0

[Install]
WantedBy=multi-user.target
```

### 6.2 Containerized deployment

- Base image: an `nvidia/cuda` runtime image matching your driver, plus Python 3.10.
- Bake `transformers==4.27.1` and the rest of `requirements.txt` into the image.
- Bake or volume-mount the weights so the container does not download from the Hub at start (also closes the RCE-on-pull window).
- Run with `--gpus all`, a non-root user, and no host mounts beyond the weights.

> [!TIP]
> Pin every dependency in the image. Because the upstream repo is in maintenance, an unpinned `pip install` can pull a `transformers` version the model's remote code does not support.

### 6.3 Multi-GPU for large/long contexts

Use `utils.load_model_on_gpus(..., num_gpus=N)` ([Section 2.4](#24-multi-gpu-device-map-utilspy)) when one card cannot hold the model. Note this splits **one** model across cards for capacity — it does not increase throughput. For throughput, run independent single-GPU replicas behind a load balancer.

### 6.4 Capacity notes

- `api.py` is single-worker and synchronous: one in-flight request per process. Scale by running multiple replicas (one per GPU) behind a proxy, not by raising `workers` on one GPU.
- VRAM grows with conversation length; enforce a `max_length` ceiling and consider trimming `history` server-side for long sessions.

---

## 7. Operations

### 7.1 Starting and stopping

| Action | Command |
|---|---|
| Start API | `python api.py` (or `systemctl start chatglm`) |
| Start Gradio UI | `python web_demo.py` |
| Pin a GPU | `CUDA_VISIBLE_DEVICES=0 python api.py` |
| Health check | `curl -X POST localhost:8000 -H 'Content-Type: application/json' -d '{"prompt":"ping","history":[]}'` |

### 7.2 GPU memory management

`api.py` calls `torch_gc()` after each request to release cache and IPC handles:

```python
def torch_gc():
    if torch.cuda.is_available():
        with torch.cuda.device(CUDA_DEVICE):
            torch.cuda.empty_cache()
            torch.cuda.ipc_collect()
```

If you build your own server, replicate this between requests. If VRAM stays pinned despite `empty_cache()`, the only reliable recovery is to restart the process (a known issue — see #250, #1144).

### 7.3 Monitoring

There is no built-in metrics endpoint. For production, add:

- GPU utilization/VRAM monitoring (`nvidia-smi`, DCGM exporter).
- Latency and error-rate metrics at the reverse-proxy layer.
- Log rotation — `api.py` prints every prompt/response to stdout; capture and rotate it, and scrub if prompts are sensitive.

### 7.4 Fine-tuning operations runbook

1. Install fine-tuning deps: `pip install rouge_chinese nltk jieba datasets`.
2. Stage the dataset so `AdvertiseGen/` (or your data) sits in `ptuning/`.
3. `cd ptuning && bash train.sh` — checkpoints land in `output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-*` every 1000 steps.
4. `bash evaluate.sh` — predictions and Rouge/BLEU-4 scores in `generated_predictions.txt`.
5. Deploy the prefix checkpoint by loading the base model + `PrefixEncoder` weights ([User Guide §7.5](zai-org__ChatGLM-6B__UserGuide.md)).

---

## 8. Internals

### 8.1 Inference flow

```
user text → tokenizer.encode → model.stream_chat → (28 GLM layers) → logits
          → sampling (top_p, temperature) → detokenize → yield partial response
```

`stream_chat` yields the cumulative response on each step, which is what produces the typewriter effect in the demos. `chat` is the non-streaming equivalent.

### 8.2 P-Tuning v2 mechanism

P-Tuning v2 **freezes the entire base model** and trains a small `PrefixEncoder` that injects learned key/value prefixes into every attention layer (length `pre_seq_len`). This is why:

- A P-Tuning checkpoint contains **only** `transformer.prefix_encoder.*` weights (a few MB), not a full model.
- Inference must load the base model first, then graft the prefix weights on top (`model.transformer.prefix_encoder.load_state_dict(...)`).
- `prefix_projection=False` (default) uses a plain embedding prefix; setting it `True` adds an MLP reparameterization.

### 8.3 Quantization

`.quantize(4|8)` is implemented in the model's remote code and depends on `cpm_kernels` for the GPU path. On CPU, the kernels are compiled at runtime via `gcc`/OpenMP. The prefix encoder is kept in FP32 even when the base model is quantized (`model.transformer.prefix_encoder.float()`), because the small prefix is cheap and benefits from full precision.

### 8.4 Repository structure

```
ChatGLM-6B/
├── api.py                 FastAPI REST server (POST /)
├── cli_demo.py            Terminal streaming chat
├── cli_demo_vision.py     VisualGLM terminal demo
├── web_demo.py            Gradio UI (current)
├── web_demo2.py           Streamlit UI (community)
├── web_demo_old.py        Gradio UI (legacy, no typewriter)
├── web_demo_vision.py     VisualGLM Gradio UI
├── utils.py               Multi-GPU device-map helper
├── requirements.txt       Runtime dependencies
├── FAQ.md / UPDATE.md / PROJECT.md
├── examples/              Capability screenshots
├── limitations/           Failure-case screenshots
├── improve/               Community fine-tuning data sample
└── ptuning/
    ├── main.py            Seq2seq train/eval driver
    ├── arguments.py       Model/Data argument dataclasses
    ├── trainer.py         Vendored HF Trainer (4.27.1)
    ├── trainer_seq2seq.py Vendored HF Seq2SeqTrainer
    ├── deepspeed.json     ZeRO stage-2 config
    ├── train.sh / train_chat.sh / ds_train_finetune.sh
    ├── evaluate.sh / evaluate_finetune.sh / web_demo.sh
    └── web_demo.py        Web demo that loads a prefix checkpoint
```

---

## 9. Troubleshooting (operational)

| Symptom | Likely cause | Fix |
|---|---|---|
| `Could not find module 'nvcuda.dll'` (Windows) | No CUDA runtime; CPU-only host | Load weights locally, use `.float()` CPU path |
| `RuntimeError: Unknown platform: darwin` (macOS) | CUDA quantization path unavailable | Load locally; use MPS or CPU |
| `clang: error: unsupported option '-fopenmp'` (macOS) | OpenMP missing for CPU quant kernels | Install OpenMP (FAQ Q1); clear HF module cache |
| `ChatGLMTokenizer has no attribute vocab_size` / `KeyError: '<|endoftext|>'` | `transformers` too new | Pin `transformers==4.27.1` |
| VRAM never released after requests | CUDA cache/IPC handles retained | Call `empty_cache()` + `ipc_collect()`; restart if persistent (#250, #1144) |
| Fine-tune: `ValueError: None is not in list` | Column name mismatch in `train.sh` | Align `--prompt_column`/`--response_column`/`--history_column` with dataset keys (#1471) |
| DeepSpeed: `Cannot copy out of meta tensor` (stage 3) | ZeRO stage-3 unsupported by this path | Use the shipped stage-2 `deepspeed.json` (#995) |
| OOM during training | Quant/batch too large for VRAM | INT4 + raise `gradient_accumulation_steps`, lower `per_device_train_batch_size` |

---

## 10. Suggested Improvements & Technical Debt

These observations are derived from the source and the issue tracker. They are guidance for operators forking or maintaining the code internally — upstream is in maintenance and unlikely to action them.

- **Pin `transformers` explicitly in `requirements.txt`.** It currently lists `transformers==4.27.1`, which is correct — keep it; do not relax to a range. A surprising number of issues stem from version drift.
- **Harden `api.py` before any networked use.** Add authentication, bind to localhost, add TLS via a proxy, and stop logging full prompts to stdout. (See [Section 5.3](#53-unauthenticated-network-service-apipy).)
- **Prefer `safetensors` over `torch.load` for checkpoints** to remove the pickle RCE surface where conversions exist.
- **Migrate off `trust_remote_code` by vendoring and auditing the model code**, then loading from a fixed local path with a pinned revision.
- **De-duplicate the demos.** `web_demo.py`, `web_demo_old.py`, and `ptuning/web_demo.py` share most of their logic; `cli_demo.py`/`cli_demo_vision.py` likewise. A small shared module would reduce drift.
- **Consider the newer models.** For new work, ChatGLM2-6B/ChatGLM3/GLM-4 supersede this model on quality, context length (32K+), and inference speed. Document the migration path for downstream consumers.
- **Add a metrics/health endpoint** to `api.py` for production observability (currently only the inference `POST /` exists).

---

## 11. Quick Reference

### Loading recipes

| Goal | Code |
|---|---|
| FP16 GPU | `.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()` |
| INT8 GPU | `....quantize(8).half().cuda()` |
| INT4 GPU | `....quantize(4).half().cuda()` |
| Pre-quantized INT4 | `.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()` |
| CPU | `....float()` |
| Apple MPS | `.from_pretrained("<local path>", trust_remote_code=True).half().to('mps')` |
| Multi-GPU | `load_model_on_gpus("THUDM/chatglm-6b", num_gpus=2)` |
| Pin revision | add `revision="v1.1.0"` |

### Key facts

| Item | Value |
|---|---|
| Base model ID | `THUDM/chatglm-6b` |
| Vision model ID | `THUDM/visualglm-6b` |
| API endpoint | `POST /` on `0.0.0.0:8000` |
| Transformer layers | 28 (+ embeddings + head) |
| Required `transformers` | 4.27.1 |
| Code license | Apache-2.0 |
| Weights license | Model License (registration for commercial) |
| Soft-prompt length (P-Tuning) | `PRE_SEQ_LEN=128` |
| Tunable params (P-Tuning v2) | ~0.1% of model |

---

_Generated June 2026 from zai-org/ChatGLM-6B at 401bf3a. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._