# ChatGLM-6B User Guide

ChatGLM-6B is an open, bilingual (Chinese–English) conversational language model with **6.2 billion parameters**, built on the General Language Model (GLM) architecture. Thanks to built-in quantization, you can run it locally on a consumer GPU — as little as **6 GB of VRAM** at the INT4 quantization level.

This repository does not contain the model weights themselves. It ships a small set of **ready-to-run demos** (command line, web UI, and a REST API) plus a **P-Tuning v2 fine-tuning harness** that wrap the published `THUDM/chatglm-6b` weights on the Hugging Face Hub. This guide walks you through installing the dependencies, loading the model, running each demo, shrinking the memory footprint with quantization, and fine-tuning the model on your own data.

> [!NOTE]
> ChatGLM-6B is the **first-generation** model in the GLM chat family. The maintainers have since released [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B), [ChatGLM3](https://github.com/THUDM/ChatGLM3), and [GLM-4](https://github.com/THUDM/GLM-4), each with stronger performance, longer context, and faster inference. If you are starting a new project, evaluate the newer models first. This guide documents ChatGLM-6B exactly as it exists in this repository.

---

## 1. Quick Start

If you already have a CUDA GPU with at least 13 GB of VRAM and a working Python environment, you can be chatting with the model in three steps.

```bash
# 1. Get the demo scripts
git clone https://github.com/THUDM/ChatGLM-6B
cd ChatGLM-6B

# 2. Install dependencies
pip install -r requirements.txt

# 3. Launch the command-line chat demo
python cli_demo.py
```

The first run downloads the model weights (~13 GB) from the Hugging Face Hub automatically. Once loaded, type a message and press Enter:

```text
欢迎使用 ChatGLM-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序

用户：你好

ChatGLM-6B：你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。
```

Type `clear` to wipe the conversation history, or `stop` to quit.

> [!TIP]
> Short on VRAM? Skip ahead to [Section 6: Running on Limited Hardware](#6-running-on-limited-hardware). With INT4 quantization the model fits in 6 GB, and it can even run on CPU (with ~32 GB of RAM).

---

## 2. Requirements

### Hardware

The memory you need depends on the **quantization level** you load the model at. This table is the single most important reference in this guide — consult it before you do anything else.

| Quantization level | Minimum GPU VRAM (inference) | Minimum GPU VRAM (efficient fine-tuning) |
|---|---|---|
| FP16 (no quantization) | 13 GB | 14 GB |
| INT8 | 8 GB | 9 GB |
| INT4 | 6 GB | 7 GB |

> [!IMPORTANT]
> These are the model's own published figures. VRAM use grows as a conversation gets longer (the context accumulates), so leave headroom above the minimum. After 2–3 turns, INT8 settles around 10 GB and INT4 around 6 GB.

CPU-only inference is possible but slow, and needs roughly **32 GB of system RAM** at FP16 (much less if you load a pre-quantized INT4 checkpoint — about 5.2 GB).

### Software

| Dependency | Version | Purpose |
|---|---|---|
| `python` | 3.7+ | Runtime |
| `torch` | >= 1.10 | Tensor backend |
| `transformers` | **4.27.1** (recommended); >= 4.23.1 works | Model loading via `AutoModel` |
| `cpm_kernels` | latest | CUDA kernels for INT4/INT8 quantized inference |
| `sentencepiece` | latest | Tokenizer |
| `protobuf` | latest | Tokenizer model parsing |
| `gradio` | latest | Web demo (`web_demo.py`) |
| `mdtex2html` | latest | Markdown/LaTeX rendering in the web demo |
| `accelerate` | latest | Multi-GPU model dispatch |

All of these are pinned in `requirements.txt` and installed by `pip install -r requirements.txt`.

> [!WARNING]
> The `transformers` version matters. The model's remote code was written against `transformers==4.27.1`. Newer versions (e.g. 4.34+) can raise tokenizer or `.generate()` errors. If you hit a loading error, pin `transformers` to `4.27.1` first.

For CPU inference of a quantized model you additionally need a C compiler with OpenMP:

=== "Linux"

    Most distributions already include `gcc` with OpenMP. The model was tested with `gcc 11.3.0`.

=== "Windows"

    Install [TDM-GCC](https://jmeubank.github.io/tdm-gcc/) and tick the **`openmp`** option during setup. Tested with `TDM-GCC 10.3.0`.

=== "macOS"

    macOS lacks OpenMP by default. Install it manually — see [Troubleshooting → `clang: error: unsupported option '-fopenmp'`](#error-clang-error-unsupported-option--fopenmp).

---

## 3. Installation

### Step 1 — Clone the repository

```bash
git clone https://github.com/THUDM/ChatGLM-6B
cd ChatGLM-6B
```

### Step 2 — Install dependencies

```bash
pip install -r requirements.txt
```

```text
Successfully installed transformers-4.27.1 torch-2.x cpm_kernels-x.x.x gradio-x.x.x ...
```

### Step 3 — Get the model weights

You have two options.

**Option A — automatic download (default).** Every demo calls `AutoModel.from_pretrained("THUDM/chatglm-6b", ...)`, which downloads the weights and the model implementation from the Hugging Face Hub on first run. No extra steps required — just run a demo.

**Option B — manual download (slow or unreliable networks).** Download the weights ahead of time, then point the demos at a local folder.

```bash
# Install Git LFS first: https://git-lfs.com
git clone https://huggingface.co/THUDM/chatglm-6b
```

If the large weight files download slowly, fetch only the code first and then add the weights by hand:

```bash
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/THUDM/chatglm-6b
```

Then download the parameter files from the [Tsinghua Cloud mirror](https://cloud.tsinghua.edu.cn/d/fb9f16d6dc8f482596c2/) and place them into your local `chatglm-6b` directory. Finally, replace `THUDM/chatglm-6b` in the demo scripts with the path to that folder.

> [!TIP]
> The model implementation evolves over time. To pin a specific revision for reproducibility, add `revision="v1.1.0"` to the `from_pretrained` call, or run `git checkout v1.1.0` inside a locally cloned weights folder. `v1.1.0` rebalances the Chinese/English training mix and reduces stray Chinese words in English answers.

---

## 4. Calling the Model From Python

Every demo in this repository is a thin wrapper around the same three calls: load a tokenizer, load the model, then call `model.chat(...)` or `model.stream_chat(...)`. You can use the model directly in your own code the same way.

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])
print(response)
```

```text
你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。
```

To continue a multi-turn conversation, pass the returned `history` back into the next call:

```python
response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
print(response)
```

> [!NOTE]
> `trust_remote_code=True` is **required**. The model's architecture is defined in Python files that ship alongside the weights on the Hub, and `transformers` only executes them when you opt in. See the Admin Guide's Security section for what this implies.

### Streaming token-by-token

For a typewriter effect, use `stream_chat`, which yields the partial response as it is generated:

```python
for response, history in model.stream_chat(tokenizer, "讲个笑话", history=[]):
    print(response)
```

Each iteration returns the full response generated *so far*, not just the newest token.

### Generation parameters

`model.chat` / `model.stream_chat` accept these keyword arguments. The defaults below are what the demos use.

| Parameter | Default | Effect |
|---|---|---|
| `max_length` | `2048` | Maximum total token length (prompt + response). |
| `top_p` | `0.7` | Nucleus sampling threshold. Lower = more focused, higher = more diverse. |
| `temperature` | `0.95` | Sampling randomness. Lower = more deterministic. |

> [!TIP]
> ChatGLM-6B uses relative position encoding, so it can technically accept arbitrarily long context. In practice, quality degrades once the total length exceeds **2048 tokens** (its training length).

---

## 5. Running the Demos

The repository includes several demos. Pick the one that matches how you want to interact with the model.

| Demo script | Interface | Extra dependencies | Notes |
|---|---|---|---|
| `cli_demo.py` | Command line | none beyond base | Streaming chat in your terminal. |
| `web_demo.py` | Gradio web UI | `gradio` | Typewriter effect, adjustable sliders. The current recommended web demo. |
| `web_demo2.py` | Streamlit web UI | `streamlit`, `streamlit_chat` | Community-contributed alternative UI. |
| `web_demo_old.py` | Gradio web UI | `gradio` | Older UI without the typewriter effect. |
| `api.py` | REST API (FastAPI) | `fastapi`, `uvicorn` | HTTP endpoint for programmatic access. |
| `cli_demo_vision.py` | Command line | VisualGLM deps | Multimodal — uses the **VisualGLM-6B** model. |
| `web_demo_vision.py` | Gradio web UI | VisualGLM deps | Multimodal image chat. |

### Command-line demo

```bash
python cli_demo.py
```

Interactive chat in the terminal. Commands: type `clear` to reset the conversation history, `stop` to exit. Press `Ctrl+C` mid-generation to interrupt the current response (it is caught and stops the stream rather than killing the program).

### Gradio web demo

Install Gradio, then launch:

```bash
pip install gradio
python web_demo.py
```

```text
Running on local URL:  http://127.0.0.1:7860
```

Open the printed URL in your browser. The interface gives you a chat box plus three sliders:

| Slider | Range | Default |
|---|---|---|
| Maximum length | 0 – 4096 | 2048 |
| Top P | 0 – 1 | 0.7 |
| Temperature | 0 – 1 | 0.95 |

The **Submit** button sends your message; **Clear History** resets the conversation.

> [!WARNING]
> The web demo launches with `share=False`, so the UI is served only on your local machine. To expose a temporary public link, edit the last line of `web_demo.py` to `demo.queue().launch(share=True, inbrowser=True)`. Public sharing routes all traffic through Gradio's relay servers, which noticeably slows the typewriter effect.

### Streamlit web demo

A community-contributed alternative UI:

```bash
pip install streamlit streamlit_chat
streamlit run web_demo2.py
```

The Streamlit demo exposes the same `max_length`, `top_p` (default `0.6` here), and `temperature` controls in a sidebar, and keeps up to 20 conversation turns.

### REST API

Install the API dependencies and start the server:

```bash
pip install fastapi uvicorn
python api.py
```

The server listens on `0.0.0.0:8000`. Send a chat request with a POST to `/`:

```bash
curl -X POST "http://127.0.0.1:8000" \
     -H 'Content-Type: application/json' \
     -d '{"prompt": "你好", "history": []}'
```

```json
{
  "response": "你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。",
  "history": [["你好", "你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。"]],
  "status": 200,
  "time": "2023-03-23 21:38:40"
}
```

The request body accepts these JSON fields:

| Field | Required | Default | Description |
|---|---|---|---|
| `prompt` | yes | — | The user message. |
| `history` | no | `null` | Prior turns as a list of `[query, response]` pairs. Pass back the `history` from the previous response to continue a conversation. |
| `max_length` | no | `2048` | Maximum total token length. |
| `top_p` | no | `0.7` | Nucleus sampling threshold. |
| `temperature` | no | `0.95` | Sampling randomness. |

> [!CAUTION]
> `api.py` binds to `0.0.0.0` (all network interfaces) and has **no authentication**. Anyone who can reach port 8000 can use your GPU and read every prompt. Only run it on a trusted network, or put it behind a reverse proxy that adds authentication. See the Admin Guide for hardening guidance.

### Multimodal demos (VisualGLM-6B)

`cli_demo_vision.py` and `web_demo_vision.py` run the separate **VisualGLM-6B** model (`THUDM/visualglm-6b`), which understands images. They require additional packages:

```bash
pip install torchvision
# Plus SwissArmyTransformer: https://github.com/THUDM/SwissArmyTransformer/
```

The vision web demo lets you upload an image and ask questions about it; the default decoding parameters differ slightly (`top_p` 0.4, `temperature` 0.8). For full details, see the [VisualGLM-6B repository](https://github.com/THUDM/VisualGLM-6B).

---

## 6. Running on Limited Hardware

### Quantization (INT8 / INT4)

By default the model loads at FP16 (~13 GB VRAM). To fit a smaller GPU, quantize at load time:

```python
# Only 4-bit and 8-bit quantization are supported
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(8).half().cuda()
```

Quantization first loads the FP16 model into RAM (~13 GB) before compressing it. If your **system RAM** is also tight, load a pre-quantized checkpoint instead — INT4 weights need only ~5.2 GB of RAM to load:

```python
# For INT8, change "chatglm-6b-int4" to "chatglm-6b-int8"
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()
```

> [!NOTE]
> Quantization costs some quality, but the model remains fluent even at INT4. Pre-quantized weight files are available from the [Tsinghua Cloud mirror](https://cloud.tsinghua.edu.cn/d/674208019e314311ab5c/).

### CPU inference

No GPU? Load in full precision on CPU (needs ~32 GB RAM):

```python
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).float()
```

For less RAM, load a pre-quantized checkpoint on CPU:

```python
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).float()
```

> [!IMPORTANT]
> CPU inference of a quantized model requires `gcc` and OpenMP to compile the quantization kernels at runtime. See [Requirements → Software](#software). If you see `Could not find module 'nvcuda.dll'` (Windows) or `RuntimeError: Unknown platform: darwin` (macOS), [load the model from a local folder](#step-3--get-the-model-weights) instead of from the Hub.

### Apple Silicon (Mac, MPS backend)

On an Apple Silicon or AMD-GPU Mac you can run on the GPU via PyTorch's MPS backend. Install [PyTorch-Nightly](https://developer.apple.com/metal/pytorch) (the version should be `2.1.0.dev2023xxxx`, **not** `2.0.0`), then load **from a local path** with the `mps` device:

```python
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).half().to('mps')
```

> [!WARNING]
> On macOS only local loading is supported. A half-precision model needs ~13 GB of free RAM; on a 16 GB machine the OS will swap to disk and inference becomes very slow. Prefer a quantized INT4 checkpoint — but note that the GPU quantization kernel is CUDA-only, so on Mac an INT4 model runs on **CPU** with `.float()`.

### Multiple GPUs

If no single GPU is large enough but you have several, split the model across them with the bundled helper:

```python
from utils import load_model_on_gpus

model = load_model_on_gpus("THUDM/chatglm-6b", num_gpus=2)
```

This evenly distributes the model's 28 transformer layers (plus embeddings and the output head) across `num_gpus` cards using `accelerate`. Install it first with `pip install accelerate`. To control the placement yourself, pass a `device_map` argument.

---

## 7. Fine-Tuning on Your Own Data

The `ptuning/` directory implements efficient fine-tuning based on **P-Tuning v2**, which trains only a small prefix (about **0.1%** of the model's parameters). Combined with INT4 quantization and gradient checkpointing, fine-tuning fits in as little as **7 GB of VRAM**. Full-parameter fine-tuning is also supported via DeepSpeed.

The examples below use **ADGEN**, an advertisement-generation dataset where the input is a set of clothing attributes (`content`) and the target is a marketing description (`summary`).

### Step 1 — Install fine-tuning dependencies

```bash
pip install rouge_chinese nltk jieba datasets
```

### Step 2 — Download the dataset

Download the processed ADGEN dataset from [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1), and unpack it so that the `AdvertiseGen/` directory sits inside `ptuning/`. Each line looks like:

```json
{"content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳", "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点..."}
```

### Step 3 — Train with P-Tuning v2

```bash
cd ptuning
bash train.sh
```

`train.sh` sets the two key knobs at the top — `PRE_SEQ_LEN=128` (the soft-prompt length) and `LR=2e-2` (the learning rate) — and trains for 3000 steps at INT4 quantization. With the default batch settings (`per_device_train_batch_size=1`, `gradient_accumulation_steps=16`, giving an effective batch size of 16), training needs about **6.7 GB of VRAM**. Checkpoints are written to `output/adgen-chatglm-6b-pt-128-2e-2/`.

> [!TIP]
> To speed up training when you have spare VRAM, increase `per_device_train_batch_size` and decrease `gradient_accumulation_steps` by the same factor — the effective batch size (their product) stays constant. Remove the `--quantization_bit 4` line to train at FP16.

### Step 4 — Evaluate

```bash
bash evaluate.sh
```

This runs prediction over the validation set and reports Chinese **Rouge** and **BLEU-4** scores. Generated text is saved to `output/adgen-chatglm-6b-pt-8-1e-2/generated_predictions.txt`.

### Step 5 — Use your fine-tuned model

A P-Tuning v2 checkpoint saves only the `PrefixEncoder` weights, so inference loads the original ChatGLM-6B **plus** your prefix:

```python
import os, torch
from transformers import AutoConfig, AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)

prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
new_prefix_state_dict = {
    k[len("transformer.prefix_encoder."):]: v
    for k, v in prefix_state_dict.items()
    if k.startswith("transformer.prefix_encoder.")
}
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

model = model.quantize(4).half().cuda()      # optional quantization
model.transformer.prefix_encoder.float()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])
```

> [!IMPORTANT]
> Set `pre_seq_len` to the value you trained with. The repository also ships a ready-made web demo for prefix checkpoints — run `bash web_demo.sh` (edit the checkpoint path inside it first).

### Multi-turn (chat) fine-tuning

To fine-tune on multi-turn conversations, provide a dataset with `prompt`, `response`, and `history` columns and run:

```bash
bash train_chat.sh
```

Each training row carries the prior turns in its `history` field, which the trainer automatically concatenates. Note that content beyond `max_source_length` (256 in `train_chat.sh`) is truncated.

### Full-parameter fine-tuning (DeepSpeed)

For full fine-tuning of all parameters, install [DeepSpeed](https://github.com/microsoft/DeepSpeed) and run:

```bash
bash ds_train_finetune.sh
```

This launches a 4-GPU DeepSpeed (ZeRO stage 2) job at FP16, learning rate `1e-4`. Evaluate a full-parameter checkpoint with `bash evaluate_finetune.sh`.

### Using your own dataset

Edit `train.sh` / `evaluate.sh` and change `--train_file`, `--validation_file`, and `--test_file` to your JSON file paths, and set `--prompt_column` / `--response_column` to the keys holding your input and output text. Increase `--max_source_length` / `--max_target_length` to fit your longest examples.

### Reference: fine-tuning method comparison

The maintainers' published results on ADGEN (`max_steps=3000`, source/target length 64):

| Metric | Finetune | P-Tuning v2 | LoRA |
|---|---|---|---|
| BLEU-4 | 8.01 | 8.10 | 7.62 |
| Rouge-1 | 31.23 | 31.12 | 30.60 |
| Rouge-2 | 7.36 | 7.11 | 6.96 |
| Rouge-l | 25.08 | 24.97 | 24.80 |
| Training Loss | 3.00 | 3.74 | 3.32 |

---

## 8. Troubleshooting

These entries are grounded in errors reported by the community in the project's issue tracker.

### Error: `Could not find module 'nvcuda.dll'`

**Cause:** On Windows, the model tried to load CUDA quantization kernels but no NVIDIA CUDA runtime is present (you are running CPU-only, or CUDA is not installed).

**Solution:** [Load the model from a local folder](#step-3--get-the-model-weights) rather than directly from the Hub, and use CPU inference (`.float()`). The remote-code path assumes a CUDA build; the local copy lets you run on CPU.

**Prevention:** On a CPU-only machine, always download the weights locally and load with `.float()`.

### Error: `clang: error: unsupported option '-fopenmp'`

**Cause:** On macOS, loading a quantized model compiles CUDA-free kernels that need OpenMP, which macOS does not ship.

**Solution:** Install OpenMP manually (example for clang 14.x):

```bash
curl -O https://mac.r-project.org/openmp/openmp-14.0.6-darwin20-Release.tar.gz
sudo tar fvxz openmp-14.0.6-darwin20-Release.tar.gz -C /
```

This installs `/usr/local/lib/libomp.dylib` and the OpenMP headers. The model then runs single-core on CPU.

**Prevention:** If a previous run failed, clear the Hugging Face module cache before retrying:

```bash
rm -rf ${HOME}/.cache/huggingface/modules/transformers_modules/chatglm-6b-int4
```

> [!CAUTION]
> That command uses `rm -rf`. Be certain of the path before running it.

### Error: `RuntimeError: Unknown platform: darwin`

**Cause:** Same root cause as the `nvcuda.dll` error — the CUDA quantization path does not exist on macOS.

**Solution:** [Load the model from a local path](#step-3--get-the-model-weights) and use the MPS or CPU backend as described in [Section 6](#6-running-on-limited-hardware).

### Problem: GPU memory is never released

**Cause:** After inference, the CUDA cache holds onto allocated memory; `torch.cuda.empty_cache()` alone does not always free it.

**Solution:** Free both the cache and inter-process handles, as `api.py` does:

```python
import torch
with torch.cuda.device("cuda:0"):
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
```

If memory is still pinned, the only reliable fix is to end the Python process.

### Error: tokenizer / `.generate()` failures after upgrading `transformers`

**Cause:** A `transformers` version newer than the model's remote code expects (e.g. errors like `ChatGLMTokenizer has no attribute vocab_size`, or `KeyError: '<|endoftext|>'`).

**Solution:** Pin the supported version:

```bash
pip install transformers==4.27.1
```

**Prevention:** Keep `transformers==4.27.1` in your environment for this model.

### Problem: Fine-tuning fails with `ValueError: None is not in list`

**Cause:** A column name in `train.sh` (`--prompt_column` / `--response_column` / `--history_column`) does not match a key in your dataset's JSON, so the trainer cannot find the expected field.

**Solution:** Verify that the column names you pass exactly match the keys in your JSON file, and that the dataset path is correct.

### Problem: Out-of-memory during inference or fine-tuning

**Cause:** The chosen quantization level exceeds your VRAM, or a long conversation has grown the context.

**Solution:** Drop to a lower quantization level (INT8 → INT4), start a fresh conversation to clear history, or — for fine-tuning — reduce `per_device_train_batch_size` and raise `gradient_accumulation_steps` to keep the effective batch size constant.

---

## 9. Frequently Asked Questions

**Q: How many parameters does ChatGLM-6B have, and what languages does it support?**
About 6.2 billion parameters. It is bilingual, optimized primarily for Chinese with English support. English answers are weaker than Chinese ones because the training data was predominantly Chinese.

**Q: Do I need a GPU?**
No, but it helps a lot. CPU inference works with ~32 GB of RAM (or ~5.2 GB if you load a pre-quantized INT4 checkpoint) but is much slower than GPU inference.

**Q: What is the smallest GPU that can run it?**
A 6 GB card at INT4 quantization. See the [hardware table](#hardware).

**Q: Why does loading require `trust_remote_code=True`?**
The model's architecture lives in Python files published alongside the weights on the Hugging Face Hub. `transformers` only executes that code when you explicitly opt in. Only load models from sources you trust.

**Q: Can I use this commercially?**
The code is licensed under Apache-2.0. The model **weights** are governed by a separate [Model License](https://github.com/THUDM/ChatGLM-6B/blob/main/MODEL_LICENSE) — fully open for academic research, and free for commercial use after registering via the maintainers' [questionnaire](https://open.bigmodel.cn/mla/form).

**Q: How do I make output deterministic?**
Lower `temperature` toward 0 and `top_p` toward a small value. Generation still uses sampling, so it is not perfectly reproducible, but lower values make it far more stable.

**Q: My English answers contain stray Chinese words. Is that a bug?**
It is a known limitation of the model's small size and Chinese-heavy training data. Pinning `revision="v1.1.0"` reduces it — that checkpoint rebalanced the language mix.

**Q: Can ChatGLM-6B do math and coding reliably?**
Not reliably. Its 6 B size limits factual recall and logical reasoning; it can produce incorrect math and code. For stronger capabilities, use a newer model in the family (ChatGLM2/3, GLM-4).

**Q: How do I fine-tune on multi-turn conversations instead of single Q&A pairs?**
Use a dataset with `prompt`, `response`, and `history` columns and run `bash train_chat.sh`. See [Section 7](#multi-turn-chat-fine-tuning).

**Q: What's the difference between P-Tuning v2 and full fine-tuning here?**
P-Tuning v2 (`train.sh`) trains only a ~0.1% prefix and fits in 7 GB VRAM. Full fine-tuning (`ds_train_finetune.sh`) updates every parameter, needs DeepSpeed and multiple GPUs, and produces marginally higher scores (see the [comparison table](#reference-fine-tuning-method-comparison)).

**Q: Should I use ChatGLM-6B for a new project?**
Consider the newer models first — [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B), [ChatGLM3](https://github.com/THUDM/ChatGLM3), and [GLM-4](https://github.com/THUDM/GLM-4) all improve on it. ChatGLM-6B remains useful for reproducing first-generation results and for very constrained hardware.

---

## 10. Reference

### Demo scripts at a glance

| Script | Launch command | Model used |
|---|---|---|
| `cli_demo.py` | `python cli_demo.py` | `THUDM/chatglm-6b` |
| `web_demo.py` | `python web_demo.py` | `THUDM/chatglm-6b` |
| `web_demo2.py` | `streamlit run web_demo2.py` | `THUDM/chatglm-6b` |
| `web_demo_old.py` | `python web_demo_old.py` | `THUDM/chatglm-6b` |
| `api.py` | `python api.py` | `THUDM/chatglm-6b` |
| `cli_demo_vision.py` | `python cli_demo_vision.py` | `THUDM/visualglm-6b` |
| `web_demo_vision.py` | `python web_demo_vision.py` | `THUDM/visualglm-6b` |

### Model checkpoints on the Hub

| Checkpoint | Precision | Load with |
|---|---|---|
| `THUDM/chatglm-6b` | FP16 | `.half().cuda()` |
| `THUDM/chatglm-6b-int8` | INT8 (pre-quantized) | `.half().cuda()` |
| `THUDM/chatglm-6b-int4` | INT4 (pre-quantized) | `.half().cuda()` |
| `THUDM/visualglm-6b` | FP16 (multimodal) | `.half().cuda()` |

### Generation defaults

| Parameter | `model.chat` default | `web_demo.py` slider | `web_demo2.py` slider |
|---|---|---|---|
| `max_length` | 2048 | 2048 (0–4096) | 2048 (0–4096) |
| `top_p` | 0.7 | 0.7 (0–1) | 0.6 (0–1) |
| `temperature` | 0.95 | 0.95 (0–1) | 0.95 (0–1) |

### CLI demo commands

| Input | Action |
|---|---|
| (any text) | Send a message to the model |
| `clear` | Reset the conversation history |
| `stop` | Exit the program |
| `Ctrl+C` | Interrupt the current response |

### Fine-tuning scripts

| Script | Method | GPUs | Key parameters |
|---|---|---|---|
| `train.sh` | P-Tuning v2 | 1 | `PRE_SEQ_LEN=128`, `LR=2e-2`, `quantization_bit=4`, `max_steps=3000` |
| `train_chat.sh` | P-Tuning v2 (multi-turn) | 1 | `LR=1e-2`, `max_source_length=256`, history column |
| `evaluate.sh` | P-Tuning v2 prediction | 1 | loads `ptuning_checkpoint`, reports Rouge + BLEU-4 |
| `ds_train_finetune.sh` | Full fine-tune (DeepSpeed) | 4 | ZeRO stage 2, FP16, `LR=1e-4`, `max_steps=5000` |
| `evaluate_finetune.sh` | Full-finetune prediction | 1 | `fp16_full_eval` |
| `web_demo.sh` | Web demo with prefix checkpoint | 1 | `pre_seq_len=128` |

---

_Generated June 2026 from zai-org/ChatGLM-6B at 401bf3a. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._