ChatGLM-6B User Guide

ChatGLM-6B is an open, bilingual (Chinese–English) conversational language model with 6.2 billion parameters, built on the General Language Model (GLM) architecture. Thanks to built-in quantization, you can run it locally on a consumer GPU — as little as 6 GB of VRAM at the INT4 quantization level.

This repository does not contain the model weights themselves. It ships a small set of ready-to-run demos (command line, web UI, and a REST API) plus a P-Tuning v2 fine-tuning harness that wrap the published THUDM/chatglm-6b weights on the Hugging Face Hub. This guide walks you through installing the dependencies, loading the model, running each demo, shrinking the memory footprint with quantization, and fine-tuning the model on your own data.

Note

ChatGLM-6B is the first-generation model in the GLM chat family. The maintainers have since released ChatGLM2-6B, ChatGLM3, and GLM-4, each with stronger performance, longer context, and faster inference. If you are starting a new project, evaluate the newer models first. This guide documents ChatGLM-6B exactly as it exists in this repository.

1. Quick Start

If you already have a CUDA GPU with at least 13 GB of VRAM and a working Python environment, you can be chatting with the model in three steps.

# 1. Get the demo scripts
git clone https://github.com/THUDM/ChatGLM-6B
cd ChatGLM-6B

# 2. Install dependencies
pip install -r requirements.txt

# 3. Launch the command-line chat demo
python cli_demo.py

The first run downloads the model weights (~13 GB) from the Hugging Face Hub automatically. Once loaded, type a message and press Enter:

欢迎使用 ChatGLM-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序

用户：你好

ChatGLM-6B：你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。

Type clear to wipe the conversation history, or stop to quit.

Tip

Short on VRAM? Skip ahead to Section 6: Running on Limited Hardware. With INT4 quantization the model fits in 6 GB, and it can even run on CPU (with ~32 GB of RAM).

2. Requirements

Hardware

The memory you need depends on the quantization level you load the model at. This table is the single most important reference in this guide — consult it before you do anything else.

Important

These are the model's own published figures. VRAM use grows as a conversation gets longer (the context accumulates), so leave headroom above the minimum. After 2–3 turns, INT8 settles around 10 GB and INT4 around 6 GB.

CPU-only inference is possible but slow, and needs roughly 32 GB of system RAM at FP16 (much less if you load a pre-quantized INT4 checkpoint — about 5.2 GB).

Software

All of these are pinned in requirements.txt and installed by pip install -r requirements.txt.

Warning

The transformers version matters. The model's remote code was written against transformers==4.27.1. Newer versions (e.g. 4.34+) can raise tokenizer or .generate() errors. If you hit a loading error, pin transformers to 4.27.1 first.

For CPU inference of a quantized model you additionally need a C compiler with OpenMP:

Linux

Most distributions already include gcc with OpenMP. The model was tested with gcc 11.3.0.

Windows

Install TDM-GCC and tick the openmp option during setup. Tested with TDM-GCC 10.3.0.

macOS

macOS lacks OpenMP by default. Install it manually — see Troubleshooting → clang: error: unsupported option '-fopenmp'.

3. Installation

Step 1 — Clone the repository

git clone https://github.com/THUDM/ChatGLM-6B
cd ChatGLM-6B

Step 2 — Install dependencies

pip install -r requirements.txt

Successfully installed transformers-4.27.1 torch-2.x cpm_kernels-x.x.x gradio-x.x.x ...

Step 3 — Get the model weights

You have two options.

Option A — automatic download (default). Every demo calls AutoModel.from_pretrained("THUDM/chatglm-6b", ...), which downloads the weights and the model implementation from the Hugging Face Hub on first run. No extra steps required — just run a demo.

Option B — manual download (slow or unreliable networks). Download the weights ahead of time, then point the demos at a local folder.

# Install Git LFS first: https://git-lfs.com
git clone https://huggingface.co/THUDM/chatglm-6b

If the large weight files download slowly, fetch only the code first and then add the weights by hand:

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/THUDM/chatglm-6b

Then download the parameter files from the Tsinghua Cloud mirror and place them into your local chatglm-6b directory. Finally, replace THUDM/chatglm-6b in the demo scripts with the path to that folder.

Tip

The model implementation evolves over time. To pin a specific revision for reproducibility, add revision="v1.1.0" to the from_pretrained call, or run git checkout v1.1.0 inside a locally cloned weights folder. v1.1.0 rebalances the Chinese/English training mix and reduces stray Chinese words in English answers.

4. Calling the Model From Python

Every demo in this repository is a thin wrapper around the same three calls: load a tokenizer, load the model, then call model.chat(...) or model.stream_chat(...). You can use the model directly in your own code the same way.

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])
print(response)

你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。

To continue a multi-turn conversation, pass the returned history back into the next call:

response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
print(response)

Note

trust_remote_code=True is required. The model's architecture is defined in Python files that ship alongside the weights on the Hub, and transformers only executes them when you opt in. See the Admin Guide's Security section for what this implies.

Streaming token-by-token

For a typewriter effect, use stream_chat, which yields the partial response as it is generated:

for response, history in model.stream_chat(tokenizer, "讲个笑话", history=[]):
    print(response)

Each iteration returns the full response generated so far, not just the newest token.

Generation parameters

model.chat / model.stream_chat accept these keyword arguments. The defaults below are what the demos use.

Tip

ChatGLM-6B uses relative position encoding, so it can technically accept arbitrarily long context. In practice, quality degrades once the total length exceeds 2048 tokens (its training length).

5. Running the Demos

The repository includes several demos. Pick the one that matches how you want to interact with the model.

Command-line demo

python cli_demo.py

Interactive chat in the terminal. Commands: type clear to reset the conversation history, stop to exit. Press Ctrl+C mid-generation to interrupt the current response (it is caught and stops the stream rather than killing the program).

Gradio web demo

Install Gradio, then launch:

pip install gradio
python web_demo.py

Running on local URL:  http://127.0.0.1:7860

Open the printed URL in your browser. The interface gives you a chat box plus three sliders:

The Submit button sends your message; Clear History resets the conversation.

Warning

The web demo launches with share=False, so the UI is served only on your local machine. To expose a temporary public link, edit the last line of web_demo.py to demo.queue().launch(share=True, inbrowser=True). Public sharing routes all traffic through Gradio's relay servers, which noticeably slows the typewriter effect.

Streamlit web demo

A community-contributed alternative UI:

pip install streamlit streamlit_chat
streamlit run web_demo2.py

The Streamlit demo exposes the same max_length, top_p (default 0.6 here), and temperature controls in a sidebar, and keeps up to 20 conversation turns.

REST API

Install the API dependencies and start the server:

pip install fastapi uvicorn
python api.py

The server listens on 0.0.0.0:8000. Send a chat request with a POST to /:

curl -X POST "http://127.0.0.1:8000" \
     -H 'Content-Type: application/json' \
     -d '{"prompt": "你好", "history": []}'

{
  "response": "你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。",
  "history": [["你好", "你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。"]],
  "status": 200,
  "time": "2023-03-23 21:38:40"
}

The request body accepts these JSON fields:

Caution

api.py binds to 0.0.0.0 (all network interfaces) and has no authentication. Anyone who can reach port 8000 can use your GPU and read every prompt. Only run it on a trusted network, or put it behind a reverse proxy that adds authentication. See the Admin Guide for hardening guidance.

Multimodal demos (VisualGLM-6B)

cli_demo_vision.py and web_demo_vision.py run the separate VisualGLM-6B model (THUDM/visualglm-6b), which understands images. They require additional packages:

pip install torchvision
# Plus SwissArmyTransformer: https://github.com/THUDM/SwissArmyTransformer/

The vision web demo lets you upload an image and ask questions about it; the default decoding parameters differ slightly (top_p 0.4, temperature 0.8). For full details, see the VisualGLM-6B repository.

6. Running on Limited Hardware

Quantization (INT8 / INT4)

By default the model loads at FP16 (~13 GB VRAM). To fit a smaller GPU, quantize at load time:

# Only 4-bit and 8-bit quantization are supported
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(8).half().cuda()

Quantization first loads the FP16 model into RAM (~13 GB) before compressing it. If your system RAM is also tight, load a pre-quantized checkpoint instead — INT4 weights need only ~5.2 GB of RAM to load:

# For INT8, change "chatglm-6b-int4" to "chatglm-6b-int8"
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()

Note

Quantization costs some quality, but the model remains fluent even at INT4. Pre-quantized weight files are available from the Tsinghua Cloud mirror.

CPU inference

No GPU? Load in full precision on CPU (needs ~32 GB RAM):

model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).float()

For less RAM, load a pre-quantized checkpoint on CPU:

model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).float()

Important

CPU inference of a quantized model requires gcc and OpenMP to compile the quantization kernels at runtime. See Requirements → Software. If you see Could not find module 'nvcuda.dll' (Windows) or RuntimeError: Unknown platform: darwin (macOS), load the model from a local folder instead of from the Hub.

Apple Silicon (Mac, MPS backend)

On an Apple Silicon or AMD-GPU Mac you can run on the GPU via PyTorch's MPS backend. Install PyTorch-Nightly (the version should be 2.1.0.dev2023xxxx, not 2.0.0), then load from a local path with the mps device:

model = AutoModel.from_pretrained("your local path", trust_remote_code=True).half().to('mps')

Warning

On macOS only local loading is supported. A half-precision model needs ~13 GB of free RAM; on a 16 GB machine the OS will swap to disk and inference becomes very slow. Prefer a quantized INT4 checkpoint — but note that the GPU quantization kernel is CUDA-only, so on Mac an INT4 model runs on CPU with .float().

Multiple GPUs

If no single GPU is large enough but you have several, split the model across them with the bundled helper:

from utils import load_model_on_gpus

model = load_model_on_gpus("THUDM/chatglm-6b", num_gpus=2)

This evenly distributes the model's 28 transformer layers (plus embeddings and the output head) across num_gpus cards using accelerate. Install it first with pip install accelerate. To control the placement yourself, pass a device_map argument.

7. Fine-Tuning on Your Own Data

The ptuning/ directory implements efficient fine-tuning based on P-Tuning v2, which trains only a small prefix (about 0.1% of the model's parameters). Combined with INT4 quantization and gradient checkpointing, fine-tuning fits in as little as 7 GB of VRAM. Full-parameter fine-tuning is also supported via DeepSpeed.

The examples below use ADGEN, an advertisement-generation dataset where the input is a set of clothing attributes (content) and the target is a marketing description (summary).

Step 1 — Install fine-tuning dependencies

pip install rouge_chinese nltk jieba datasets

Step 2 — Download the dataset

Download the processed ADGEN dataset from Google Drive or Tsinghua Cloud, and unpack it so that the AdvertiseGen/ directory sits inside ptuning/. Each line looks like:

{"content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳", "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点..."}

Step 3 — Train with P-Tuning v2

cd ptuning
bash train.sh

train.sh sets the two key knobs at the top — PRE_SEQ_LEN=128 (the soft-prompt length) and LR=2e-2 (the learning rate) — and trains for 3000 steps at INT4 quantization. With the default batch settings (per_device_train_batch_size=1, gradient_accumulation_steps=16, giving an effective batch size of 16), training needs about 6.7 GB of VRAM. Checkpoints are written to output/adgen-chatglm-6b-pt-128-2e-2/.

Tip

To speed up training when you have spare VRAM, increase per_device_train_batch_size and decrease gradient_accumulation_steps by the same factor — the effective batch size (their product) stays constant. Remove the --quantization_bit 4 line to train at FP16.

Step 4 — Evaluate

bash evaluate.sh

This runs prediction over the validation set and reports Chinese Rouge and BLEU-4 scores. Generated text is saved to output/adgen-chatglm-6b-pt-8-1e-2/generated_predictions.txt.

Step 5 — Use your fine-tuned model

A P-Tuning v2 checkpoint saves only the PrefixEncoder weights, so inference loads the original ChatGLM-6B plus your prefix:

import os, torch
from transformers import AutoConfig, AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)

prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
new_prefix_state_dict = {
    k[len("transformer.prefix_encoder."):]: v
    for k, v in prefix_state_dict.items()
    if k.startswith("transformer.prefix_encoder.")
}
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

model = model.quantize(4).half().cuda()      # optional quantization
model.transformer.prefix_encoder.float()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])

Important

Set pre_seq_len to the value you trained with. The repository also ships a ready-made web demo for prefix checkpoints — run bash web_demo.sh (edit the checkpoint path inside it first).

Multi-turn (chat) fine-tuning

To fine-tune on multi-turn conversations, provide a dataset with prompt, response, and history columns and run:

bash train_chat.sh

Each training row carries the prior turns in its history field, which the trainer automatically concatenates. Note that content beyond max_source_length (256 in train_chat.sh) is truncated.

Full-parameter fine-tuning (DeepSpeed)

For full fine-tuning of all parameters, install DeepSpeed and run:

bash ds_train_finetune.sh

This launches a 4-GPU DeepSpeed (ZeRO stage 2) job at FP16, learning rate 1e-4. Evaluate a full-parameter checkpoint with bash evaluate_finetune.sh.

Using your own dataset

Edit train.sh / evaluate.sh and change --train_file, --validation_file, and --test_file to your JSON file paths, and set --prompt_column / --response_column to the keys holding your input and output text. Increase --max_source_length / --max_target_length to fit your longest examples.

Reference: fine-tuning method comparison

The maintainers' published results on ADGEN (max_steps=3000, source/target length 64):

8. Troubleshooting

These entries are grounded in errors reported by the community in the project's issue tracker.

Error: Could not find module 'nvcuda.dll'

Cause: On Windows, the model tried to load CUDA quantization kernels but no NVIDIA CUDA runtime is present (you are running CPU-only, or CUDA is not installed).

Solution: Load the model from a local folder rather than directly from the Hub, and use CPU inference (.float()). The remote-code path assumes a CUDA build; the local copy lets you run on CPU.

Prevention: On a CPU-only machine, always download the weights locally and load with .float().

Error: clang: error: unsupported option '-fopenmp'

Cause: On macOS, loading a quantized model compiles CUDA-free kernels that need OpenMP, which macOS does not ship.

Solution: Install OpenMP manually (example for clang 14.x):

curl -O https://mac.r-project.org/openmp/openmp-14.0.6-darwin20-Release.tar.gz
sudo tar fvxz openmp-14.0.6-darwin20-Release.tar.gz -C /

This installs /usr/local/lib/libomp.dylib and the OpenMP headers. The model then runs single-core on CPU.

Prevention: If a previous run failed, clear the Hugging Face module cache before retrying:

rm -rf ${HOME}/.cache/huggingface/modules/transformers_modules/chatglm-6b-int4

Caution

That command uses rm -rf. Be certain of the path before running it.

Error: RuntimeError: Unknown platform: darwin

Cause: Same root cause as the nvcuda.dll error — the CUDA quantization path does not exist on macOS.

Solution: Load the model from a local path and use the MPS or CPU backend as described in Section 6.

Problem: GPU memory is never released

Cause: After inference, the CUDA cache holds onto allocated memory; torch.cuda.empty_cache() alone does not always free it.

Solution: Free both the cache and inter-process handles, as api.py does:

import torch
with torch.cuda.device("cuda:0"):
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()

If memory is still pinned, the only reliable fix is to end the Python process.

Error: tokenizer / .generate() failures after upgrading transformers

Cause: A transformers version newer than the model's remote code expects (e.g. errors like ChatGLMTokenizer has no attribute vocab_size, or KeyError: '<|endoftext|>').

Solution: Pin the supported version:

pip install transformers==4.27.1

Prevention: Keep transformers==4.27.1 in your environment for this model.

Problem: Fine-tuning fails with ValueError: None is not in list

Cause: A column name in train.sh (--prompt_column / --response_column / --history_column) does not match a key in your dataset's JSON, so the trainer cannot find the expected field.

Solution: Verify that the column names you pass exactly match the keys in your JSON file, and that the dataset path is correct.

Problem: Out-of-memory during inference or fine-tuning

Cause: The chosen quantization level exceeds your VRAM, or a long conversation has grown the context.

Solution: Drop to a lower quantization level (INT8 → INT4), start a fresh conversation to clear history, or — for fine-tuning — reduce per_device_train_batch_size and raise gradient_accumulation_steps to keep the effective batch size constant.

9. Frequently Asked Questions

Q: How many parameters does ChatGLM-6B have, and what languages does it support? About 6.2 billion parameters. It is bilingual, optimized primarily for Chinese with English support. English answers are weaker than Chinese ones because the training data was predominantly Chinese.

Q: Do I need a GPU? No, but it helps a lot. CPU inference works with ~32 GB of RAM (or ~5.2 GB if you load a pre-quantized INT4 checkpoint) but is much slower than GPU inference.

Q: What is the smallest GPU that can run it? A 6 GB card at INT4 quantization. See the hardware table.

Q: Why does loading require trust_remote_code=True? The model's architecture lives in Python files published alongside the weights on the Hugging Face Hub. transformers only executes that code when you explicitly opt in. Only load models from sources you trust.

Q: Can I use this commercially? The code is licensed under Apache-2.0. The model weights are governed by a separate Model License — fully open for academic research, and free for commercial use after registering via the maintainers' questionnaire.

Q: How do I make output deterministic? Lower temperature toward 0 and top_p toward a small value. Generation still uses sampling, so it is not perfectly reproducible, but lower values make it far more stable.

Q: My English answers contain stray Chinese words. Is that a bug? It is a known limitation of the model's small size and Chinese-heavy training data. Pinning revision="v1.1.0" reduces it — that checkpoint rebalanced the language mix.

Q: Can ChatGLM-6B do math and coding reliably? Not reliably. Its 6 B size limits factual recall and logical reasoning; it can produce incorrect math and code. For stronger capabilities, use a newer model in the family (ChatGLM2/3, GLM-4).

Q: How do I fine-tune on multi-turn conversations instead of single Q&A pairs? Use a dataset with prompt, response, and history columns and run bash train_chat.sh. See Section 7.

Q: What's the difference between P-Tuning v2 and full fine-tuning here? P-Tuning v2 (train.sh) trains only a ~0.1% prefix and fits in 7 GB VRAM. Full fine-tuning (ds_train_finetune.sh) updates every parameter, needs DeepSpeed and multiple GPUs, and produces marginally higher scores (see the comparison table).

Q: Should I use ChatGLM-6B for a new project? Consider the newer models first — ChatGLM2-6B, ChatGLM3, and GLM-4 all improve on it. ChatGLM-6B remains useful for reproducing first-generation results and for very constrained hardware.

10. Reference

Demo scripts at a glance

Model checkpoints on the Hub

Generation defaults

CLI demo commands

Fine-tuning scripts