bitnet.cpp User Guide

bitnet.cpp is the official inference framework for 1-bit large language models — models whose weights are stored in roughly 1.58 bits each (the ternary values −1, 0, +1) instead of the usual 16 or 32 bits. It lets you run models such as BitNet b1.58 fast and losslessly on an ordinary CPU, and (through a separate code path) on an NVIDIA GPU. A 100-billion-parameter BitNet model can run on a single CPU at roughly human reading speed (5–7 tokens per second), which makes capable local language models practical on everyday hardware.

This guide shows you how to install bitnet.cpp, build a model, run inference from the command line and as a server, benchmark performance, and convert your own models. Everything here is drawn directly from the framework's scripts and configuration as they exist at the documented commit.

Note

bitnet.cpp is built on top of the llama.cpp inference engine and uses lookup-table kernel techniques pioneered by T-MAC. When you run inference, the Python scripts in this project drive a compiled llama-cli / llama-server binary under the hood. You do not need to understand llama.cpp to use bitnet.cpp, but it explains why the build step compiles a large C++ project.

1. Quick Start

The fastest path from nothing to a running chat model is three commands after installing the prerequisites.

# 1. Clone the repository (the --recursive flag pulls in the bundled llama.cpp)
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

# 2. Create the environment and install Python dependencies
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp
pip install -r requirements.txt

# 3. Download a model, build the kernels, and quantize — all in one command
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

Once setup_env.py finishes, run the model in chat mode:

python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv

> You are a helpful assistant
Hello! How can I help you today?

Important

The --recursive flag on git clone is mandatory. bitnet.cpp embeds llama.cpp as a Git submodule under 3rdparty/llama.cpp, and the build will fail without it. If you already cloned without --recursive, run git submodule update --init --recursive inside the repository.

Tip

The -cnv (conversation) flag turns on chat mode and is intended for instruct models. When -cnv is on, the text you pass to -p becomes the system prompt rather than a one-shot completion prompt.

2. Installation

2.1 Requirements

Note

A C++ toolchain is required because building a model compiles the bundled llama.cpp engine and the BitNet ternary kernels. This is a one-time step per machine/model, not something you do on every run.

2.2 Installing the compiler

Debian / Ubuntu

Use the official LLVM installer script to get a recent Clang:

bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"

Windows

Install Visual Studio 2022. In the installer, enable at least:

• Desktop development with C++
• C++ CMake Tools for Windows
• Git for Windows
• C++ Clang Compiler for Windows
• MS-Build Support for LLVM (clang) Toolset

These options also install CMake and the other tools you need.

macOS

Install the Xcode command-line tools and a recent Clang via Homebrew (brew install llvm cmake). The Apple Silicon (ARM) path uses the I2_S and TL1 kernels.

Warning

On Windows you must run every build command from a Developer Command Prompt or Developer PowerShell for VS 2022, not a plain terminal. Otherwise CMake cannot find clang / clang++. See Troubleshooting for the exact environment-initialization commands.

2.3 Install Python dependencies

# (Recommended) isolated environment
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp

pip install -r requirements.txt

Note

requirements.txt pulls in the conversion dependencies from the bundled llama.cpp (the various convert_* requirement files). You do not add packages to it directly; the model-conversion utilities rely on those exact pinned versions.

3. Preparing a Model with setup_env.py

setup_env.py is the one-stop build-and-prepare tool. A single invocation performs four steps in order: installs the GGUF Python helper, generates or copies the compute kernels, compiles the C++ project, and downloads + converts + quantizes your model into a ready-to-run .gguf file.

3.1 Two ways to point it at a model

You can either let it download a known model from Hugging Face, or point it at a directory you already have.

Download by Hugging Face repo

python setup_env.py --hf-repo microsoft/BitNet-b1.58-2B-4T -q i2_s

Use a local model directory

huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

3.2 Options

Important

The valid --quant-type values depend on your CPU:

Architecture	Allowed quant types
ARM (arm64 / aarch64 / Apple Silicon)	i2_s, tl1
x86-64 (Intel / AMD)	i2_s, tl2

Passing a type that doesn't match your architecture is rejected. i2_s works everywhere and is the default.

3.3 What the build produces

After setup_env.py completes you will have:

• A compiled engine under build/bin/ (llama-cli, llama-server, llama-quantize, and the other llama.cpp tools).
• A quantized model file named ggml-model-<quant-type>.gguf inside your model directory (for example models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf).
• Per-step logs in your --log-dir (default logs/) — download_model.log, convert_to_f32_gguf.log, quantize_to_i2s.log, codegen.log, compile.log, and so on. These are where to look first if a step fails.

Tip

If setup_env.py reports that a step failed, open the matching log file named after that step in your log directory. The error message tells you which command failed and why — this is far more informative than the one-line console summary.

Note

If the target .gguf already exists and is non-empty, the model-preparation step is skipped and reused. To force a fresh conversion, delete the existing ggml-model-*.gguf first.

4. Supported Models

bitnet.cpp demonstrates inference using existing 1-bit LLMs published on Hugging Face. The table below lists the models the framework officially recognizes, with the kernel support per CPU type. ✅ = supported, ❌ = not built for that combination.

The full set of Hugging Face repositories accepted by --hf-repo is:

• microsoft/BitNet-b1.58-2B-4T
• 1bitLLM/bitnet_b1_58-large
• 1bitLLM/bitnet_b1_58-3B
• HF1BitLLM/Llama3-8B-1.58-100B-tokens
• tiiuae/Falcon3-1B-Instruct-1.58bit, Falcon3-1B-1.58bit
• tiiuae/Falcon3-3B-Instruct-1.58bit, Falcon3-3B-1.58bit
• tiiuae/Falcon3-7B-Instruct-1.58bit, Falcon3-7B-1.58bit
• tiiuae/Falcon3-10B-Instruct-1.58bit, Falcon3-10B-1.58bit
• tiiuae/Falcon-E-1B-Instruct, Falcon-E-1B-Base
• tiiuae/Falcon-E-3B-Instruct, Falcon-E-3B-Base

Note

These are demonstration models, not the limit of what is possible. The framework's purpose is to show that 1-bit inference is fast and accurate; the maintainers hope it encourages more 1-bit models at larger scales.

5. Running Inference

5.1 Single prompt and chat mode

run_inference.py is the command-line inference tool. It locates the compiled llama-cli binary in build/bin/ and runs your prompt through it.

# One-shot completion
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "Microsoft Corporation is an American multinational corporation and technology company headquartered in"

# Interactive chat (instruct models)
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "You are a helpful assistant" -cnv

Options

Tip

Set --threads to roughly the number of physical cores on your machine for best throughput. The default of 2 is conservative; on an 8-core desktop, -t 8 is usually faster.

Note

bitnet.cpp runs entirely on the CPU here — run_inference.py always passes -ngl 0 (zero GPU layers) to the engine. GPU acceleration is a separate code path covered in Section 8.

5.2 Running as an HTTP server

run_inference_server.py starts a long-running HTTP server (the llama.cpp llama-server) so you can send requests over the network with continuous batching enabled.

python run_inference_server.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  --host 127.0.0.1 --port 8080

Starting server on 127.0.0.1:8080

Options

Warning

The server binds to 127.0.0.1 (localhost only) by default. Setting --host 0.0.0.0 exposes it to your network — only do this on a trusted network, and put authentication/TLS in front of it before exposing it publicly. bitnet.cpp's server has no built-in authentication.

Note

Chat mode (-cnv) is not available on the server — that flag is deliberately omitted because llama-server does not support it. Use the server's HTTP API (inherited from llama.cpp) for conversational requests instead.

6. Benchmarking Performance

bitnet.cpp ships an end-to-end benchmark so you can measure throughput on your own hardware.

python utils/e2e_benchmark.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -n 200 -p 256 -t 4

6.1 Benchmarking without a real model

If you want to measure kernel performance for a layout that no public model uses, generate a dummy model of the right shape and benchmark it:

# Generate a dummy 125M model in the TL1 layout
python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large \
  --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M

# Benchmark it: process 512 prompt tokens, generate 128
python utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128

Tip

Use the dummy-model workflow to estimate how a hypothetical model size would perform on your machine before committing to downloading and converting real weights.

6.2 What to expect

Published results give a sense of the speedups over standard full-precision inference (your numbers will vary with hardware and threads):

• ARM CPUs: 1.37×–5.07× faster, with energy use cut by 55.4%–70.0%.
• x86 CPUs: 2.37×–6.17× faster, with energy use cut by 71.9%–82.2%.
• Larger models see the bigger speedups.

7. Converting Your Own Models

If you have BitNet-style weights that aren't already in GGUF form, the utils/ directory has conversion tools. The most common path is converting a .safetensors checkpoint.

# 1. Download the bf16 safetensors weights
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 \
  --local-dir ./models/bitnet-b1.58-2B-4T-bf16

# 2. Convert to a GGUF model
python ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16

7.1 Conversion and tuning tools at a glance

Note

Most users never call these directly — setup_env.py invokes the right converter for you. Reach for them when you're bringing a new model or tuning performance. The complete options for each are in the Admin Guide.

7.2 Quantizing embeddings for extra speed

The token-embedding layer can be stored in a smaller format to save memory and gain speed. The easiest way is to pass --quant-embd to setup_env.py, which converts embeddings to Q6_K automatically:

python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s --quant-embd

Tip

Q6_K is the recommended embedding format — it preserves accuracy almost perfectly while reducing memory and improving throughput. Quantizing the embeddings all the way down to I2_S is not supported (it causes the model to fail), so stick with the default --quant-embd behavior.

8. GPU Inference (Experimental)

In addition to the primary CPU framework, bitnet.cpp includes a separate GPU inference kernel in the gpu/ directory. It implements a custom W2A8 (2-bit weight × 8-bit activation) CUDA kernel for NVIDIA GPUs and is tuned for the BitNet-b1.58-2B-4T model.

Important

The GPU path is independent of the CPU path described above. It has its own dependencies, its own build step, and its own conversion scripts. It targets NVIDIA Ampere-class GPUs (compute capability 8.0, e.g. A100) and requires a CUDA toolchain plus PyTorch ≥ 2.2 with xformers.

8.1 Build and test the kernel

# Separate environment for the GPU path
conda create --name bitnet-gpu "python<3.13"
conda activate bitnet-gpu
pip install -r gpu/requirements.txt

# Compile the CUDA kernel into a shared library
cd gpu/bitnet_kernels
bash compile.sh
cd ..

# Run the kernel performance tests
python test.py

8.2 End-to-end GPU generation

# Download and convert the 2B model into the int2 + fp16 checkpoint pair
mkdir checkpoints
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 \
  --local-dir ./checkpoints/bitnet-b1.58-2B-4T-bf16
python ./convert_safetensors.py \
  --safetensors_file ./checkpoints/bitnet-b1.58-2B-4T-bf16/model.safetensors \
  --output checkpoints/model_state.pt --model_name 2B
python ./convert_checkpoint.py --input ./checkpoints/model_state.pt
rm ./checkpoints/model_state.pt

# Interactive chat on the GPU
python3 ./generate.py ./checkpoints/ --interactive --chat_format

generate.py accepts: a checkpoint directory (positional), --interactive for a prompt loop, --chat_format for chat-style turns, and --sampling to use top-p sampling instead of greedy decoding.

Note

The GPU kernel reports performance like Shape(2560, 2560), W2A8: 13.32us, torch BF16: 18.32us from test.py. On an A100, the custom kernel is roughly 1.3×–3.6× faster than the equivalent BF16 matrix multiply, and end-to-end generation is around 3× faster than a comparable BF16 2B model.

9. Troubleshooting

Error: 'clang' is not recognized as an internal or external command

Cause: On Windows, your terminal isn't initialized with the Visual Studio tools, so CMake cannot find the Clang compiler.

Solution: Initialize the VS environment before building. First verify with clang -v. If it errors, run the appropriate command for your shell:

:: Command Prompt
"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64

# PowerShell
Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"
Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"

Prevention: Always build from a Developer Command Prompt / Developer PowerShell for VS 2022 on Windows.

Error: build fails in log.cpp with std::chrono errors

Cause: A known incompatibility introduced in a recent version of the bundled llama.cpp.

Solution: Apply the upstream fix referenced in the project FAQ (the tinglou/llama.cpp commit 4e3db1e3 that corrects the std::chrono usage in log.cpp).

Prevention: Use the commit of bitnet.cpp documented here, and avoid manually advancing the 3rdparty/llama.cpp submodule past a tested point.

Error: model produces garbage / random characters on x86

Cause: On x86 CPUs without AVX2, the I2_S quantization path can produce incorrect output because there is no scalar fallback for the missing instruction set (reported as community issue #547; see also #348).

Solution: Run on a CPU with AVX2 support, or use a different quantization type that is supported for your architecture (tl2 on x86).

Prevention: Confirm your CPU advertises AVX2 (cat /proc/cpuinfo | grep avx2 on Linux) before relying on the I2_S kernel.

Error: Cmake is not available. Please install CMake and try again.

Cause: setup_env.py could not find a cmake executable on your PATH.

Solution: Install CMake ≥ 3.22 and ensure it is on your PATH. On Windows, installing the "C++ CMake Tools for Windows" Visual Studio component provides it.

Prevention: Verify with cmake --version before running setup_env.py.

Error: Arch <name> is not supported yet

Cause: bitnet.cpp recognizes x86_64 and arm64 architectures only; the build step bails out on anything else.

Solution: Run on a supported architecture. The recognized aliases are AMD64/x86/x86_64 → x86-64 and aarch64/arm64/ARM64 → ARM.

Error: Pretuned kernels not found for model <name>

Cause: You passed --use-pretuned but preset_kernels/ has no pre-tuned kernel for that model.

Solution: Drop --use-pretuned so the kernels are generated for your model, or choose a model that has pre-tuned kernels shipped (bitnet_b1_58-large, bitnet_b1_58-3B, Llama3-8B-1.58-100B-tokens).

Error: llama-cli.exe not produced on Windows

Cause: A historical issue (#461) where LLAMA_BUILD_EXAMPLES defaulted off, so the CLI executable wasn't built.

Solution: Rebuild after a clean checkout at the documented commit; verify build/bin/Release/llama-cli.exe exists after setup_env.py finishes.

Error: server crashes after the first response (Falcon3-10B)

Cause: A reported instability (#532) generating with Falcon3-10B-Instruct-1.58bit in server mode.

Solution: If you hit this, try a smaller Falcon3 variant or the BitNet-b1.58-2B-4T model, and report your configuration upstream.

10. Frequently Asked Questions

Q: What does "1-bit" or "1.58-bit" actually mean? The model's weights are ternary — each is one of −1, 0, or +1. Three such values carry log₂(3) ≈ 1.58 bits of information, hence "1.58-bit." Storing weights this way is what makes the model small and fast.

Q: Do I need a GPU? No. The primary framework runs entirely on the CPU and is its main strength. A GPU path exists separately for NVIDIA Ampere cards but is experimental and tuned for one model.

Q: Which quantization type should I pick? Use the default i2_s unless you have a reason not to — it works on both x86 and ARM. On ARM you may also use tl1; on x86 you may also use tl2. The framework rejects a type that doesn't match your CPU.

Q: Why does the build take so long? Because setup_env.py compiles the bundled llama.cpp engine and the BitNet kernels from C++ source. This happens once per machine/model; afterward, running inference is instant.

Q: Can I run a model I downloaded myself? Yes. Point setup_env.py at the directory with -md, or convert the weights with the utils/convert-* scripts and then run run_inference.py on the resulting .gguf file.

Q: How do I make generation faster? Increase --threads toward your physical core count, enable embedding quantization with --quant-embd, and (advanced) tune the kernel block sizes with utils/tune_gemm_config.py.

Q: How big a model can I run? bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU at 5–7 tokens per second. Your practical ceiling is system RAM — the model must fit in memory.

Q: Is the output identical to the full-precision model? The framework is designed for fast and lossless inference of the ternary model — it faithfully runs the 1.58-bit weights. It does not magically recover precision that was removed during the model's training/quantization; accuracy is that of the underlying 1-bit model.

Q: The server won't accept -cnv. Why? Chat mode is a CLI-only feature. run_inference_server.py intentionally omits it because llama-server doesn't support it. Use the HTTP API for chat-style interaction.

Q: Where do I see what went wrong during a build? In your log directory (default logs/). Each build step writes its own log file named after the step (compile.log, quantize_to_i2s.log, etc.).

Q: Does bitnet.cpp support NPUs? Not yet. The project notes NPU support is planned for a future release. Current targets are CPU (primary) and NVIDIA GPU (experimental).

11. Reference

11.1 Command summary

11.2 Default file locations

11.3 Further reading

• Technical report: 1-bit AI Infra: Fast and Lossless BitNet b1.58 Inference on CPUs
• Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
• The Era of 1-bit LLMs