Hugging Face Transformers User Guide

Version: 5.6.0.dev0 | License: Apache 2.0 | Python: 3.10–3.14 | Source: huggingface/transformers @ 71560f4 (2026-04-17)

1. Quick Start

Get a model running in under two minutes with the pipeline() factory — the fastest path to working inference.

Tip

pipeline() handles tokenization, model loading, and post-processing automatically. Start here before diving into lower-level APIs.

Install and run your first pipeline

pip install transformers torch

from transformers import pipeline

# Text generation
generator = pipeline("text-generation", model="gpt2")
result = generator("The future of AI is", max_new_tokens=50)
print(result[0]["generated_text"])
# The future of AI is bright, with breakthroughs in language understanding...

# Sentiment analysis
classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
print(classifier("I love using Transformers!"))
# [{'label': 'POSITIVE', 'score': 0.9998}]

Run a chat model in the terminal

pip install transformers[serving]
transformers serve --model meta-llama/Llama-3.2-1B-Instruct

Then in a second terminal:

transformers chat

You are now in an interactive Rich-based TUI connected to the local serve endpoint.

2. Installation

System requirements

Important

PyTorch is not installed by pip install transformers alone. Use pip install transformers[torch] or install your framework separately. Without a deep-learning backend (PyTorch, TensorFlow, Flax, or JAX), model inference will not work.

Base install

pip install transformers

Installs core library with hard dependencies: huggingface-hub>=1.5.0,<2.0, numpy>=1.17, packaging>=20.0, pyyaml>=5.1, regex>=2025.10.22, tokenizers>=0.22.0,<=0.23.0, typer, safetensors>=0.4.3, tqdm>=4.27.

Install extras

Install only what you need. Each extra pulls in the relevant framework and supporting libraries.

Warning

On Python 3.13+, kenlm is excluded from the [audio] extra automatically. If you need kenlm for language-model-based ASR decoding, use Python 3.12 or earlier, or install kenlm from source.

Install from source

pip install git+https://github.com/huggingface/transformers.git

For editable (development) install:

git clone https://github.com/huggingface/transformers
cd transformers
pip install -e ".[dev]"

Verify installation

transformers env

Expected output (abbreviated):

- `transformers` version: 5.6.0.dev0
- Platform: Linux-5.15.0 x86_64
- Python version: 3.11.9
- PyTorch version (GPU?): 2.5.0 (True)
- `accelerate` version: 0.33.0
- `tokenizers` version: 0.22.0

3. Configuration

Environment variables

These variables tune library behavior without touching code. Set them in your shell, .env file, or process environment before importing Transformers.

# Example: verbose logging + suppress advisory warnings
export TRANSFORMERS_VERBOSITY=info
export TRANSFORMERS_NO_ADVISORY_WARNINGS=1
python my_script.py

Logging in code

import transformers

transformers.logging.set_verbosity_info()
transformers.logging.enable_explicit_format()

Model cache

By default models download to ~/.cache/huggingface/hub. Override with:

export HF_HOME=/data/hf_cache

Or per-session:

from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-uncased", cache_dir="/data/my_models")

Download a model explicitly

transformers download bert-base-uncased
transformers download meta-llama/Llama-3.2-1B-Instruct --repo-type model

4. Core Usage

4.1 Auto classes

The Auto* family resolves the correct model/tokenizer class from a checkpoint name or local path. This is the recommended pattern for virtually all use cases.

from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Encoder model (embeddings, classification backbone)
model = AutoModel.from_pretrained("bert-base-uncased")

# Causal LM (text generation)
lm = AutoModelForCausalLM.from_pretrained("gpt2")

Common Auto* classes:

4.2 Tokenizers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Single sentence
encoding = tokenizer("Hello, world!", return_tensors="pt")
# {'input_ids': tensor([[101, 7592, 1010, 2088, 999, 102]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])}

# Batch with padding and truncation
batch = tokenizer(
    ["Short sentence.", "A much longer sentence that needs truncation or padding."],
    padding=True,
    truncation=True,
    max_length=32,
    return_tensors="pt",
)

# Decode back to text
tokenizer.decode(encoding["input_ids"][0])
# '[CLS] hello, world! [SEP]'

4.3 Running inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tokenizer("Once upon a time", return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Once upon a time there was a young princess who lived in a...

4.4 Text generation and GenerationConfig

Control every aspect of generation through GenerationConfig or inline kwargs.

from transformers import GenerationConfig

gen_config = GenerationConfig(
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1,
)

outputs = model.generate(**inputs, generation_config=gen_config)

Common generation parameters:

4.5 Streaming output

Use TextStreamer for real-time printing or TextIteratorStreamer to consume tokens in a loop.

from transformers import TextStreamer, AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

streamer = TextStreamer(tokenizer, skip_prompt=True)
inputs = tokenizer("The quick brown fox", return_tensors="pt")
model.generate(**inputs, max_new_tokens=100, streamer=streamer)
# tokens print to stdout as they are generated

For async / UI use:

from transformers import TextIteratorStreamer
import threading

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True)

def generate():
    model.generate(**inputs, max_new_tokens=100, streamer=streamer)

thread = threading.Thread(target=generate)
thread.start()

for token_text in streamer:
    print(token_text, end="", flush=True)

thread.join()

AsyncTextIteratorStreamer is available for async for loops in async frameworks.

4.6 Loading models efficiently

Warning

Never pass trust_remote_code=True to from_pretrained() for models from unverified publishers. This parameter executes arbitrary Python code downloaded from the Hub inside your process. Only use it for well-known model families from verified organizations when no native Transformers implementation exists.

# Load in half precision (saves ~50% VRAM)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", torch_dtype=torch.float16)

# Automatic device placement (requires accelerate)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", device_map="auto")

# Load to a specific device
model = AutoModelForCausalLM.from_pretrained("gpt2").to("cuda")

4.7 Saving and loading locally

# Save
model.save_pretrained("./my_model")
tokenizer.save_pretrained("./my_model")

# Load back
model = AutoModelForCausalLM.from_pretrained("./my_model")
tokenizer = AutoTokenizer.from_pretrained("./my_model")

5. Common Tasks

5.1 Text generation

from transformers import pipeline

generator = pipeline("text-generation", model="gpt2", device=0)  # device=0 for first GPU

outputs = generator(
    "In the beginning",
    max_new_tokens=100,
    num_return_sequences=3,
    do_sample=True,
    temperature=0.8,
)
for i, out in enumerate(outputs):
    print(f"--- Sequence {i+1} ---")
    print(out["generated_text"])

5.2 Text classification / sentiment analysis

sentiment-analysis is an alias for text-classification.

from transformers import pipeline

classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")

results = classifier([
    "This movie was fantastic!",
    "I would not recommend this product.",
    "The service was average.",
])

for r in results:
    print(r["label"], f"{r['score']:.3f}")
# POSITIVE 0.999
# NEGATIVE 0.997
# NEGATIVE 0.523

5.3 Token classification (NER)

from transformers import pipeline

ner = pipeline("token-classification", model="dbmdz/bert-large-cased-finetuned-conll03-english", aggregation_strategy="simple")

entities = ner("Hugging Face was founded in New York by Clément Delangue.")
for e in entities:
    print(f"{e['word']:20} {e['entity_group']:10} {e['score']:.3f}")
# Hugging Face         ORG        0.998
# New York             LOC        0.999
# Clément Delangue     PER        0.995

ner is an alias for token-classification.

5.4 Fill-mask

from transformers import pipeline

unmasker = pipeline("fill-mask", model="bert-base-uncased")
results = unmasker("The capital of France is [MASK].")
for r in results[:3]:
    print(f"{r['token_str']:15} score={r['score']:.4f}")
# paris           score=0.9978
# lyon            score=0.0005
# france          score=0.0003

5.5 Automatic speech recognition

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")

result = asr("path/to/audio.wav")
print(result["text"])
# "Hello, this is a test of speech recognition."

# With timestamps
result = asr("path/to/audio.wav", return_timestamps=True)
print(result["chunks"])
# [{'timestamp': (0.0, 2.5), 'text': 'Hello, this is a test'}, ...]

5.6 Audio classification

from transformers import pipeline

classifier = pipeline("audio-classification", model="superb/wav2vec2-base-superb-ks")
result = classifier("path/to/audio.wav")
print(result)
# [{'score': 0.997, 'label': 'yes'}, {'score': 0.001, 'label': 'no'}, ...]

5.7 Image classification

from transformers import pipeline
from PIL import Image

classifier = pipeline("image-classification", model="google/vit-base-patch16-224")

result = classifier("path/to/image.jpg")
for r in result[:3]:
    print(f"{r['label']:40} {r['score']:.4f}")
# tabby, tabby cat                         0.5312
# tiger cat                                0.2297
# Egyptian cat                             0.1090

5.8 Object detection

from transformers import pipeline

detector = pipeline("object-detection", model="facebook/detr-resnet-50")
results = detector("path/to/image.jpg")
for obj in results:
    print(obj["label"], f"score={obj['score']:.3f}", obj["box"])
# cat score=0.998 {'xmin': 42, 'ymin': 71, 'xmax': 315, 'ymax': 474}

5.9 Image segmentation

from transformers import pipeline

segmenter = pipeline("image-segmentation", model="facebook/detr-resnet-50-panoptic")
segments = segmenter("path/to/image.jpg")
for seg in segments:
    print(seg["label"], seg["score"])

5.10 Zero-shot classification

Classify text into arbitrary categories without task-specific training.

from transformers import pipeline

classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

result = classifier(
    "The new iPhone features a 48MP camera and improved battery life.",
    candidate_labels=["technology", "sports", "politics", "entertainment"],
)
print(result["labels"][0], result["scores"][0])
# technology 0.9923

5.11 Zero-shot image classification

from transformers import pipeline

classifier = pipeline("zero-shot-image-classification", model="openai/clip-vit-large-patch14")

result = classifier(
    "path/to/image.jpg",
    candidate_labels=["a dog", "a cat", "a car", "a mountain"],
)
print(result[0])
# {'score': 0.941, 'label': 'a cat'}

5.12 Depth estimation

from transformers import pipeline

depth = pipeline("depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")
result = depth("path/to/image.jpg")
# result["depth"] — PIL Image of the depth map
# result["predicted_depth"] — raw tensor

5.13 Image-text to text (multimodal)

from transformers import pipeline

pipe = pipeline("image-text-to-text", model="llava-hf/llava-1.5-7b-hf")

result = pipe(
    images="path/to/image.jpg",
    text="<image>\nDescribe what you see in this image.",
    max_new_tokens=200,
)
print(result[0]["generated_text"])

5.14 Document question answering

from transformers import pipeline

dqa = pipeline("document-question-answering", model="impira/layoutlm-document-qa")

result = dqa(
    image="path/to/invoice.jpg",
    question="What is the total amount?",
)
print(result[0]["answer"])
# $1,234.56

5.15 Video classification

from transformers import pipeline

classifier = pipeline("video-classification", model="MCG-NJU/videomae-base-finetuned-kinetics")
result = classifier("path/to/video.mp4", top_k=3)
for r in result:
    print(r["label"], f"{r['score']:.3f}")

5.16 Feature extraction

Get raw embeddings from any encoder model.

from transformers import pipeline

extractor = pipeline("feature-extraction", model="bert-base-uncased")
features = extractor("Hello, world!", return_tensors=True)
print(features.shape)  # [1, sequence_length, hidden_size]

5.17 Mask generation (SAM-style)

from transformers import pipeline

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
masks = generator("path/to/image.jpg", points_per_batch=64)
# masks["masks"] — list of boolean numpy arrays

5.18 Text to audio

from transformers import pipeline

synth = pipeline("text-to-audio", model="facebook/musicgen-small")
output = synth("A cheerful acoustic guitar melody")
# output["audio"] — numpy array
# output["sampling_rate"] — int

import scipy.io.wavfile as wavfile
wavfile.write("output.wav", output["sampling_rate"], output["audio"])

5.19 Quantization for smaller memory footprint

Load quantized models to run large models on consumer hardware.

Tip

BitsAndBytesConfig with load_in_4bit=True is the most common starting point for fitting large models into limited VRAM. Requires bitsandbytes package and CUDA.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

Supported quantization backends:

22+ quantization configs available — see the Admin Guide for the full list and configuration reference.

5.20 Training with Trainer

from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)
from datasets import load_dataset

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

dataset = load_dataset("imdb")

def tokenize(examples):
    return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=512)

tokenized = dataset.map(tokenize, batched=True)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    warmup_steps=500,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
)

trainer.train()
trainer.save_model("./my_fine_tuned_model")

5.21 Learning rate schedulers

Pass lr_scheduler_type in TrainingArguments:

training_args = TrainingArguments(
    ...,
    lr_scheduler_type="cosine",
    warmup_ratio=0.06,
    learning_rate=2e-5,
)

5.22 Data collators

from transformers import DataCollatorWithPadding

collator = DataCollatorWithPadding(tokenizer=tokenizer)
# Pass as collate_fn to DataLoader or as data_collator to Trainer

5.23 Serving: OpenAI-compatible API

Start a local inference server:

pip install transformers[serving]
transformers serve --model gpt2 --port 8000

Note

The built-in server has no authentication. For any deployment beyond localhost, place it behind a reverse proxy (nginx, Traefik, Caddy) with authentication middleware. The /load_model and /reset endpoints can load arbitrary models — restrict access in shared environments.

Available endpoints:

Use with any OpenAI-compatible client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="gpt2",
    messages=[{"role": "user", "content": "What is machine learning?"}],
    max_tokens=200,
)
print(response.choices[0].message.content)

5.24 Chat CLI

Start an interactive chat session:

# Connect to local serve endpoint (default)
transformers chat

# Connect to HF Inference API
transformers chat --model meta-llama/Llama-3.2-1B-Instruct

In-session commands:

5.25 Converting checkpoints

Convert PyTorch weights to TensorFlow:

transformers pt-to-tf --model bert-base-uncased --output-dir ./tf_model

Enable Git LFS for large model files:

transformers lfs-enable-largefiles .

Upload large files via multipart:

transformers lfs-multipart-upload ./large_file.bin

6. Troubleshooting

Error: CUDA out of memory

Cause: The model and its activations exceed available GPU VRAM.

Solution:

1. Load in lower precision: torch_dtype=torch.float16 or torch_dtype=torch.bfloat16
2. Use 4-bit quantization: BitsAndBytesConfig(load_in_4bit=True)
3. Enable device mapping: device_map="auto" (requires accelerate)
4. Reduce batch size or max_new_tokens
5. Use torch.backends.cuda.enable_flash_sdp(True) for attention memory savings

model = AutoModelForCausalLM.from_pretrained(
    "large-model-name",
    torch_dtype=torch.float16,
    device_map="auto",
)

Prevention: Check model size against VRAM before loading. A 7B model in fp16 needs ~14 GB VRAM.

Error: No module named 'transformers.onnx'

Cause: The transformers.onnx submodule was removed in a previous version.

Solution: Use optimum for ONNX export instead:

pip install optimum[exporters]
optimum-cli export onnx --model bert-base-uncased ./bert_onnx/

Prevention: Do not reference transformers.onnx in new code; it no longer exists in the library.

Error: from_pretrained fails with Qwen3.5 MoE on transformers >= 5.4.0

Cause: Breaking change in MoE model loading in 5.4.0 (tracked in issue #45478).

Solution:

1. Pin an older version: pip install "transformers==5.3.x"
2. Or check the issue tracker for a patched version / workaround commit

Prevention: Pin exact versions for production deployments of MoE models until the fix is confirmed stable.

Error: cross-device tensor error with CPU offload on Gemma 26B

Cause: Known issue with Gemma4 26B-A4B models when cpu_offload is enabled (tracked in issue #45482).

Solution:

• Disable CPU offload: remove cpu_offload_gbytes or set device_map without offload
• Use a GPU with sufficient VRAM for the full model
• Monitor the upstream issue #45482 for the official fix

Prevention: Test CPU offload configurations on smaller Gemma variants before scaling to 26B.

Error: Gemma4 31B-IT Multi-GPU CUDA OOM

Cause: Multi-GPU tensor placement with large Gemma4 models can produce OOM even when total VRAM is sufficient (tracked in issue #45229).

Solution:

• Explicitly set a max_memory dict to control per-GPU allocation:

  model = AutoModelForCausalLM.from_pretrained(
      "google/gemma-4-31b-it",
      device_map="auto",
      max_memory={0: "20GiB", 1: "20GiB"},
  )
  

• Use torch_dtype=torch.bfloat16 to halve memory
• Apply 4-bit quantization via BitsAndBytesConfig

Prevention: Always specify max_memory when distributing across 2+ GPUs.

Error: problem_type="single_label_classification" produces zero loss with num_labels=1

Cause: Known bug — setting num_labels=1 with single_label_classification leads to degenerate zero loss (tracked in issue #45479).

Solution: Use num_labels=2 for binary classification, or use problem_type="regression" with num_labels=1 for regression tasks.

Prevention: Do not combine num_labels=1 with single_label_classification; binary classification always requires num_labels=2.

Error: resize_token_embeddings doesn't update decoder on t5gemma

Cause: Known issue — resize_token_embeddings does not propagate to decoder.embed_tokens in t5gemma models (tracked in issue #45335).

Solution: Manually resize both encoder and decoder embeddings:

model.resize_token_embeddings(new_vocab_size)
model.decoder.embed_tokens = model.shared  # re-tie if needed

Prevention: After adding tokens to a t5gemma model, verify embedding sizes on both encoder and decoder sides.

Error: tokenizers version mismatch

Cause: tokenizers must be >=0.22.0,<=0.23.0. Using a newer or older version may cause silent tokenization bugs.

Solution:

pip install "tokenizers>=0.22.0,<=0.23.0"

Prevention: Pin tokenizers in requirements.txt; do not let it upgrade freely.

Warning: Some weights were not initialized

Cause: Normal when loading a base model and adding a task-specific head. The head weights are randomly initialized and require fine-tuning.

Solution: This is expected. Train the model before using it for inference.

Prevention: For inference only, always load a fine-tuned checkpoint, not a base model.

Error: Pipeline device mismatch (tensors on different devices)

Cause: Inputs are on CPU but model is on GPU, or vice versa.

Solution:

# Specify device explicitly when creating pipeline
pipe = pipeline("text-classification", model="...", device=0)  # GPU 0

# Or use device_map for large models
pipe = pipeline("text-generation", model="...", device_map="auto")

Prevention: Always set device or device_map when constructing pipelines on GPU systems.

Error: huggingface-hub version outside bounds

Cause: huggingface-hub requires >=1.5.0,<2.0. Hub 2.0+ has breaking changes.

Solution:

pip install "huggingface-hub>=1.5.0,<2.0"

Prevention: Pin in requirements files; set upper bounds to prevent silent breaking upgrades.

Slow generation / high latency

Cause: Model on CPU, or using eager attention instead of flash attention.

Solution:

1. Move model to GPU: .to("cuda") or device_map="auto"
2. Enable Flash Attention 2 (requires flash-attn package):

   model = AutoModelForCausalLM.from_pretrained("...", attn_implementation="flash_attention_2")
   

3. Use torch.compile() for repeated inference
4. Reduce max_new_tokens if low latency is the priority

7. FAQ

Q: What is the difference between pipeline() and using AutoModel directly?

pipeline() is a high-level convenience wrapper that bundles a model, tokenizer, and post-processing logic into a single call. It is ideal for quick experiments and production inference when you do not need to inspect intermediate representations. AutoModel gives you direct access to model outputs (logits, hidden states, attention weights) and is necessary for custom training loops, feature extraction research, or any use case where you need control over pre/post-processing.

Q: Do I need a GPU to use Transformers?

No. All models run on CPU. However, large models (7B+ parameters) will be extremely slow on CPU — expect minutes per token for generation. For practical use of large models, a GPU with sufficient VRAM is strongly recommended. For CPU-only deployment, consider smaller distilled models (DistilBERT, DistilGPT-2, TinyLlama) or quantized models with a CPU-compatible quantization backend.

Q: How do I pick the right model for my task?

Use the Hugging Face Hub model page (huggingface.co/models) and filter by task. For most NLP tasks, start with a model that is already fine-tuned for that task rather than a base model. Example: for sentiment analysis use a model fine-tuned on SST-2 or financial sentiment datasets, not bert-base-uncased.

Q: What is the difference between AutoModelForCausalLM and AutoModelForSeq2SeqLM?

Causal LM (AutoModelForCausalLM) is a decoder-only architecture (GPT-2, LLaMA, Gemma, Mistral) that generates text auto-regressively — each token depends on all previous tokens. Seq2Seq LM (AutoModelForSeq2SeqLM) is an encoder-decoder architecture (T5, BART, mBART) where an encoder processes the full input before a decoder generates the output. Use causal LM for open-ended generation; use seq2seq for translation, summarization, or structured output tasks.

Q: How do I run a model that requires authentication (gated models)?

Log in with the Hugging Face CLI before loading:

pip install huggingface_hub
huggingface-cli login

Then from_pretrained will use your stored token automatically. You can also pass token="hf_..." directly to from_pretrained.

Q: Can I run Transformers without internet access (air-gapped)?

Yes. First download the model with transformers download or snapshot_download on a connected machine. Transfer the cached files to the target machine. Then load from the local path:

model = AutoModelForCausalLM.from_pretrained("/path/to/local/model")

Set HF_DATASETS_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 to prevent any outbound requests.

Q: What is the difference between PreTrainedTokenizer and PreTrainedTokenizerFast?

PreTrainedTokenizerFast is backed by the Rust-based tokenizers library and is significantly faster — especially for batch tokenization of large datasets. PreTrainedTokenizer is a pure-Python fallback. Most modern models ship a fast tokenizer. AutoTokenizer selects the fast version automatically when available. Prefer fast tokenizers for any training or large-scale processing pipeline.

Q: How do I use Trainer with multiple GPUs?

Use accelerate launch instead of python:

pip install accelerate
accelerate config  # one-time setup
accelerate launch train.py

Or set CUDA_VISIBLE_DEVICES and let Trainer detect available GPUs automatically when device_map is not set. For distributed training across nodes, configure accelerate with DeepSpeed or FSDP.

Q: Can I serve models with an OpenAI-compatible API?

Yes. transformers serve starts a FastAPI server with /v1/chat/completions, /v1/responses, /v1/audio/transcriptions, /v1/models, /health, /load_model, and /reset endpoints. Install with pip install transformers[serving] and run transformers serve --model <model-id>. Any OpenAI-compatible client (Python openai SDK, curl, etc.) can connect to http://localhost:8000/v1.

Q: How do I add a new model to the library?

Use the scaffolding command:

transformers add-new-model-like --model-name MyModel --model-type existing_model_type

This generates boilerplate files following library conventions. See the Transformers contribution guide for full instructions on adding architectures.

Q: What quantization method should I use for my use case?

Q: How do I print my environment to report a bug?

transformers env

Include this output in every GitHub issue report. It captures the exact versions of Transformers, PyTorch, CUDA, tokenizers, and accelerate.

Q: What Python versions are supported?

Python 3.10 through 3.14. Python 3.9 and earlier are not supported as of v5.6.0.dev0. Note that the kenlm dependency in transformers[audio] is excluded on Python 3.13+.

Q: How do I disable the tqdm progress bars?

from transformers.utils import logging
logging.disable_progress_bar()

Or set TRANSFORMERS_NO_ADVISORY_WARNINGS=1 to suppress advisory-level output. For no output at all, set TRANSFORMERS_VERBOSITY=error.

Q: Is there a way to cache model downloads to a custom location?

Yes. Set HF_HOME environment variable before running:

export HF_HOME=/data/hf_cache

Or pass cache_dir to any from_pretrained call:

model = AutoModel.from_pretrained("bert-base-uncased", cache_dir="/data/hf_cache")