# Stable Diffusion — User Guide

## 1. Quick Start

Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images from text prompts. It operates in the latent space of a pretrained autoencoder, making high-resolution image synthesis computationally efficient compared to pixel-space diffusion models.

**Key features:**

- Text-to-image generation from natural language prompts
- Image-to-image translation (style transfer, guided editing)
- Image inpainting with mask-based conditioning
- Unconditional image generation (faces, bedrooms, churches)
- Class-conditional ImageNet synthesis
- Super-resolution via latent diffusion
- Training and fine-tuning with PyTorch Lightning
- Multiple sampling methods: DDIM, PLMS, DPM-Solver

**Prerequisites:** Python 3.8+, NVIDIA GPU with CUDA 11.3+, ~10 GB VRAM for inference, ~24 GB VRAM for training

**Minimal install:**

```bash
conda env create -f environment.yaml
conda activate ldm
```

**Generate your first image:**

```bash
python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" \
    --plms --outdir outputs/txt2img-samples
```

> [!TIP]
> Use `--plms` for faster, higher-quality sampling compared to the default DDIM sampler. For the quickest results with acceptable quality, try `--ddim_steps 50`.

**What to try next:**

- Image-to-image with `scripts/img2img.py`
- Inpainting with `scripts/inpaint.py`
- Training your own model with `main.py`

## 2. Installation

### System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Python | 3.8.5 | 3.8.x |
| CUDA Toolkit | 11.3 | 11.3+ |
| GPU VRAM | 8 GB (inference) | 24+ GB (training) |
| RAM | 16 GB | 32 GB |
| Disk | 10 GB (code + model) | 50 GB (with datasets) |

### Installation via Conda (Recommended)

```bash
git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion
conda env create -f environment.yaml
conda activate ldm
```

> [!IMPORTANT]
> The `environment.yaml` installs two editable packages from Git: `taming-transformers` (from CompVis) and `CLIP` (from OpenAI). Both are cloned and installed in development mode (`-e`) during environment creation. These are required dependencies — the build will fail without network access to GitHub.

### Core Dependencies

| Package | Version | Purpose |
|---|---|---|
| `pytorch` | 1.11.0 | Deep learning framework |
| `torchvision` | 0.12.0 | Image transforms and utilities |
| `pytorch-lightning` | 1.4.2 | Training framework |
| `transformers` | 4.19.2 | CLIP text encoder |
| `omegaconf` | 2.1.1 | YAML configuration management |
| `einops` | 0.3.0 | Tensor operations |
| `kornia` | 0.6 | Computer vision operations |
| `albumentations` | 0.4.3 | Image augmentation |
| `opencv-python` | 4.1.2.30 | Image I/O and processing |
| `invisible-watermark` | latest | Output watermarking |
| `taming-transformers` | Git main | VQ-VAE codebook (editable install) |
| `clip` | Git main | CLIP text/image encoder (editable install) |

### Downloading Model Weights

Pre-trained model checkpoints are not included in the repository. Download the Stable Diffusion v1 weights and place them in the project directory:

```bash
# Example: place the checkpoint at the project root
# The inference scripts expect a --ckpt flag pointing to the .ckpt file
ls -lh sd-v1-4.ckpt
```

```text
-rw-r--r-- 1 user user 4.0G Oct  1 12:00 sd-v1-4.ckpt
```

> [!WARNING]
> Model checkpoints are typically 2-7 GB each. Ensure sufficient disk space and a stable network connection before downloading. The model card is available at `Stable_Diffusion_v1_Model_Card.md` in the repository root.

### Verify Installation

```bash
conda activate ldm
python -c "import ldm; print('LDM package loaded successfully')"
```

```text
LDM package loaded successfully
```

### Installing the Package in Development Mode

The repository includes a `setup.py` that registers the `latent-diffusion` package (version `0.0.1`):

```bash
pip install -e .
```

This makes the `ldm` package importable from anywhere in the conda environment. The minimal install requires only `torch`, `numpy`, and `tqdm`.

## 3. Configuration

Stable Diffusion uses YAML configuration files managed by OmegaConf. Configs define the model architecture, training parameters, data pipeline, and logging.

### Configuration Structure

Every config file follows this top-level structure:

```yaml
model:
  base_learning_rate: 1.0e-04
  target: ldm.models.diffusion.ddpm.LatentDiffusion
  params:
    # Diffusion parameters
    # UNet config
    # First-stage (autoencoder) config
    # Conditioning-stage config

data:
  target: main.DataModuleFromConfig
  params:
    batch_size: 4
    # Dataset configs

lightning:
  trainer:
    # PyTorch Lightning Trainer args
  callbacks:
    # Callback configs
```

### Pre-built Configurations

| Config File | Task | Conditioning |
|---|---|---|
| `configs/stable-diffusion/v1-inference.yaml` | Text-to-image (SD v1) | CLIP text (`crossattn`) |
| `configs/latent-diffusion/txt2img-1p4B-eval.yaml` | Text-to-image (1.4B LDM) | BERT text (`crossattn`) |
| `configs/latent-diffusion/cin-ldm-vq-f8.yaml` | Class-conditional ImageNet | Class label (`crossattn`) |
| `configs/latent-diffusion/cin256-v2.yaml` | Class-conditional ImageNet v2 | Class label (`crossattn`) |
| `configs/latent-diffusion/celebahq-ldm-vq-4.yaml` | Unconditional faces (CelebA-HQ) | None |
| `configs/latent-diffusion/ffhq-ldm-vq-4.yaml` | Unconditional faces (FFHQ) | None |
| `configs/latent-diffusion/lsun_bedrooms-ldm-vq-4.yaml` | Unconditional bedrooms | None |
| `configs/latent-diffusion/lsun_churches-ldm-kl-8.yaml` | Unconditional churches | None |
| `configs/retrieval-augmented-diffusion/768x768.yaml` | Retrieval-augmented diffusion | Retrieval |

### Key Model Parameters (Stable Diffusion v1)

From `configs/stable-diffusion/v1-inference.yaml`:

| Parameter | Value | Description |
|---|---|---|
| `linear_start` | `0.00085` | Noise schedule start |
| `linear_end` | `0.0120` | Noise schedule end |
| `timesteps` | `1000` | Total diffusion timesteps |
| `image_size` | `64` | Latent spatial resolution |
| `channels` | `4` | Latent channels |
| `scale_factor` | `0.18215` | Latent space scaling factor |
| `conditioning_key` | `crossattn` | Cross-attention conditioning |
| `cond_stage_trainable` | `false` | Frozen CLIP encoder |

> [!NOTE]
> The `image_size` parameter refers to the latent resolution, not the output pixel resolution. With the default `f8` autoencoder (8x downsampling), a latent size of 64 corresponds to 512x512 pixel output.

### Overriding Config Values from CLI

Config parameters can be overridden from the command line using dot notation:

```bash
python main.py -b configs/stable-diffusion/v1-inference.yaml \
    model.base_learning_rate=2.0e-05 \
    data.params.batch_size=2
```

See the Admin Guide for the complete configuration reference.

## 4. Core Usage

### Text-to-Image Generation

Generate images from text prompts using `scripts/txt2img.py`:

```bash
python scripts/txt2img.py \
    --prompt "a painting of a sunset over mountains, oil on canvas" \
    --outdir outputs/txt2img-samples \
    --plms \
    --ckpt sd-v1-4.ckpt
```

Key options for text-to-image generation include prompt text, output directory, sampler selection (PLMS or DDIM), step count, guidance scale, image dimensions, batch size, and random seed for reproducibility.

> [!TIP]
> Higher guidance scale values (e.g., `7.5`-`15.0`) produce images that more closely match the prompt but with less diversity. Lower values (e.g., `1.0`-`5.0`) produce more varied but less prompt-faithful results.

### Image-to-Image Translation

Transform existing images with text guidance using `scripts/img2img.py`:

```bash
python scripts/img2img.py \
    --prompt "A fantasy landscape, trending on artstation" \
    --init-img assets/stable-samples/img2img/sketch-mountains-input.jpg \
    --outdir outputs/img2img-samples \
    --ckpt sd-v1-4.ckpt
```

The strength parameter controls how much the output deviates from the input image — higher values produce more dramatic transformations while lower values preserve more of the original structure.

### Inpainting

Fill masked regions of an image using `scripts/inpaint.py`:

```bash
python scripts/inpaint.py \
    --indir data/inpainting_examples \
    --outdir outputs/inpainting-samples \
    --ckpt sd-v1-4.ckpt
```

The inpainting script expects pairs of files in the input directory: original images and corresponding mask images (with `_mask` suffix). White regions in the mask indicate areas to be inpainted. Example pairs are provided in `data/inpainting_examples/`.

> [!IMPORTANT]
> Inpainting masks must be the same dimensions as the source image. White pixels (255) mark regions to regenerate; black pixels (0) mark regions to preserve.

### Unconditional Sampling

Generate samples from unconditional models (faces, bedrooms, churches):

```bash
python scripts/sample_diffusion.py \
    -r models/ldm/lsun_churches256 \
    -n 10 \
    --batch_size 5
```

### Retrieval-Augmented Diffusion

Generate high-resolution (768x768) images using retrieval-augmented diffusion with the config at `configs/retrieval-augmented-diffusion/768x768.yaml`. This model conditions on retrieved nearest-neighbor images using `scripts/knn2img.py`.

### Training

Train or fine-tune models using `main.py` with a YAML config:

```bash
python main.py \
    --base configs/latent-diffusion/lsun_churches-ldm-kl-8.yaml \
    -t --gpus 0,1 \
    -n my_experiment
```

> [!WARNING]
> Training requires significant GPU resources. The default configuration uses DDP (Distributed Data Parallel) when GPUs are specified. Training on a single consumer GPU may require reducing batch size and enabling gradient accumulation.

### Training CLI Reference (main.py)

| Flag | Type | Default | Description |
|---|---|---|---|
| `-n` / `--name` | `str` | `""` | Postfix for the log directory name |
| `-r` / `--resume` | `str` | `""` | Resume from log directory or checkpoint path |
| `-b` / `--base` | `str[]` | `[]` | Paths to base config YAML files (merged left-to-right) |
| `-t` / `--train` | `bool` | `False` | Enable training mode |
| `--no-test` | `bool` | `False` | Disable test evaluation after training |
| `-p` / `--project` | `str` | — | Name or path to project |
| `-d` / `--debug` | `bool` | `False` | Enable post-mortem debugging |
| `-s` / `--seed` | `int` | `23` | Random seed for reproducibility |
| `-f` / `--postfix` | `str` | `""` | Additional postfix for the log directory name |
| `-l` / `--logdir` | `str` | `logs` | Root directory for training logs |
| `--scale_lr` | `bool` | `True` | Scale learning rate by GPU count, batch size, and gradient accumulation |

> [!NOTE]
> In addition to these flags, `main.py` accepts all PyTorch Lightning `Trainer` arguments (e.g., `--gpus`, `--max_epochs`, `--precision`). Unknown `--key value` arguments are parsed as CLI config parameters; `nested.key=value` arguments override config file values.

## 5. Common Tasks

### Generating Multiple Images with Different Seeds

```bash
for seed in 42 123 456 789; do
    python scripts/txt2img.py \
        --prompt "a cyberpunk cityscape at night" \
        --seed $seed \
        --outdir outputs/seed-comparison \
        --ckpt sd-v1-4.ckpt \
        --plms
done
```

### Resuming Training from a Checkpoint

```bash
python main.py \
    --resume logs/2024-01-15T12-30-00_my_experiment \
    -t --gpus 0,
```

The `--resume` flag accepts either a log directory path or a direct checkpoint file. When given a directory, it automatically loads `checkpoints/last.ckpt` and merges the saved config files from the `configs/` subdirectory.

> [!NOTE]
> You cannot combine `--name` (`-n`) and `--resume` (`-r`) simultaneously. To resume training with a new experiment name, use `--resume_from_checkpoint` instead.

### Running on CPU (No GPU)

Omit the `--gpus` flag when calling `main.py` — the trainer automatically falls back to CPU mode when no GPUs are configured:

```bash
python main.py --base configs/latent-diffusion/celebahq-ldm-vq-4.yaml -t
```

### Adjusting Learning Rate Scaling

By default, the effective learning rate is automatically scaled:

```text
effective_lr = accumulate_grad_batches * num_gpus * batch_size * base_learning_rate
```

Disable this with `--scale_lr False` to use the raw `base_learning_rate` from the config.

### Using Debug Mode

Enable post-mortem debugging for development:

```bash
python main.py --base configs/stable-diffusion/v1-inference.yaml -t -d
```

Debug mode enables `pudb` (or `pdb` as fallback) post-mortem debugging on exceptions and moves the log directory to `debug_runs/` after completion.

### Saving Checkpoints via Signal

During training, send `SIGUSR1` to trigger an immediate checkpoint save:

```bash
kill -USR1 <pid>
```

This saves `last.ckpt` in the checkpoint directory without interrupting training. Send `SIGUSR2` to drop into an interactive `pudb` debugger.

### Using the Jupyter Notebook for Super-Resolution

The repository includes `scripts/latent_imagenet_diffusion.ipynb` and `notebook_helpers.py` for interactive super-resolution experiments in Google Colab:

```python
from notebook_helpers import get_model, run

model = get_model("superresolution")
logs = run(model, selected_path, task="superresolution", custom_steps=100)
```

> [!NOTE]
> The notebook helper `download_models()` currently only supports `"superresolution"` mode. Other modes raise `NotImplementedError`.

### Training Autoencoders

Train first-stage autoencoder models (VQ-VAE or KL-VAE) using the configs in `configs/autoencoder/`:

```bash
python main.py \
    --base configs/autoencoder/autoencoder_kl_32x32x4.yaml \
    -t --gpus 0,
```

Available autoencoder configurations:

| Config | Latent Shape | Embed Dim |
|---|---|---|
| `autoencoder_kl_8x8x64.yaml` | 8x8 | 64 |
| `autoencoder_kl_16x16x16.yaml` | 16x16 | 16 |
| `autoencoder_kl_32x32x4.yaml` | 32x32 | 4 |
| `autoencoder_kl_64x64x3.yaml` | 64x64 | 3 |

## 6. Troubleshooting

### Error: `ModuleNotFoundError: No module named 'ldm.util'; 'ldm' is not a package`

**Cause:** A file or directory named `ldm` in the current working directory shadows the `ldm` package. This commonly happens when running scripts from a directory that contains an `ldm` file that isn't the package, or when the package isn't installed.

**Solution:**
1. Ensure you are in the repository root directory
2. Install the package in development mode:
   ```bash
   pip install -e .
   ```
3. Verify no conflicting `ldm` files exist in your working directory

**Prevention:** Always run scripts from the repository root and install the package via `pip install -e .`.

*Related: GitHub issues #818, #865*

### Error: `CUDA driver initialization failed, you might not have a CUDA gpu`

**Cause:** CUDA toolkit version mismatch, missing NVIDIA drivers, or no compatible GPU available.

**Solution:**
1. Verify GPU visibility:
   ```bash
   nvidia-smi
   ```
2. Check CUDA toolkit version matches PyTorch build:
   ```bash
   python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"
   ```
3. Reinstall PyTorch with the correct CUDA version:
   ```bash
   conda install pytorch=1.11.0 torchvision=0.12.0 cudatoolkit=11.3 -c pytorch
   ```

**Prevention:** Always create the environment using `conda env create -f environment.yaml` to ensure version consistency.

*Related: GitHub issue #94*

### Error: `ImportError: cannot import name 'VectorQuantizer2'`

**Cause:** The `taming-transformers` package is not installed or an incompatible version is present.

**Solution:**
```bash
pip install -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers
```

**Prevention:** Use `conda env create -f environment.yaml` for a clean setup — it includes the correct taming-transformers version as an editable install.

*Related: GitHub issue #72*

### Error: `AttributeError: module 'ldm.modules.encoders.modules' has no attribute 'FrozenCLIPEmbedder'`

**Cause:** The CLIP package or `transformers` library version is incompatible, or the `ldm` package import path is broken.

**Solution:**
1. Reinstall CLIP:
   ```bash
   pip install -e git+https://github.com/openai/CLIP.git@main#egg=clip
   ```
2. Verify `transformers` version:
   ```bash
   pip install transformers==4.19.2
   ```

**Prevention:** Do not upgrade `transformers` beyond 4.19.2 within this environment.

*Related: GitHub issue #190*

> [!CAUTION]
> Upgrading `transformers` to a newer version may break compatibility with the CLIP encoder. Pin the version at `4.19.2` as specified in `environment.yaml`.

### Error: `OSError: Error no file named model_index.json found in directory`

**Cause:** Attempting to load a checkpoint using the `diffusers` library format when the model is in the original CompVis `.ckpt` format, or vice versa.

**Solution:** Use the correct loading method for the checkpoint format:
- For `.ckpt` files: Use `--ckpt path/to/model.ckpt` with the provided scripts
- For `diffusers` format: Use the `diffusers` library directly

*Related: GitHub issue #394*

### Error: `KeyError: "filename 'storages' not found"`

**Cause:** Corrupted or incompletely downloaded checkpoint file.

**Solution:**
1. Delete the corrupted checkpoint
2. Re-download the complete checkpoint file
3. Verify file integrity (check file size matches expected values)

*Related: GitHub issue #395*

> [!WARNING]
> Partially downloaded checkpoint files will cause cryptic PyTorch deserialization errors. Always verify the file size after download — Stable Diffusion v1 checkpoints are approximately 4 GB.

### Error: `RuntimeError: Sizes of tensors must match except in dimension 1`

**Cause:** Input image dimensions are not divisible by 64, or a mismatch between the model config and checkpoint.

**Solution:**
1. Ensure input images have dimensions divisible by 64
2. Verify the config file matches the checkpoint being loaded
3. Check that `image_size` and `channels` in the config match the checkpoint's expected values

*Related: GitHub issue #841*

### Wrong GPU Selected / Out of Memory

**Cause:** PyTorch defaults to GPU 0. On multi-GPU systems, GPU 0 may already be in use.

**Solution:**
```bash
CUDA_VISIBLE_DEVICES=1 python scripts/txt2img.py --prompt "test" --ckpt sd-v1-4.ckpt
```

Or specify GPUs in the training command:
```bash
python main.py --base config.yaml -t --gpus 1,2
```

*Related: GitHub issue #349*

> [!TIP]
> Monitor GPU memory usage with `nvidia-smi -l 1` in a separate terminal during generation to identify memory bottlenecks.

### Error: `RuntimeError: One of the differentiated Tensors does not require grad`

**Cause:** Selectively freezing UNet blocks while training causes gradient computation errors when PyTorch Lightning's gradient checkpointing expects gradients from all parameters.

**Solution:** Ensure all parameters in the forward path either all require grad or none do. When fine-tuning specific layers, freeze via `param.requires_grad = False` and verify no checkpointed layers reference frozen params.

*Related: GitHub issue #784*

## 7. FAQ

**Q: What image resolutions does Stable Diffusion v1 support?**

Stable Diffusion v1 was trained on 512x512 images. The model can generate at other resolutions, but quality degrades significantly outside the 512x512 range. Input dimensions should be divisible by 64 due to the autoencoder's downsampling factor.

**Q: How much VRAM do I need?**

Inference with default settings requires approximately 8-10 GB VRAM. Training requires 24+ GB VRAM depending on batch size and model size. Reduce VRAM usage by lowering output dimensions or reducing batch size.

**Q: Can I run Stable Diffusion on macOS / Apple Silicon?**

The original CompVis implementation targets NVIDIA CUDA GPUs. Community forks and the `diffusers` library provide MPS (Metal) support for Apple Silicon. See GitHub issue #25 for community setup instructions.

**Q: What is the difference between DDIM and PLMS sampling?**

DDIM (Denoising Diffusion Implicit Models) is the default sampler. PLMS (Pseudo Linear Multi-Step) typically produces higher-quality results in fewer steps. Both are implemented in `ldm/models/diffusion/` as `ddim.py` and `plms.py`. A DPM-Solver implementation is also available at `ldm/models/diffusion/dpm_solver/`.

**Q: How do I use my own dataset for training?**

Create a custom dataset class inheriting from `torch.utils.data.Dataset` or `Txt2ImgIterableBaseDataset` (for text-image pairs). Reference it in your YAML config under `data.params.train.target` using the full Python import path. The `DataModuleFromConfig` class in `main.py` handles instantiation via OmegaConf's `target` resolution.

**Q: What are the `taming-transformers` and `CLIP` dependencies?**

`taming-transformers` provides the VQ-VAE codebook implementation (`VectorQuantizer2`) used by the VQ-based autoencoders. `CLIP` provides the frozen text encoder (`FrozenCLIPEmbedder`) that converts text prompts into conditioning vectors for text-to-image generation.

**Q: Can I fine-tune on my own images?**

Yes. Prepare a dataset class, write a YAML config pointing to it, and run `python main.py --base your_config.yaml -t --gpus 0,`. Use `--resume` to initialize from a pre-trained checkpoint.

**Q: What does `scale_factor: 0.18215` mean in the config?**

This scaling factor normalizes the latent space values output by the autoencoder before they enter the diffusion process. It was computed empirically during training to ensure the latent distribution has approximately unit variance.

**Q: What is the difference between VQ-based and KL-based autoencoders?**

VQ (Vector Quantized) autoencoders discretize the latent space using a codebook (e.g., `VQModel` with `n_embed=8192`). KL autoencoders (`AutoencoderKL`) use a continuous Gaussian latent space regularized with a KL divergence term. Stable Diffusion v1 uses the KL autoencoder (`kl-f8`, 4-channel, embed_dim=4).

**Q: What datasets are supported out of the box?**

The data loaders support ImageNet (train/validation via `ImageNetTrain`/`ImageNetValidation`), ImageNet super-resolution (`ImageNetSR`), LSUN scenes (churches, bedrooms, cats via `LSUNChurchesTrain`, etc.), CelebA-HQ and FFHQ faces (via `taming-transformers`), and custom text-image datasets via `Txt2ImgIterableBaseDataset`.

---

_Generated May 2026 from CompVis/stable-diffusion at 21f890f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._