# Annotated Deep Learning Paper Implementations — User Guide

## 1. Quick Start

**labml_nn** is a collection of simple, annotated PyTorch implementations of neural networks and related algorithms from landmark deep learning research papers. Each implementation is documented with inline explanations, and the companion website ([nn.labml.ai](https://nn.labml.ai)) renders the source code as side-by-side formatted notes — code on one side, explanation on the other.

**Key features:**

- 60+ paper implementations covering transformers, diffusion models, GANs, optimizers, normalization layers, reinforcement learning, and more
- Every implementation is a working PyTorch module you can import and use directly
- Side-by-side annotated documentation at [nn.labml.ai](https://nn.labml.ai)
- Experiment scripts included for training on standard datasets (MNIST, CIFAR-10, Tiny Shakespeare, Atari Breakout)
- Translations available in Chinese, Japanese, and Sinhala
- MIT licensed

**Prerequisites:** Python 3.7+, PyTorch 1.10+

**Install:**

```bash
pip install labml-nn
```

**Minimal example — use a multi-head attention layer:**

```python
from labml_nn.transformers import MultiHeadAttention
import torch

# 8-head attention with 512-dimensional model
mha = MultiHeadAttention(heads=8, d_model=512)
q = k = v = torch.randn(32, 10, 512)  # batch=32, seq=10, dim=512
output = mha(query=q, key=k, value=v)
print(output.shape)
```

```text
torch.Size([32, 10, 512])
```

> [!TIP]
> Browse the full implementation catalog at [nn.labml.ai](https://nn.labml.ai) to find the paper you want, then import the corresponding module from `labml_nn`.

**What to try next:**

- Browse the implementation catalog below to find specific papers
- Run one of the included experiment scripts to train a model
- Read the Admin Guide for architecture details and module internals

---

## 2. Installation

### System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Python | 3.7+ | 3.10+ |
| PyTorch | 1.10+ | Latest stable |
| GPU | Not required (CPU works) | CUDA-capable GPU for training |
| RAM | 4 GB | 16 GB+ for large models (GPT-NeoX) |
| Disk | 200 MB for package | 50+ GB if downloading GPT-NeoX checkpoints |

### Install Methods

=== "pip (recommended)"
    ```bash
    pip install labml-nn
    ```

=== "From source"
    ```bash
    git clone https://github.com/labmlai/annotated_deep_learning_paper_implementations.git
    cd annotated_deep_learning_paper_implementations
    pip install -r requirements.txt
    pip install -e .
    ```

> [!IMPORTANT]
> The package name on PyPI is `labml-nn` (with a hyphen), but the Python import is `labml_nn` (with an underscore).

### Dependencies

| Package | Version | Purpose |
|---|---|---|
| `torch` | ≥1.10 | Core deep learning framework |
| `torchvision` | ≥0.11 | Image datasets and transforms |
| `torchtext` | ≥0.11 | Text datasets and tokenizers |
| `labml` | ≥0.4.147 | Experiment tracking and configuration |
| `labml-helpers` | ≥0.4.84 | Dataset loaders, training loops, metrics |
| `numpy` | ≥1.19 | Numerical operations |
| `einops` | ≥0.3.0 | Tensor manipulation |
| `fairscale` | any | Distributed training utilities |

**Optional dependencies** (required only for specific implementations):

| Package | Required For |
|---|---|
| `matplotlib` | Visualization in experiments |
| `gym[atari]` | Reinforcement learning (PPO, DQN) |
| `opencv-python` | RL game frame processing, U-Net |
| `Pillow` | Image processing |
| `faiss` | kNN-LM nearest neighbor search |
| `bitsandbytes` | LLM.int8() quantization for GPT-NeoX |
| `transformers` | Sampling experiment (GPT-2 model) |

### Post-Install Verification

```bash
python -c "import labml_nn; print('labml_nn installed successfully')"
```

```text
labml_nn installed successfully
```

> [!WARNING]
> If you see `ModuleNotFoundError: No module named 'labml'`, ensure you installed `labml-nn` (which pulls in `labml` as a dependency), not just `labml_nn` from source without installing dependencies.

### Install Troubleshooting

### Error: `pip install labml-nn` generated errors

**Cause:** Version conflicts between PyTorch, torchvision, and torchtext. Community issue [#247](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/247).

**Solution:**

1. Install PyTorch first following [pytorch.org/get-started](https://pytorch.org/get-started/locally/) for your platform
2. Then install labml-nn:

```bash
pip install labml-nn
```

**Prevention:** Always install PyTorch before labml-nn to ensure compatible versions.

### Error: `No module named 'labml_helpers'`

**Cause:** The `labml-helpers` package was not installed alongside `labml-nn`.

**Solution:**

```bash
pip install labml-helpers>=0.4.84
```

---

## 3. Implementation Catalog

The library organizes implementations by research area. Every module listed below is a working PyTorch implementation you can import and use directly.

### Transformers

The largest collection, covering the original transformer and 20+ variants.

| Implementation | Paper | Module Path |
|---|---|---|
| Multi-Head Attention | Attention Is All You Need | `labml_nn.transformers.mha` |
| Encoder/Decoder Models | Attention Is All You Need | `labml_nn.transformers.models` |
| Feed-Forward Network | Attention Is All You Need | `labml_nn.transformers.feed_forward` |
| Positional Encoding | Attention Is All You Need | `labml_nn.transformers.positional_encoding` |
| Transformer XL | Transformer-XL | `labml_nn.transformers.xl` |
| Rotary Positional Embeddings (RoPE) | RoFormer | `labml_nn.transformers.rope` |
| ALiBi | Train Short, Test Long | `labml_nn.transformers.alibi` |
| RETRO | RETRO | `labml_nn.transformers.retro` |
| Compressive Transformer | Compressive Transformers | `labml_nn.transformers.compressive` |
| GPT Architecture | GPT-2 | `labml_nn.transformers.gpt` |
| GLU Variants | GLU Variants Improve Transformer | `labml_nn.transformers.glu_variants` |
| kNN-LM | Generalization through Memorization | `labml_nn.transformers.knn` |
| Feedback Transformer | Feedback Memory | `labml_nn.transformers.feedback` |
| Switch Transformer | Switch Transformers | `labml_nn.transformers.switch` |
| Fast Weights Transformer | Fast Weight Memory Systems | `labml_nn.transformers.fast_weights` |
| FNet | FNet: Fourier Transforms | `labml_nn.transformers.fnet` |
| Attention Free Transformer | AFT | `labml_nn.transformers.aft` |
| Masked Language Model (BERT) | BERT | `labml_nn.transformers.mlm` |
| MLP-Mixer | MLP-Mixer | `labml_nn.transformers.mlp_mixer` |
| gMLP | Pay Attention to MLPs | `labml_nn.transformers.gmlp` |
| Vision Transformer (ViT) | An Image Is Worth 16x16 Words | `labml_nn.transformers.vit` |
| Primer EZ | Primer | `labml_nn.transformers.primer_ez` |
| Hourglass Transformer | Hierarchical Transformers | `labml_nn.transformers.hour_glass` |
| Triton Flash Attention | Flash Attention | `labml_nn.transformers.flash` |
| JAX Transformer | — | `labml_nn.transformers.jax_transformer` |

### Diffusion Models

| Implementation | Paper | Module Path |
|---|---|---|
| DDPM | Denoising Diffusion Probabilistic Models | `labml_nn.diffusion.ddpm` |
| DDIM | Denoising Diffusion Implicit Models | `labml_nn.diffusion.stable_diffusion.sampler.ddim` |
| Latent Diffusion | Latent Diffusion Models | `labml_nn.diffusion.stable_diffusion.latent_diffusion` |
| Stable Diffusion | Stable Diffusion | `labml_nn.diffusion.stable_diffusion` |

### Generative Adversarial Networks (GANs)

| Implementation | Paper | Module Path |
|---|---|---|
| Original GAN | Generative Adversarial Nets | `labml_nn.gan.original` |
| DCGAN | DCGAN | `labml_nn.gan.dcgan` |
| CycleGAN | Unpaired Image-to-Image Translation | `labml_nn.gan.cycle_gan` |
| Wasserstein GAN | Wasserstein GAN | `labml_nn.gan.wasserstein` |
| WGAN-GP | Improved Training of WGANs | `labml_nn.gan.wasserstein.gradient_penalty` |
| StyleGAN 2 | Analyzing and Improving StyleGAN | `labml_nn.gan.stylegan` |

### Optimizers

| Implementation | Paper | Module Path |
|---|---|---|
| Adam | Adam | `labml_nn.optimizers.adam` |
| AMSGrad | On the Convergence of Adam | `labml_nn.optimizers.amsgrad` |
| Adam with Warmup | — | `labml_nn.optimizers.adam_warmup` |
| Noam | Attention Is All You Need | `labml_nn.optimizers.noam` |
| Rectified Adam (RAdam) | On the Variance of Adaptive Learning Rate | `labml_nn.optimizers.radam` |
| AdaBelief | AdaBelief Optimizer | `labml_nn.optimizers.ada_belief` |
| Sophia-G | Sophia: A Scalable Stochastic Second-order Optimizer | `labml_nn.optimizers.sophia` |

### Normalization Layers

| Implementation | Paper | Module Path |
|---|---|---|
| Batch Normalization | Batch Normalization | `labml_nn.normalization.batch_norm` |
| Layer Normalization | Layer Normalization | `labml_nn.normalization.layer_norm` |
| Instance Normalization | Instance Normalization | `labml_nn.normalization.instance_norm` |
| Group Normalization | Group Normalization | `labml_nn.normalization.group_norm` |
| Weight Standardization | Micro-Batch Training | `labml_nn.normalization.weight_standardization` |
| Batch-Channel Normalization | Batch-Channel Normalization | `labml_nn.normalization.batch_channel_norm` |
| DeepNorm | DeepNet | `labml_nn.normalization.deep_norm` |

### Sampling Techniques

| Implementation | Description | Module Path |
|---|---|---|
| Greedy Sampling | Always picks the highest-probability token | `labml_nn.sampling.greedy` |
| Temperature Sampling | Scales logits before sampling | `labml_nn.sampling.temperature` |
| Top-k Sampling | Samples from the top-k highest-probability tokens | `labml_nn.sampling.top_k` |
| Nucleus Sampling | Samples from the smallest set whose cumulative probability ≥ p | `labml_nn.sampling.nucleus` |

### Reinforcement Learning

| Implementation | Paper | Module Path |
|---|---|---|
| PPO with GAE | Proximal Policy Optimization | `labml_nn.rl.ppo` |
| DQN with Dueling + Prioritized Replay | Playing Atari with Deep RL | `labml_nn.rl.dqn` |

### Other Implementations

| Implementation | Paper | Module Path |
|---|---|---|
| LoRA | Low-Rank Adaptation | `labml_nn.lora` |
| GPT-NeoX (20B) | GPT-NeoX-20B | `labml_nn.neox` |
| RWKV | RWKV: Reinventing RNNs | `labml_nn.rwkv` |
| ResNet | Deep Residual Learning | `labml_nn.resnet` |
| ConvMixer | Patches Are All You Need | `labml_nn.conv_mixer` |
| U-Net | U-Net: Convolutional Networks | `labml_nn.unet` |
| Capsule Networks | Dynamic Routing Between Capsules | `labml_nn.capsule_networks` |
| LSTM | Long Short-Term Memory | `labml_nn.lstm` |
| Recurrent Highway Networks | Recurrent Highway Networks | `labml_nn.recurrent_highway_networks` |
| HyperNetworks / HyperLSTM | HyperNetworks | `labml_nn.hypernetworks` |
| Sketch RNN | A Neural Representation of Sketch Drawings | `labml_nn.sketch_rnn` |
| Graph Attention Networks (GAT) | Graph Attention Networks | `labml_nn.graphs.gat` |
| GATv2 | How Attentive are GATs? | `labml_nn.graphs.gatv2` |
| Counterfactual Regret Minimization | CFR | `labml_nn.cfr` |
| Kuhn Poker (CFR) | — | `labml_nn.cfr.kuhn` |
| Knowledge Distillation | Distilling the Knowledge | `labml_nn.distillation` |
| PonderNet | PonderNet: Learning to Ponder | `labml_nn.adaptive_computation.ponder_net` |
| Evidential Deep Learning | Evidential Deep Learning | `labml_nn.uncertainty.evidence` |
| Fuzzy Tiling Activations | Fuzzy Tiling Activations | `labml_nn.activations.fta` |
| Zero3 Memory Optimization | ZeRO | `labml_nn.scaling.zero3` |

---

## 4. Core Usage

### Importing and Using Modules

Every implementation is a standard PyTorch `nn.Module`. Import the class, instantiate it with the required parameters, and use it in your models.

**Using a transformer encoder:**

```python
from labml_nn.transformers import Encoder, MultiHeadAttention
from labml_nn.transformers.feed_forward import FeedForward
from labml_nn.transformers.models import TransformerLayer, EmbeddingsWithPositionalEncoding
import torch.nn as nn

# Build a 6-layer transformer encoder
d_model = 512
heads = 8
d_ff = 2048
n_layers = 6

self_attn = MultiHeadAttention(heads=heads, d_model=d_model)
feed_forward = FeedForward(d_model=d_model, d_ff=d_ff)
layer = TransformerLayer(d_model=d_model, self_attn=self_attn,
                         src_attn=None, feed_forward=feed_forward)
encoder = Encoder(layer=layer, n_layers=n_layers)
```

> [!NOTE]
> Each module is self-contained. You can use individual layers (like `MultiHeadAttention`) in your own architectures without importing the full transformer stack.

**Using a normalization layer:**

```python
from labml_nn.normalization.layer_norm import LayerNorm
import torch

ln = LayerNorm(normalized_shape=[512])
x = torch.randn(32, 10, 512)
output = ln(x)
```

**Using an optimizer:**

```python
from labml_nn.optimizers.adam import Adam
from labml_nn.optimizers import WeightDecay

optimizer = Adam(
    model.parameters(),
    lr=1e-3,
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=WeightDecay(0.01)
)
```

> [!TIP]
> The custom optimizer implementations include a `WeightDecay` class that applies decoupled weight decay (AdamW-style) separately from the gradient update. Pass `WeightDecay(decay_rate)` to any optimizer's `weight_decay` parameter.

**Using a sampling technique with a language model:**

```python
from labml_nn.sampling.nucleus import NucleusSampler
from labml_nn.sampling.temperature import TemperatureSampler

# Nucleus sampling with p=0.95 and temperature=0.8
sampler = NucleusSampler(p=0.95, sampler=TemperatureSampler(temperature=0.8))
next_token = sampler(logits)
```

All sampler classes accept a `logits` tensor of shape `[..., n_tokens]` and return sampled token indices. You can chain them — for example, `TopKSampler` wraps an inner `TemperatureSampler`.

### Using the Configuration System

Many experiments use the `labml` configuration system (`BaseConfigs`). This provides declarative, composable experiment setups.

```python
from labml_nn.transformers.configs import TransformerConfigs

# Create a configured transformer
conf = TransformerConfigs()
conf.d_model = 256
conf.heads = 4
conf.d_ff = 1024
conf.n_layers = 4
```

The `TransformerConfigs` class exposes these configurable parameters:

| Parameter | Type | Default | Description |
|---|---|---|---|
| `d_model` | `int` | — | Model dimension (embedding size) |
| `heads` | `int` | — | Number of attention heads |
| `d_ff` | `int` | 2048 | Feed-forward hidden layer size |
| `dropout` | `float` | 0.1 | Dropout probability |
| `n_layers` | `int` | — | Number of transformer layers |
| `n_src_vocab` | `int` | — | Source vocabulary size |
| `n_tgt_vocab` | `int` | — | Target vocabulary size |

The `FeedForwardConfigs` class supports GLU variants and gating:

| Parameter | Type | Default | Description |
|---|---|---|---|
| `d_model` | `int` | — | Number of input features |
| `d_ff` | `int` | 2048 | Hidden layer size |
| `dropout` | `float` | 0.1 | Dropout probability |
| `activation` | `nn.Module` | ReLU | Activation function (ReLU or GELU) |
| `is_gated` | `bool` | `False` | Whether to use gated FFN |
| `bias1` | `bool` | `True` | Bias in first linear layer |
| `bias2` | `bool` | `True` | Bias in second linear layer |
| `bias_gate` | `bool` | `False` | Bias in gate linear layer |
| `glu_variant` | `str` | `'none'` | Predefined GLU variant |

### Reading the Annotated Source

The primary value of this project is its annotated source code. Each Python file contains extensive docstrings written as tutorial prose, with mathematical notation in LaTeX. The website at [nn.labml.ai](https://nn.labml.ai) renders these as side-by-side views — code on the right, explanation on the left.

To browse locally, the `docs/` directory contains pre-rendered HTML files for each implementation.

> [!NOTE]
> The documentation website is available in three languages: [English](https://nn.labml.ai), [Chinese](https://nn.labml.ai/zh/), and [Japanese](https://nn.labml.ai/ja/).

---

## 5. Common Tasks

### Train a ResNet on CIFAR-10

**Goal:** Run the included ResNet experiment to train on CIFAR-10.

**Prerequisites:** `labml-nn` installed, GPU recommended.

**Steps:**

1. Run the experiment script:

```bash
cd annotated_deep_learning_paper_implementations
python -m labml_nn.resnet.experiment
```

2. The experiment uses `labml` for tracking. Monitor progress in your terminal or via the labml dashboard.

**Verify:** Training loss should decrease over epochs. The experiment uses the `CIFAR10Configs` configuration class.

### Train a Capsule Network on MNIST

**Goal:** Train a capsule network classifier on MNIST.

**Steps:**

1. Run the experiment:

```bash
python -m labml_nn.capsule_networks.mnist
```

Or use the provided Colab notebook:

[Open In Colab](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/capsule_networks/mnist.ipynb)

### Experiment with Language Model Sampling

**Goal:** Compare greedy, temperature, top-k, and nucleus sampling using GPT-2.

**Prerequisites:** `transformers` package installed (`pip install transformers`).

**Steps:**

1. Run the sampling experiment:

```bash
python -m labml_nn.sampling.experiment
```

2. This loads HuggingFace's GPT-2 model and generates text using each sampling strategy with the prompt "I saw an interesting dream last night."

> [!TIP]
> Modify the `prompt` variable in `labml_nn/sampling/experiment.py` to test with your own prompts. Adjust sampler parameters (temperature, k, p) to see how they affect output diversity.

### Generate Text with GPT-NeoX

**Goal:** Run inference with the 20-billion parameter GPT-NeoX model.

**Prerequisites:** A GPU with 48 GB+ VRAM. The checkpoint download requires ~40 GB of disk space.

**Steps:**

1. Checkpoints are automatically downloaded on first run:

```bash
python -m labml_nn.neox.samples.generate
```

2. The script downloads slim checkpoint weights from the Eleuther AI repository and loads them into the model.

> [!WARNING]
> GPT-NeoX requires significant GPU memory. For reduced memory usage, use the LLM.int8() quantized version at `labml_nn.neox.samples.llm_int8`, which requires the `bitsandbytes` package (`pip install bitsandbytes -U`).

### Fine-Tune GPT-2 with LoRA

**Goal:** Fine-tune GPT-2 on Tiny Shakespeare using Low-Rank Adaptation.

**Steps:**

1. Run the LoRA experiment:

```bash
python -m labml_nn.lora.experiment
```

2. This freezes the pre-trained GPT-2 weights and trains only the low-rank decomposition matrices (A and B), dramatically reducing trainable parameters. The scaling factor is controlled by `alpha / r`, where `r` is the rank of the decomposition.

### Train a U-Net for Image Segmentation

**Goal:** Train U-Net on the Carvana car segmentation dataset.

**Prerequisites:** Download the Carvana dataset, `opencv-python` installed.

**Steps:**

1. Run the experiment:

```bash
python -m labml_nn.unet.experiment
```

### Run PPO on Atari Breakout

**Goal:** Train a PPO agent to play Atari Breakout.

**Prerequisites:** `gym[atari]` and `opencv-python` installed.

**Steps:**

1. Run the PPO experiment:

```bash
python -m labml_nn.rl.ppo.experiment
```

> [!NOTE]
> The RL experiments use a custom `Game` wrapper that applies the same action for 4 frames, converts observations to grayscale at 84×84 resolution, and stacks 4 consecutive frames. Each life is treated as a separate episode. The environment is `BreakoutNoFrameskip-v4`.

### Run Knowledge Distillation on CIFAR-10

**Goal:** Train a small model to mimic a larger model's predictions.

**Steps:**

1. First train the large (teacher) model:

```bash
python -m labml_nn.distillation.large
```

2. Then train the small (student) model:

```bash
python -m labml_nn.distillation.small
```

3. Finally, run the distillation experiment that transfers knowledge:

```bash
python -m labml_nn.distillation
```

### Train RWKV Language Model

**Goal:** Train an RWKV model, a recurrence-based alternative to transformers.

**Steps:**

1. Run the RWKV experiment:

```bash
python -m labml_nn.rwkv.experiment
```

> [!TIP]
> RWKV uses linear-time inference (O(1) per token) compared to transformers' quadratic attention, making it more efficient for long sequences.

---

## 6. Troubleshooting

### Error: `RuntimeError: stack expects each tensor to be equal size, but got [2, 32] at entry 0 and [1, 32] at entry 29`

**Cause:** Mismatched tensor sizes in the RETRO implementation when batch elements have different chunk counts. Community issue [#135](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/135).

**Solution:** Ensure all inputs in the batch have the same sequence length. Pad shorter sequences to match the longest sequence in the batch before passing to the RETRO model.

**Prevention:** Use a collation function that pads sequences to uniform length.

### Error: PPO code running error

**Cause:** Version incompatibilities with the `gym` package. Community issue [#245](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/245).

**Solution:**

1. Install a compatible version of gym:

```bash
pip install gym[atari]==0.21.0
```

2. Ensure `opencv-python` is installed for frame processing.

### Error: U-Net error in DDPM

**Cause:** Shape mismatch in the U-Net skip connections used within the DDPM implementation. Community issue [#270](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/270).

**Solution:** Ensure input image dimensions are divisible by the number of downsampling stages. For the default DDPM U-Net configuration, use image sizes that are multiples of 16.

> [!WARNING]
> The DDPM implementation uses center-cropping for skip connections (community issue [#281](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/281) discusses why center crop is used instead of padding). If input dimensions are not cleanly divisible, spatial dimension mismatches will occur during the concatenation step.

### Error: `ValueError: Invalid learning rate: -0.001`

**Cause:** Negative learning rate passed to an optimizer. The `GenericAdaptiveOptimizer` base class validates that `lr >= 0`, `eps >= 0`, and `0 <= beta < 1` for both beta parameters.

**Solution:** Ensure all optimizer hyperparameters are non-negative:

```python
from labml_nn.optimizers.adam import Adam

# Correct: all positive values
optimizer = Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999), eps=1e-8)
```

### Error: `ImportError: Please install bitsandbytes`

**Cause:** The `bitsandbytes` package is required for LLM.int8() quantization but not installed.

**Solution:**

```bash
pip install bitsandbytes -U
```

### Error: `Head size {d_k} too large for flash attention`

**Cause:** The GPT-NeoX attention layer raises this when the per-head dimension exceeds what the flash attention kernel supports.

**Solution:** Use the standard attention path instead of flash attention, or reduce the model's head dimension.

### Error: Transformer-XL shift method bug

**Cause:** Potential issue in the relative position encoding shift operation. Community issue [#185](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/185).

**Solution:** Check the `labml_nn.transformers.xl.relative_mha` module for the latest fixes. Update to the latest `labml-nn`.

> [!CAUTION]
> Several implementations include known community-reported issues. Always check the [GitHub Issues](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues) page for the latest status before relying on an implementation in production.

### Error: `RuntimeError: GenericAdaptiveOptimizer does not support sparse gradients`

**Cause:** Passing sparse gradients (e.g., from `nn.Embedding` with `sparse=True`) to the custom optimizers.

**Solution:** Use PyTorch's built-in `SparseAdam` for layers with sparse gradients, or set `sparse=False` on the embedding layer.

### Error: Missing `FloatDynamicHyperParam` / `IntDynamicHyperParam` in `labml.configs`

**Cause:** These classes may not be available in your installed version of `labml`. Community issue [#291](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/291).

**Solution:**

```bash
pip install labml --upgrade
```

### Error: DDPM interpolation formula error

**Cause:** Potential mathematical error in the DDPM noise schedule interpolation. Community issue [#238](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/238).

**Solution:** Cross-reference the implementation against the original DDPM paper equations. Check the GitHub issue for discussion and proposed fixes.

### Error: GAE formula bug in PPO

**Cause:** Incorrect Generalized Advantage Estimation computation. Community issue [#255](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/255) (fixed/closed).

**Solution:** Update to the latest version of `labml-nn` where this has been addressed.

---

## 7. FAQ

**Q: Is this library meant for production use or for learning?**

The primary purpose is educational — each implementation prioritizes clarity and readability over optimization. However, every module is a functional PyTorch `nn.Module` that can be used in real projects. For production workloads, consider using official or optimized implementations (e.g., HuggingFace Transformers) alongside these for understanding.

**Q: How do I use a specific paper's implementation in my own project?**

Import the relevant module and use it as a standard PyTorch layer:

```python
from labml_nn.normalization.layer_norm import LayerNorm
from labml_nn.transformers.mha import MultiHeadAttention
```

Each module is self-contained and can be composed with your own code.

**Q: What datasets do the experiments use?**

Most experiments use standard benchmarks: MNIST, CIFAR-10, Tiny Shakespeare (text), Carvana (segmentation), and Atari games (RL). Datasets are downloaded automatically via `torchvision` or `labml` utilities on first run.

**Q: Do I need a GPU?**

Not for importing modules or small-scale experimentation. However, training experiments (especially GPT-NeoX, Stable Diffusion, or RL agents) require CUDA-capable GPUs. GPT-NeoX inference requires 48 GB+ VRAM (or use the LLM.int8() variant for reduced memory).

**Q: How do I contribute a new paper implementation?**

Follow the project's annotation style — each Python file should contain literate-programming-style docstrings that explain the algorithm alongside the code. See the [contributing discussion](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/275) on GitHub.

**Q: The StyleGAN 2 implementation seems to ignore the latent vector. Is this a bug?**

Yes, this is a known issue ([#107](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/107)). Check the GitHub issue for the latest status and community workarounds.

**Q: Is the classifier-free guidance in the diffusion implementation correct?**

Community issue [#177](https://github.com/labmlai/annotated_deep_learning_paper_implementations/issues/177) reports a potential error. Verify against the original paper if using in production.

**Q: What is the `labml` configuration system?**

The `labml` library provides a declarative configuration system where experiment parameters are defined as class attributes on `BaseConfigs` subclasses, with options registered via `@option` decorators. This allows composable experiment definitions — see the Admin Guide for full details.

**Q: Where are the annotated docs rendered?**

The website [nn.labml.ai](https://nn.labml.ai) renders the annotated source as side-by-side formatted notes. The `docs/` directory in the repository contains pre-rendered HTML pages. Translations are available in Chinese (`/zh/`), Japanese (`/ja/`), and Sinhala (`/si/`).

**Q: Can I use this with JAX instead of PyTorch?**

The library is PyTorch-native. However, there is a JAX implementation of the basic transformer at `labml_nn.transformers.jax_transformer`. All other implementations are PyTorch only.

**Q: How do I train with multiple GPUs?**

The GPT-NeoX implementation includes pipeline-parallel training support via `PipelineParallelTrainerConf`. The LoRA fine-tuning example trains on Tiny Shakespeare. For other models, integrate with PyTorch's `DistributedDataParallel` or use `fairscale` (included as a dependency).

---

_Generated May 2026 from labmlai/annotated_deep_learning_paper_implementations at 33ab022. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._