# Segment Anything Model (SAM) — User Guide

## 1. Quick Start

Segment Anything Model (SAM) is a promptable image segmentation model from Meta AI Research. Given an image and a prompt — a point, a bounding box, or a low-resolution mask — SAM produces high-quality object masks. SAM can also automatically generate masks for every object in an image without any prompt at all.

> [!NOTE]
> SAM (this repository) accepts **point, box, and mask** prompts. It does **not** accept text prompts — there is no text encoder in the released model. A successor, **Segment Anything Model 2 (SAM 2)**, extends SAM to video and lives at [facebookresearch/segment-anything-2](https://github.com/facebookresearch/segment-anything-2). This guide documents the original image-only SAM.

**Key features:**

- Promptable segmentation with points, boxes, or masks as input
- Automatic mask generation for entire images without prompts
- Three model sizes: ViT-H (default, highest quality), ViT-L, and ViT-B (fastest)
- ONNX export for deployment in non-Python environments
- Batch inference support for processing multiple prompts simultaneously
- Multi-mask output with quality scores for ambiguous prompts
- Crop-based generation for higher resolution mask coverage
- Interactive browser demo using ONNX Runtime Web

**Prerequisites:** Python 3.8+, PyTorch 1.7+, torchvision

**Install and run in 60 seconds:**

```bash
pip install git+https://github.com/facebookresearch/segment-anything.git
pip install opencv-python pycocotools
```

```python
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["default"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
)
```

> [!TIP]
> Start with `vit_b` for experimentation — it loads fastest and uses the least GPU memory. Switch to `vit_h` when you need maximum mask quality.

**What to try next:** See the Prompted Segmentation section for point, box, and mask prompts. See Automatic Mask Generation to segment every object in an image.

## 2. Installation

### System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Python | 3.8+ | 3.10+ |
| PyTorch | 1.7+ | 2.0+ |
| GPU VRAM (ViT-B) | ~4 GB | 8 GB |
| GPU VRAM (ViT-L) | ~8 GB | 12 GB |
| GPU VRAM (ViT-H) | ~12 GB | 16 GB+ |
| RAM | 8 GB | 16 GB+ |

### Install from Source

```bash
pip install git+https://github.com/facebookresearch/segment-anything.git
```

### Install with Optional Dependencies

The base install has no mandatory dependencies beyond PyTorch. Optional extras are available:

```bash
# All optional dependencies (matplotlib, pycocotools, opencv-python, onnx, onnxruntime)
pip install 'segment-anything[all] @ git+https://github.com/facebookresearch/segment-anything.git'

# Development tools only (flake8, isort, black, mypy)
pip install 'segment-anything[dev] @ git+https://github.com/facebookresearch/segment-anything.git'
```

> [!IMPORTANT]
> The `pycocotools` package is required if you use `output_mode="coco_rle"` in `SamAutomaticMaskGenerator`. The `opencv-python` package is required if you set `min_mask_region_area > 0` for postprocessing small regions.

### Download Model Checkpoints

Three model checkpoints are available, corresponding to the three ViT backbone sizes:

| Model | Checkpoint | Parameters | Encoder Depth | Encoder Heads |
|---|---|---|---|---|
| `vit_h` (default) | `sam_vit_h_4b8939.pth` | Largest | 32 layers | 16 heads |
| `vit_l` | `sam_vit_l_0b3195.pth` | Medium | 24 layers | 16 heads |
| `vit_b` | `sam_vit_b_01ec64.pth` | Smallest | 12 layers | 12 heads |

### Verify Installation

```python
import segment_anything
from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator
print("SAM installed successfully")
```

```text
SAM installed successfully
```

### Install Troubleshooting

**Error: `ModuleNotFoundError: No module named 'segment_anything'`**

Ensure you installed from the git repository, not from PyPI. The package is not published on PyPI.

**Error: `torch.cuda.OutOfMemoryError` when loading the model**

Use a smaller model variant. Switch from `vit_h` to `vit_l` or `vit_b`, or move the model to CPU with `sam.to(device="cpu")`.

**Error: `ImportError: No module named 'pycocotools'`**

Install pycocotools: `pip install pycocotools`. On Windows, you may need `pip install pycocotools-windows` instead.

**Error: `ImportError: No module named 'cv2'`**

Install OpenCV: `pip install opencv-python`. Required only when using `min_mask_region_area > 0`.

**Error: `quantize_dynamic() got an unexpected keyword argument 'optimize_model'`**

This occurs with newer versions of `onnxruntime`. Downgrade to a compatible version: `pip install onnxruntime==1.14.0`. (Community issue #585)

**Error: `RuntimeError` when using batch size > 1**

Batched inference via `Sam.forward()` requires all images in the batch to be pre-processed to the same size. Use `SamPredictor` for single-image workflows. (Community issue #277)

## 3. Configuration

SAM does not use configuration files. All behavior is controlled through constructor parameters and method arguments passed in Python code.

For the complete parameter reference for `SamAutomaticMaskGenerator` and `SamPredictor`, see the Admin Guide Configuration Reference.

## 4. Core Usage

### Prompted Segmentation with SamPredictor

`SamPredictor` is the primary interface for interactive segmentation. It computes an image embedding once, then produces masks efficiently for any number of prompts.

#### Step 1: Load a Model and Create a Predictor

```python
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

predictor = SamPredictor(sam)
```

The `sam_model_registry` maps model type strings to builder functions:

| Key | Builder Function | Description |
|---|---|---|
| `"default"` | `build_sam_vit_h` | Alias for ViT-H |
| `"vit_h"` | `build_sam_vit_h` | ViT-Huge backbone (1280-dim, 32 blocks) |
| `"vit_l"` | `build_sam_vit_l` | ViT-Large backbone (1024-dim, 24 blocks) |
| `"vit_b"` | `build_sam_vit_b` | ViT-Base backbone (768-dim, 12 blocks) |

#### Step 2: Set an Image

```python
import numpy as np
from PIL import Image

image = np.array(Image.open("photo.jpg"))
predictor.set_image(image)
```

The image must be a NumPy array in HWC uint8 format with pixel values in [0, 255]. The default expected color format is RGB. If your image is in BGR format (e.g., from OpenCV's `cv2.imread`):

```python
predictor.set_image(image, image_format="BGR")
```

> [!NOTE]
> `set_image` computes the image embedding using the image encoder. This is the most expensive step. Once set, you can run `predict` many times with different prompts efficiently without re-encoding.

#### Step 3: Predict with Point Prompts

```python
point_coords = np.array([[500, 375]])   # (X, Y) pixel coordinates
point_labels = np.array([1])            # 1 = foreground, 0 = background

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True,
)
```

**Returns:**

| Return Value | Shape | Description |
|---|---|---|
| `masks` | `CxHxW` | Binary masks (C=3 when `multimask_output=True`, C=1 otherwise) |
| `scores` | `C` | Model's predicted quality score for each mask |
| `logits` | `Cx256x256` | Low-resolution mask logits (can feed back as `mask_input`) |

> [!TIP]
> For ambiguous prompts like a single point, use `multimask_output=True` and pick the mask with the highest score. For unambiguous prompts like multiple points or a bounding box, use `multimask_output=False` for better results.

#### Step 4: Predict with Box Prompts

```python
box = np.array([100, 200, 400, 500])   # [x1, y1, x2, y2] in XYXY format

masks, scores, logits = predictor.predict(
    box=box,
    multimask_output=False,
)
```

#### Step 5: Combine Point and Box Prompts

```python
masks, scores, logits = predictor.predict(
    point_coords=np.array([[250, 350]]),
    point_labels=np.array([1]),
    box=np.array([100, 200, 400, 500]),
    multimask_output=False,
)
```

#### Step 6: Iterative Refinement with Mask Input

Feed the low-resolution logits from a previous prediction as input to refine the mask:

```python
masks, scores, logits = predictor.predict(
    point_coords=np.array([[250, 350]]),
    point_labels=np.array([1]),
    mask_input=logits[np.argmax(scores), :, :][None, :, :],  # shape 1x256x256
    multimask_output=False,
)
```

> [!WARNING]
> You must call `set_image` before `predict`. Calling `predict` without first setting an image raises `RuntimeError: An image must be set with .set_image(...) before mask prediction.`

#### Step 7: Retrieve the Image Embedding

To reuse the embedding (e.g., save it for ONNX inference):

```python
embedding = predictor.get_image_embedding()
# Shape: 1xCxHxW (typically 1x256x64x64)
```

#### Step 8: Reset Between Images

```python
predictor.reset_image()
```

This clears the stored embedding and internal state. Always call `reset_image()` or `set_image()` with a new image before switching to a different input.

### Automatic Mask Generation with SamAutomaticMaskGenerator

`SamAutomaticMaskGenerator` segments every object in an image without any manual prompts. It generates a grid of point prompts, runs SAM on each, and applies filtering and NMS to produce clean masks.

#### Basic Usage

```python
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(image)
```

The `generate` method accepts a NumPy array in HWC uint8 format and returns a list of mask dictionaries:

| Field | Type | Description |
|---|---|---|
| `segmentation` | `np.ndarray` or dict | Binary mask (HxW) or RLE dict, depending on `output_mode` |
| `bbox` | `list[float]` | Bounding box in XYWH format |
| `area` | `int` | Mask area in pixels |
| `predicted_iou` | `float` | Model's quality prediction for this mask |
| `point_coords` | `list[list[float]]` | The point prompt that generated this mask |
| `stability_score` | `float` | Measure of mask boundary stability |
| `crop_box` | `list[float]` | The image crop used for this mask, in XYWH format |

#### Tuning Generation Quality

```python
mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=32,                 # Grid density (default: 32 → 1024 points)
    pred_iou_thresh=0.88,               # Min predicted IoU to keep (default: 0.88)
    stability_score_thresh=0.95,        # Min stability score to keep (default: 0.95)
    box_nms_thresh=0.7,                 # NMS threshold for deduplication (default: 0.7)
    min_mask_region_area=100,           # Remove tiny regions (requires opencv)
)
```

> [!NOTE]
> Increasing `points_per_side` produces more masks at the cost of runtime. The total number of initial point prompts is `points_per_side ** 2` (e.g., 32 → 1024 points, 64 → 4096 points).

#### Multi-Crop Generation for Dense Scenes

```python
mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    crop_n_layers=1,                     # Run on crops at 1 additional scale
    crop_nms_thresh=0.7,                 # NMS between crops
    crop_overlap_ratio=512/1500,         # Overlap between adjacent crops
    crop_n_points_downscale_factor=1,    # Point density reduction per layer
)
```

When `crop_n_layers > 0`, the generator runs on the full image plus cropped sub-regions. Layer `n` produces `(2^n)^2` crops. Masks from different crops are deduplicated via NMS, preferring masks from smaller crops.

#### Output Modes

| Mode | Value | Requirements | Description |
|---|---|---|---|
| Binary mask | `"binary_mask"` | None | NumPy boolean array (HxW). Default. |
| Uncompressed RLE | `"uncompressed_rle"` | None | Dictionary with `size` and `counts` |
| COCO RLE | `"coco_rle"` | `pycocotools` | Compressed RLE compatible with COCO evaluation |

```python
mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    output_mode="coco_rle",
)
```

> [!WARNING]
> Using `"binary_mask"` output mode on very high-resolution images can consume large amounts of memory since each mask is stored as a full-resolution boolean array. Use `"coco_rle"` or `"uncompressed_rle"` for large images.

### Batch Inference with the Sam Model Directly

For advanced use cases, call the `Sam` model's `forward` method directly with batched inputs:

```python
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

batched_input = [
    {
        "image": preprocessed_image_tensor,          # 3xHxW float tensor
        "original_size": (orig_h, orig_w),
        "point_coords": point_coords_tensor,         # BxNx2
        "point_labels": point_labels_tensor,          # BxN
    },
]

outputs = sam(batched_input, multimask_output=False)
```

Each output dictionary contains:

| Key | Shape | Description |
|---|---|---|
| `"masks"` | `BxCxHxW` | Binary mask predictions at original resolution |
| `"iou_predictions"` | `BxC` | Predicted mask quality scores |
| `"low_res_logits"` | `BxCx256x256` | Low-resolution logits for iterative refinement |

> [!IMPORTANT]
> When calling `Sam.forward()` directly, images must already be transformed and normalized. Use `SamPredictor` for simpler single-image workflows.

## 5. Common Tasks

### Generate Masks for All Objects in an Image

**Goal:** Automatically segment every visible object.

**Steps:**

1. Load the model and create a mask generator:

```python
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator
import cv2
import numpy as np

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
generator = SamAutomaticMaskGenerator(sam)
```

2. Load and generate masks:

```python
image = cv2.imread("photo.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
masks = generator.generate(image)
print(f"Found {len(masks)} masks")
```

3. Sort by area and visualize:

```python
sorted_masks = sorted(masks, key=lambda x: x["area"], reverse=True)
for i, mask in enumerate(sorted_masks[:5]):
    print(f"Mask {i}: area={mask['area']}, iou={mask['predicted_iou']:.3f}")
```

### Segment a Specific Object with a Click

**Goal:** Select an object by clicking on it.

```python
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
predictor = SamPredictor(sam)

predictor.set_image(image)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[x, y]]),
    point_labels=np.array([1]),
    multimask_output=True,
)
best_mask = masks[np.argmax(scores)]
```

### Exclude a Region with Background Points

**Goal:** Mark an area as background to refine the mask.

```python
point_coords = np.array([[500, 375], [600, 200]])
point_labels = np.array([1, 0])  # 1=foreground, 0=background

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=False,
)
```

### Process a Directory of Images with the AMG Script

**Goal:** Batch-process images using the command-line script.

```bash
python scripts/amg.py \
    --input ./images/ \
    --output ./masks/ \
    --model-type vit_h \
    --checkpoint sam_vit_h_4b8939.pth \
    --device cuda
```

This produces a folder of PNG masks and a `metadata.csv` per input image.

To save masks in COCO RLE format as JSON instead:

```bash
python scripts/amg.py \
    --input ./images/ \
    --output ./masks/ \
    --model-type vit_h \
    --checkpoint sam_vit_h_4b8939.pth \
    --convert-to-rle
```

### Export a Model to ONNX

**Goal:** Export the prompt encoder and mask decoder for deployment.

```bash
python scripts/export_onnx_model.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --output sam_onnx.onnx
```

To export a quantized model for smaller size:

```bash
python scripts/export_onnx_model.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --output sam_onnx.onnx \
    --quantize-out sam_onnx_quantized.onnx
```

> [!TIP]
> The ONNX model exports only the prompt encoder and mask decoder — not the image encoder. You still need to compute image embeddings in PyTorch (or pre-compute and save them as `.npy` files) before running the ONNX model.

### Export with GELU Approximation for Compatibility

For runtimes with slow or missing `erf` operations (used in GELU):

```bash
python scripts/export_onnx_model.py \
    --checkpoint sam_vit_b_01ec64.pth \
    --model-type vit_b \
    --output sam_onnx.onnx \
    --gelu-approximate
```

### Run the Interactive Browser Demo

**Goal:** Try SAM in a browser with click-based prompting.

**Prerequisites:** Node.js, Yarn, a pre-computed image embedding `.npy` file, and a quantized ONNX model.

```bash
cd demo
yarn install
```

Place the ONNX model at `demo/model/sam_onnx_quantized_example.onnx` and the image embedding at `demo/src/assets/data/dogs_embedding.npy`.

```bash
yarn start
```

The demo opens a browser window where hovering over the image generates masks in real time using ONNX Runtime Web.

### Use Pre-Transformed Torch Tensors

For workflows that already have images as tensors:

```python
import torch

# Image must be 1x3xHxW with longest side resized to 1024
transformed_image = torch.randn(1, 3, 1024, 768).to("cuda")

predictor.set_torch_image(
    transformed_image,
    original_image_size=(2048, 1536),
)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
)
```

### Predict with Torch Tensors Directly

For batched GPU workflows, use `predict_torch` instead of `predict`:

```python
masks, scores, logits = predictor.predict_torch(
    point_coords=torch.tensor([[[500.0, 375.0]]], device="cuda"),
    point_labels=torch.tensor([[1]], device="cuda"),
    multimask_output=True,
)
# masks shape: Bx3xHxW (torch.Tensor on GPU)
```

## 6. Troubleshooting

### Error: `RuntimeError: An image must be set with .set_image(...) before mask prediction.`

**Cause:** You called `predict()`, `predict_torch()`, or `get_image_embedding()` without first calling `set_image()` or `set_torch_image()`.

**Solution:**

1. Call `predictor.set_image(image)` before calling `predict()`

**Prevention:** Always set an image immediately after creating the predictor or after calling `reset_image()`.

### Error: `ValueError: Can't have both points_per_side and point_grid be None.`

**Cause:** `SamAutomaticMaskGenerator` was initialized with both `points_per_side=None` and `point_grids=None`.

**Solution:**

1. Provide `points_per_side` (integer) OR `point_grids` (list of NumPy arrays)
2. Exactly one of these two parameters must be non-None

### Error: `AssertionError: Exactly one of points_per_side or point_grid must be provided.`

**Cause:** Both `points_per_side` and `point_grids` were given non-None values.

**Solution:** Set one to `None`. Use `points_per_side` for a regular grid or `point_grids` for a custom point layout.

### Error: `torch.cuda.OutOfMemoryError` during mask generation

**Cause:** The image is too large or the model too big for available GPU memory.

**Solution:**

1. Reduce `points_per_batch` (default: 64) to process fewer points at a time
2. Use a smaller model (`vit_b` instead of `vit_h`)
3. Move to CPU: `sam.to(device="cpu")` (much slower but works)
4. Resize the input image to a smaller resolution

**Prevention:** Monitor GPU memory with `torch.cuda.memory_summary()` before processing large images.

### Error: `Module not found: Error: Can't resolve 'process/browser'` in the demo

**Cause:** The webpack configuration expects the `process` polyfill.

**Solution:**

1. Ensure `process` is listed in `demo/package.json` dependencies
2. Run `yarn install` to install all dependencies

Community issue #678 provides additional context.

### Masks look low quality or noisy

**Cause:** Default thresholds may not suit all images.

**Solution:**

1. Increase `pred_iou_thresh` (e.g., from 0.88 to 0.92) to keep only high-confidence masks
2. Increase `stability_score_thresh` (e.g., from 0.95 to 0.98) to keep only stable masks
3. Set `min_mask_region_area` to a positive value (e.g., 100) to remove small disconnected regions

> [!CAUTION]
> Setting `min_mask_region_area > 0` requires `opencv-python`. If OpenCV is not installed, `SamAutomaticMaskGenerator.__init__` will raise an `ImportError`.

### Demo shows incorrect or no mask overlay

**Cause:** The ONNX model or image embedding file is not in the expected location.

**Solution:**

1. Verify `demo/model/sam_onnx_quantized_example.onnx` exists
2. Verify `demo/src/assets/data/dogs_embedding.npy` exists
3. Check the browser console for loading errors

> [!WARNING]
> The ONNX Runtime Web demo requires `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: credentialless` headers. The dev server sets these automatically. If deploying the built demo, configure your web server to include these headers.

## 7. FAQ

**Q: Which model variant should I use?**

Use `vit_h` for the best mask quality — it is the default. Use `vit_b` for the fastest inference and lowest memory usage. `vit_l` is a middle ground.

**Q: Can I run SAM on CPU?**

Yes. Pass `device="cpu"` when moving the model: `sam.to(device="cpu")`. Inference will be significantly slower but functional.

**Q: What image formats does SAM accept?**

SAM accepts NumPy arrays in HWC uint8 format (height x width x channels, values 0-255). Both RGB and BGR formats are supported via the `image_format` parameter in `set_image`.

**Q: Does SAM support video segmentation?**

SAM v1 (this repository) is designed for single-image segmentation. It does not have built-in video or temporal tracking capabilities. For video and streaming-memory tracking, use the successor model **SAM 2** at [facebookresearch/segment-anything-2](https://github.com/facebookresearch/segment-anything-2).

**Q: How do I get the best mask for a single point prompt?**

Use `multimask_output=True` and select the mask with the highest score: `best_mask = masks[np.argmax(scores)]`. For unambiguous prompts (multiple points or a box), use `multimask_output=False`.

**Q: Can I combine points and boxes in one prompt?**

Yes. Pass both `point_coords`/`point_labels` and `box` to `predictor.predict()`. The model treats them as complementary prompts.

**Q: What is `stability_score`?**

The stability score measures how much the predicted mask changes when you slightly vary the threshold used to binarize the mask logits. A high stability score means the boundary is well-defined. It is computed as the IoU between masks obtained at `threshold + offset` and `threshold - offset`.

**Q: How do I save masks to disk?**

For binary masks, use OpenCV or PIL:

```python
import cv2
cv2.imwrite("mask.png", masks[0].astype(np.uint8) * 255)
```

For COCO RLE format, use `output_mode="coco_rle"` in the automatic generator and save the result as JSON.

**Q: What does the ONNX export include?**

The ONNX export includes the prompt encoder and mask decoder only. The image encoder (ViT backbone) is not exported. You must pre-compute image embeddings in PyTorch and provide them as input to the ONNX model.

**Q: Is the SA-1B dataset still available?**

Check the project's official pages for current dataset availability. (Community issue #850)

---

_Generated June 2026 from facebookresearch/segment-anything at dca509f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._