# Sentry Administrator & Architecture Guide

This guide is for the engineers, architects, and operators who **deploy, operate,
secure, and extend** Sentry — the `getsentry/sentry` server that powers both
sentry.io and self-hosted installations. It documents the system architecture,
the service topology, the complete configuration and CLI surface, the security
model, day-2 operations, and the internals a developer needs to work in the
codebase.

For the product/feature walkthrough and SDK onboarding, see the companion
**User Guide**.

> [!NOTE]
> Sentry is a large, polyglot, multi-service platform — roughly 20,000 files
> across a Django/Python backend and a React/TypeScript frontend, fronting a
> fleet of supporting services (PostgreSQL, Redis, Kafka, ClickHouse/Snuba,
> Relay, Symbolicator). This guide maps that system from the source as it exists
> at commit `cd5d5048`. Where a subsystem is implemented across many files, the
> guide names the authoritative module so you can read further.

---

## 1. Executive Summary

Sentry ingests application telemetry (errors, traces/spans, replays, profiles,
logs, session health, cron check-ins, uptime probes), processes it through a
streaming pipeline, stores it across purpose-built backends, and serves a
React single-page application plus a public REST API for triage and analysis.

| Dimension | Summary |
|---|---|
| **Backend** | Python 3.13, Django 5.2, Django REST Framework 3.16, served by **Granian** (WSGI/ASGI). |
| **Frontend** | React 19 + TypeScript, bundled with **rspack**, styled with Emotion, state via Reflux/MobX + TanStack Query. |
| **Async processing** | Kafka consumers via **sentry-arroyo**; background tasks via the **taskworker/taskbroker** system (gRPC-based, replacing Celery). |
| **Primary datastores** | **PostgreSQL** (relational state), **ClickHouse via Snuba** (events/search/aggregates), **Redis** (cache, buffers, quotas, rate limits, digests), object/file storage (filesystem or S3/GCS), **Nodestore** (raw event blobs). |
| **Ingestion edge** | **Relay** (Rust) receives SDK events, scrubs PII, applies quotas, forwards to Kafka. **Symbolicator** symbolicates native/mobile stack traces. |
| **Deployment model** | **Hybrid-cloud silos**: a Control Silo (auth, integrations, global config) and one or more Region Silos (tenant event data). Self-hosted runs **monolith** mode (all-in-one). |
| **License** | Functional Source License **FSL-1.1-Apache-2.0** (source-available; converts to Apache-2.0 after two years). |
| **Self-hosting** | Orchestrated by the separate `getsentry/self-hosted` Docker Compose project; this repo ships the application image (`python:3.13-slim`, port 9000) and the `sentry` CLI. |

---

## 2. System Architecture

### 2.1 The event pipeline

The end-to-end path of a single error event:

```text
   Your application
        │  (SDK: sentry-python, sentry-javascript, …)
        ▼
   ┌──────────┐   envelope over HTTPS
   │  Relay   │   • inbound filters, PII scrubbing, quotas/rate limits,
   │  (Rust)  │     normalization, dynamic sampling
   └────┬─────┘
        │ publishes to Kafka topics
        ▼
   ┌──────────────────── Kafka ────────────────────┐
   │  ingest-events, ingest-transactions, replays,  │
   │  outcomes, attachments, …                       │
   └────┬───────────────────────────────────────────┘
        │  consumed by Arroyo-based consumers
        ▼
   ┌─────────────────┐      ┌──────────────────────────┐
   │ sentry consumers │ ──▶ │ Symbolicator (native/JS)  │
   │ (ingest, post-   │     └──────────────────────────┘
   │  process-forward,│
   │  snuba writers)  │ ──▶ Nodestore (raw event blob)
   └───────┬─────────┘ ──▶ PostgreSQL (Group/Issue, metadata)
           │            ──▶ Snuba ▶ ClickHouse (searchable events, aggregates)
           ▼
   ┌──────────────────┐   post-processing: grouping, alert rule
   │ taskworker tasks │   evaluation, notifications, integrations,
   │ (gRPC taskbroker)│   suspect commits, Seer triggers
   └──────────────────┘
           │
           ▼
   Notifications → email / Slack / Teams / Discord / PagerDuty / Opsgenie / webhooks
```

Reading the diagram:

1. **Ingestion (Relay).** SDKs send *envelopes* to Relay. Relay applies inbound
   filters, **server-side PII scrubbing**, **rate limits/quotas**, normalization,
   and **dynamic sampling**, then publishes to Kafka. `SENTRY_USE_RELAY = True`
   in the self-hosted settings.
2. **Streaming (Kafka + Arroyo).** Sentry's consumers (`sentry run consumer
   <name>`) read Kafka topics. The Arroyo library (`sentry-arroyo`) provides the
   consumer runtime with batching, offset management, dead-letter queues (DLQ),
   and health checks.
3. **Storage fan-out.** Events are written to **Nodestore** (raw blob),
   **PostgreSQL** (the `Group`/issue and relational metadata), and — via
   **Snuba** — to **ClickHouse**, which backs all search, stats, and aggregation.
4. **Post-processing (taskworker).** Grouping, alert-rule evaluation, digest
   building, integration syncs, suspect-commit resolution, and Seer triggers run
   as background tasks executed by `sentry run taskworker` against a gRPC
   **taskbroker**, scheduled by `sentry run taskworker-scheduler`.
5. **Serving.** The Django/DRF app (run by Granian via `sentry run web`) serves
   the REST API and the React SPA; reads come from PostgreSQL, Redis, and Snuba.

> [!IMPORTANT]
> Errors can group and appear from PostgreSQL even when Snuba is down, but
> **search, stats, dashboards, Discover, and metric alerts all depend on
> Snuba/ClickHouse**. Treat Snuba and the Kafka consumers as load-bearing for
> the analytics half of the product.

### 2.2 Hybrid-cloud silo model

To support data-residency (US/EU) and isolation, Sentry runs in a **siloed**
topology controlled by `SENTRY_SILO_MODE` (`src/sentry/conf/server.py`):

| Silo | Owns | Examples |
|---|---|---|
| **Control Silo** | Global, cross-region state | Users, authentication, API tokens, integrations/installations, audit log, organization mapping |
| **Region Silo** | Tenant event data, region-local | Organizations' projects, events, issues, releases, alerts |
| **Monolith** | Everything in one process | Self-hosted default — no silo separation |

Cross-silo communication goes through an **RPC layer** (`*/services/` packages
expose `rpc`-decorated service methods; schema validated via `sentry rpcschema`)
and an **outbox**/replica mechanism (`hybridcloud/`) that asynchronously
replicates control-silo objects (e.g. API tokens) to regions. Self-hosted
operators run **monolith** mode and can ignore silo configuration; SaaS runs
split silos with an API Gateway (`src/apigw/`) routing requests to the correct
region.

### 2.3 Backend service architecture

The Django application is organized as one project with many internal Django
apps under `src/sentry/`. Each major product/subsystem is a package:

| Subsystem (`src/sentry/…`) | Responsibility |
|---|---|
| `api/` | REST API: `Endpoint` base classes, 190+ endpoint modules, `urls.py`, serializers, permissions, pagination |
| `integrations/` | Third-party integrations (Slack, Jira, GitHub/GitLab/Bitbucket, PagerDuty, Opsgenie, MS Teams, Discord, Vercel, AWS Lambda, AI coding agents) + the integration framework |
| `workflow_engine/` | Detectors → conditions → actions automation engine (the modern alerts/automation backbone) |
| `seer/` | AI debugging (root-cause analysis, autofix, issue scanning) integration with the Seer service |
| `issues/` | Issue/Group platform, issue types, grouping orchestration, occurrence consumer |
| `grouping/` | Event fingerprinting/grouping strategies |
| `models/`, `users/models/`, `db/models/` | Django ORM models and the custom model framework (`BaseModel`, hybrid-cloud foreign keys) |
| `tasks/`, `taskworker/` | Background tasks and the taskworker runtime |
| `snuba/`, `search/`, `tsdb/` | ClickHouse/Snuba clients, search backends, time-series DB |
| `replays/`, `profiles`/`spans/`, `monitors/`, `uptime/`, `incidents/` | Per-product backends (Session Replay, Profiling/Tracing, Crons, Uptime, Metric Alerts/Incidents) |
| `relay/`, `ingest/` | Relay configuration and event-ingestion glue |
| `auth/`, `auth_v2/` | Authentication, SSO/SAML providers, superuser/staff |
| `hybridcloud/`, `services/`, `*/services/` | Silo RPC services and cross-silo replication |
| `notifications/` | Notification routing and templates |
| `sentry_apps/` | Integration Platform (Sentry Apps / internal integrations) |
| `web/` | Server-rendered shell, frontend asset serving, auth views |
| `runner/` | The `sentry` click CLI (commands in `runner/commands/`) |
| `conf/` | `server.py` — the canonical Django settings + `SENTRY_OPTIONS` |
| `migrations/` | 200+ Django migrations (schema history) |

### 2.4 Frontend architecture

The SPA lives in `static/app/` (with `static/gsApp/` and `static/gsAdmin/` for
SaaS billing/admin surfaces):

- **Stack:** React 19, TypeScript 6, **rspack** bundler (Rust webpack-compatible),
  Emotion for CSS-in-JS, **react-router-dom 6** for routing.
- **State:** legacy **Reflux** stores + **MobX** in places, with **TanStack
  Query** (`@tanstack/react-query`) for server-state/data fetching and **nuqs**
  for URL state.
- **Entry points:** `static/app/index.tsx` → `static/app/main.tsx`; routes in
  `static/app/router/`; views in `static/app/views/`; shared components in
  `static/app/components/`.
- **Build:** `pnpm` workspace; `rspack.config.ts` at the root. Dev is served by
  `sentry devserver` (which also proxies to the backend) or `pnpm dev-ui`.

> [!NOTE]
> The frontend includes onboarding doc generators (`static/app/gettingStartedDocs/`)
> with one module per platform — these render the per-SDK install snippets you see
> in the project-creation flow.

---

## 3. Requirements

### Runtime dependencies (services)

A functioning Sentry needs all of:

| Service | Purpose | Notes |
|---|---|---|
| **PostgreSQL** | Primary relational store | Uses the custom `sentry.db.postgres` engine. Recent major version. |
| **Redis** | Cache, buffers, quotas, rate limits, digests, TSDB front | **Required.** The self-hosted config raises an error if `SENTRY_REDIS_HOST` is unset. |
| **Kafka** | Event/stream transport | Drives all consumers. |
| **ClickHouse + Snuba** | Searchable event storage & aggregation | Snuba is Sentry's service in front of ClickHouse. |
| **Relay** | Event ingestion edge | `SENTRY_USE_RELAY=True`. |
| **Symbolicator** | Native/JS/mobile symbolication | Needed for de-minified/symbolicated native stack traces. |
| **Memcached** | Optional secondary cache | Recommended for throughput; configured via `SENTRY_MEMCACHED_HOST`. |
| **SMTP server** | Outbound email | Optional; defaults to the `dummy` backend (no mail) until configured. |
| **Object storage** | Attachments, debug files, source maps | Filesystem (`/data/files`) by default; S3/GCS supported. |

### Application runtime

| Component | Version (from source) |
|---|---|
| Python | **3.13** (image is `python:3.13.1-slim-bookworm`) |
| Dependency manager | **uv** (`uv.lock` + `pyproject.toml`); installed editable via `tools/fast_editable.py` |
| Node / package manager | **pnpm 10** (for building frontend assets) |
| Web server | **Granian ≥ 2.7** (`granian[pname,reload,uvloop]`) |
| Key Python deps | `django>=5.2.14`, `djangorestframework>=3.16.1`, `drf-spectacular`, `click>=8.2.1`, `confluent-kafka`, `sentry-arroyo`, `snuba-sdk`, `sentry-relay`, `symbolic`, `psycopg2-binary`, `redis`, `google-cloud-*`, `boto3`, `openai`, `pydantic<2` |

> [!CAUTION]
> Sentry pins **Pydantic v1** (`pydantic>=1.10.26,<2`). When working in the
> codebase, do not assume Pydantic v2 APIs.

---

## 4. Installation & Deployment

### 4.1 Self-hosted (recommended path)

Use the official Compose project, which wires up Sentry plus every required
service:

```bash
git clone https://github.com/getsentry/self-hosted.git
cd self-hosted
./install.sh          # interactive: builds images, runs migrations, creates the first user
docker compose up -d  # start the full stack
```

`install.sh` runs `sentry upgrade` (migrations + Kafka topics + nodestore
bootstrap) and prompts you to create the initial owner account. The web UI is
served on port **9000**.

### 4.2 The application image

The image built from this repo (`self-hosted/Dockerfile`) is the unit the Compose
project runs for every Sentry role:

- Base `python:3.13.1-slim-bookworm`, runs as non-root user `sentry` (uid/gid 999).
- `SENTRY_CONF=/etc/sentry`, `GRPC_POLL_STRATEGY=epoll1`.
- Dependencies installed with `uv sync --frozen`.
- `ENTRYPOINT ["/docker-entrypoint.sh"]`, default `CMD ["run", "web"]` →
  `sentry run web` on port 9000, `VOLUME /data`.
- The same image is launched with different commands for each role (`run web`,
  `run taskworker`, `run consumer …`, etc.).

### 4.3 Configuration files

Configuration lives in `$SENTRY_CONF` (default `/etc/sentry`), seeded from
`self-hosted/`:

| File | Format | Holds |
|---|---|---|
| `sentry.conf.py` | Python/Django | Imports `from sentry.conf.server import *`, then overrides `DATABASES`, `CACHES`, `SENTRY_OPTIONS`, backends, web host/port. |
| `config.yml` | YAML | Runtime options: `mail.*`, `filestore.*`, `system.secret-key`, `redis.clusters`, etc. UI-settable options can be pinned here. |

`sentry init <directory>` scaffolds a fresh config directory (`--dev` for
development-friendly settings).

### 4.4 Key deployment environment variables

The self-hosted `sentry.conf.py` reads these (all optional unless noted):

| Variable | Purpose |
|---|---|
| `SENTRY_SECRET_KEY` | **Required.** Signing key (≥ 32 chars). Generate with `sentry config generate-secret-key`. |
| `SENTRY_POSTGRES_HOST` / `_PORT`, `SENTRY_DB_NAME` / `_USER` / `_PASSWORD` | PostgreSQL connection. |
| `SENTRY_REDIS_HOST` / `_PORT` / `_PASSWORD` / `_DB` | **Required** Redis connection (default port 6379). |
| `SENTRY_MEMCACHED_HOST` / `_PORT` | Optional memcached cache (default 11211). |
| `SENTRY_EMAIL_HOST` / `_PORT` / `_USER` / `_PASSWORD`, `SENTRY_EMAIL_USE_TLS` / `_USE_SSL` | SMTP; absence → `dummy` backend. |
| `SENTRY_SERVER_EMAIL` | From-address for outbound mail. |
| `SENTRY_ENABLE_EMAIL_REPLIES`, `SENTRY_SMTP_HOSTNAME`, `SENTRY_MAILGUN_API_KEY` | Inbound mail / email replies. |
| `SENTRY_FILESTORE_DIR` | Local file storage location. |
| `SENTRY_SINGLE_ORGANIZATION` | Single-org UI optimizations (self-hosted default `True`). |
| `SENTRY_CONF` | Path to the config directory. |

> [!IMPORTANT]
> If you terminate TLS at a reverse proxy, uncomment the proxy settings in
> `sentry.conf.py`: `SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')`,
> `SESSION_COOKIE_SECURE = True`, `CSRF_COOKIE_SECURE = True`. Otherwise secure
> cookies and HTTPS detection will be wrong.

### 4.5 Process topology in production

A real deployment runs these roles (each is the image with a different command):

- [ ] `sentry run web` — Granian web/API server (scale horizontally behind a proxy).
- [ ] `sentry run taskworker` — background task workers (scale by replica/concurrency).
- [ ] `sentry run taskworker-scheduler` — single scheduler for periodic tasks.
- [ ] `sentry run consumer <name>` — one or more per Kafka topic (ingest, post-process-forward, snuba writers, replays, outcomes, …).
- [ ] Supporting services: PostgreSQL, Redis, Kafka, ClickHouse + Snuba, Relay, Symbolicator (run by the Compose project).

---

## 5. Configuration Reference

Sentry has **two configuration layers**:

1. **Django settings** in `src/sentry/conf/server.py` (and your `sentry.conf.py`
   overrides) — `SENTRY_*` Python constants and backend class paths. These are
   set at process start.
2. **Runtime options** (`SENTRY_OPTIONS` / the options store) — key/value options
   readable and writable at runtime via `sentry config` and the UI, stored in the
   DB/cache. Many are also pinnable in `config.yml`.

### 5.1 Pluggable backends (set in settings)

The self-hosted defaults wire Sentry's subsystems to Redis/Snuba:

| Setting | Default | Role |
|---|---|---|
| `SENTRY_CACHE` | `sentry.cache.redis.RedisCache` | Processing cache |
| `SENTRY_RATELIMITER` | `sentry.ratelimits.redis.RedisRateLimiter` | Per-project rate limiting |
| `SENTRY_BUFFER` | `sentry.buffer.redis.RedisBuffer` | Write coalescing |
| `SENTRY_QUOTAS` | `sentry.quotas.redis.RedisQuota` | Project/instance quotas |
| `SENTRY_TSDB` | `sentry.tsdb.redissnuba.RedisSnubaTSDB` | Time-series for charts/alerts |
| `SENTRY_DIGESTS` | `sentry.digests.backends.redis.RedisBackend` | Notification digests |
| `SENTRY_SEARCH` | `sentry.search.snuba.EventsDatasetSnubaSearchBackend` | Issue/event search |
| `SENTRY_TAGSTORE` | `sentry.tagstore.snuba.SnubaTagStorage` | Tag storage |
| `SENTRY_FILESTORE` (`filestore.backend`) | `filesystem` (or `s3`) | Attachments/debug files |

### 5.2 Selected environment-driven settings

These are read directly from the environment in `conf/server.py` and elsewhere
(non-exhaustive; grep `os.environ`/`os.getenv` in `conf/server.py` for the full
list):

| Variable | Default | Meaning |
|---|---|---|
| `SENTRY_ENVIRONMENT` | `production` | Deployment environment label |
| `SENTRY_SILO_MODE` | `None` (monolith) | `control` / `region` / monolith |
| `SENTRY_REGION` | `None` | Region name for region silos |
| `SENTRY_CONTROL_ADDRESS` | `None` | Control-silo address for RPC |
| `SENTRY_GRANIAN_IFACE` / `_PORT` / `_WORKERS` / `_RELOAD` | `wsgi` / `0` / `1` / – | Granian server tuning (`services/http.py`) |
| `SNUBA` | `http://127.0.0.1:1218` | Snuba service URL |
| `VROOM` | `http://127.0.0.1:8085` | Profiling (Vroom) service URL |
| `TASKWORKER_SHARED_SECRET`, `TASKWORKER_ROUTES` | – | Taskbroker auth/routing |
| `SENTRY_AI_SUGGESTED_FIX_MODEL` | `gpt-4o-mini` | Model for AI suggested fixes |
| `SENTRY_SSO_EXPIRY_SECONDS` | `None` | SSO session expiry |
| `DATABASE_ENCRYPTION_FERNET_*` | – | At-rest field encryption keys |

### 5.3 The options store (`config.yml` / `sentry config`)

YAML-pinned options use dotted keys. Examples from the shipped `config.yml`:

```yaml
# Mail
mail.backend: 'smtp'
mail.host: 'localhost'
mail.port: 25
mail.from: 'root@localhost'

# File storage
filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'

# System
system.secret-key: '...'   # prefer SENTRY_SECRET_KEY env

# Redis clusters (referenced by name from backends)
redis.clusters:
  default:
    hosts:
      0: {host: 127.0.0.1, port: 6379}
```

Manage options at runtime:

```bash
sentry config list             # list options
sentry config get mail.host    # read one
sentry config set mail.host smtp.acme.com
sentry config delete <option>
sentry configoptions sync -f options.yaml   # bulk apply from a file (automator)
```

---

## 6. Required Secrets & Credentials

Before a production deployment will run cleanly, provision these (env vars or
`config.yml`, ideally from a secrets manager):

| Secret | Required | Used for |
|---|---|---|
| `SENTRY_SECRET_KEY` / `system.secret-key` | **Yes** | Session/token signing (≥ 32 chars). |
| PostgreSQL credentials (`SENTRY_DB_*`) | **Yes** | Database access. |
| Redis credentials (`SENTRY_REDIS_*`) | **Yes** | Cache/buffers/quotas/digests. |
| `DATABASE_ENCRYPTION_FERNET_PRIMARY_KEY_ID` / `_KEYS_LOCATION` | If at-rest field encryption is used | Fernet field encryption. |
| SMTP credentials (`SENTRY_EMAIL_*`) | For outbound email | Notifications/invites. |
| `TASKWORKER_SHARED_SECRET` | For the taskbroker | Authenticating workers to the broker. |
| `WEBHOOK_SIGNING_SECRET`, `SENTRY_SUBNET_SECRET` | For webhooks / internal RPC | HMAC signing. |
| Integration OAuth apps (GitHub/GitLab/Slack/…) client IDs & secrets | Per integration | Third-party integration auth. |
| Object-storage credentials (S3/GCS) | If not using filesystem storage | Attachments, debug files, source maps. |

> [!IMPORTANT]
> Rotating `SENTRY_SECRET_KEY` invalidates every active session. Rotate
> deliberately and communicate the forced re-login.

---

## 7. The Public REST API (operational view)

Base path **`/api/0/`**, served by DRF. The SaaS host is
`https://{region}.sentry.io` (`region` ∈ `us`, `de`); on self-hosted it is your
instance URL.

- **Auth:** `Authorization: Bearer <token>` — organization auth tokens (preferred
  for automation), internal-integration tokens (Sentry Apps), or user tokens.
  Session auth is used by the web UI.
- **Schema:** generated by **drf-spectacular** (`make build-api-docs`); a curated
  public spec is `api-docs/openapi.json` (tags: Teams, Organizations, Projects,
  Events, Releases, Integration, SCIM). The full surface is the 190+ endpoint
  modules under `src/sentry/api/endpoints/` wired in `src/sentry/api/urls.py`.
- **Pagination:** cursor-based via the `Link` header.
- **Rate limiting:** enforced (`sentry.ratelimits`); clients must honor `429` +
  `Retry-After`.

> [!NOTE]
> The AST-extracted `api_endpoints.tsv` for this repo is **not** a reliable
> endpoint list — DRF endpoints are class-based and the extractor instead captured
> `@patch(...)` decorators from the test suite. The authoritative sources are
> `api-docs/openapi.json` (public subset) and `src/sentry/api/urls.py` + the
> `endpoints/` package (complete internal + public surface).

See the User Guide §13 for request examples.

---

## 8. The `sentry` CLI — Complete Reference

The CLI is a Click application (`src/sentry/runner/`, entry `runner/main.py`),
exposed as the `sentry` console script. Global options: `--config <dir>`,
`-l/--loglevel`, `--logformat`. Run `sentry help` for the live list. Command
groups and their members (from `runner/commands/`):

### Server & workers — `sentry run`, `start`, `upgrade`

| Command | Key options | Purpose |
|---|---|---|
| `run web` | `-b/--bind`, `-w/--workers`, `--upgrade`, `--with-lock`, `--noinput` | Web/API server (Granian). |
| `run taskworker` | `--concurrency`, `--namespace`, `--rpc-host`, `--push-mode`, `--max-child-task-count`, `--health-check-file-path` | Background task worker (gRPC taskbroker). |
| `run taskworker-scheduler` | `--redis-cluster` | Schedules periodic tasks. |
| `run consumer <name> [args]` | `--topic`, `--cluster`, `--consumer-group`, `--auto-offset-reset`, `--enable-dlq/--disable-dlq`, `--healthcheck-file-path`, `--strict-offset-reset`, `--max-poll-interval-ms` | Run an Arroyo Kafka consumer. |
| `run dev-consumer <names…>` | – | Fold multiple consumers into one process (dev). |
| `run backpressure-monitor` | – | Monitor consumer backpressure. |
| `run taskbroker-send-tasks` | `--kafka-topic`, `--namespace`, `--repeat`, `--infinite` | Load/test task sending. |
| `start <service>` | `-b/--bind`, `-w/--workers`, `--upgrade`, `--noinput` | Start a named service (production server wrapper). |
| `upgrade` | `--noinput`, `--lock`, `--no-repair`, `--no-post-upgrade`, `--with-nodestore`, `--create-kafka-topics`, `-v/--verbosity` | **Run DB migrations + bootstrap.** Run after every version change. |
| `repair` | – | Fix/repair common install issues. |
| `migrations run <app> <name>` | – | Run a specific data migration. |
| `devserver` / `devservices` | (dev) | Local all-in-one dev server / manage dev service containers. |

### Identity & access

| Command | Purpose |
|---|---|
| `createuser` | Create an account (`--email`, `--password`, `--superuser/--no-superuser`, `--staff/--no-staff`, `--org-id`, `--no-password`, `--no-input`, `--force-update`). |
| `createorg` | Create an organization (`--name`, `--slug`, `--owner-email`, `--no-default-team`). |
| `createproject` | Create a project (`--name`, `--platform`, `--organization`, `--team`). |
| `permissions add/remove/list` | Manage superuser permissions (`-u/--user`, `-p/--permission`). |

### Configuration

| Command | Purpose |
|---|---|
| `config list/get/set/delete` | Read/write runtime options (`--secret`, `--silent`). |
| `config dump` | Dump options (`--flags`, `--only-set`, `--pretty-print`). |
| `config generate-secret-key` | Generate a secret key. |
| `config discover` | Discover configuration. |
| `configoptions patch/sync` | Bulk option management for the automator (`--dry-run`, `-f/--file`, `--hide-drift`). |
| `createflag` / `createissueflag` | Scaffold feature-flag / issue-type-flag definitions. |

### Backup, import & export (relocation)

| Command | Purpose |
|---|---|
| `export users/organizations/config/global <dest>` | Export data as JSON (`--encrypt-with[-gcp-kms]`, `--filter-usernames`, `--filter-org-slugs`, `--indent`). |
| `import users/organizations/config/global <src>` | Import data (`--decrypt-with[-gcp-kms]`, `--merge-users`, `--overwrite-configs`, `--filter-*`). |
| `backup compare/encrypt/decrypt/sanitize` | Compare/transform backup tarballs and findings files. |

### Data lifecycle & operations

| Command | Purpose |
|---|---|
| `cleanup` | Delete data older than `--days` (default 30); `--project`, `--organization`, `--concurrency`, `--model`, `--router`, partitioned deletion via `--partition-bucket/--partition-total/--partition-key`. |
| `deletions list/run` | Inspect and run scheduled deletions (`-m/--model`, `-i/--id`, `--all`). |
| `files get/info <id>` | Inspect stored files. |
| `tsdb query …` | Query the time-series DB (e.g. `tsdb query organizations`). |
| `spans write-hashes <file>` | Span hashing utility. |

### Diagnostics, notifications & developer tools

| Command | Purpose |
|---|---|
| `sendmail <file>` | Send a test email (`--fail-silently`). |
| `notifications send email/slack/discord/msteams` | Send a test notification from a registered template source (`-s/--source`, `-o/--organization_slug`, …). |
| `notifications list registry/integrations` | List notification template sources/integrations. |
| `performance detect <path>` / `timeit <file>` | Run performance-issue detectors on a stored event (`-d/--detector`, `-v`). |
| `rpcschema` | Generate/validate the silo RPC schema (`--partial`, `--diagnose`). |
| `plugins list` | List installed plugins. |
| `exec` / `execfile` / `django` / `help` | Run a script in the app context / run Django management commands / list commands. |

> [!TIP]
> On the self-hosted Compose project, run CLI commands with
> `docker compose run --rm web sentry <command>` so they execute inside the app
> environment with the right config and network access.

---

## 9. Operations

### 9.1 Upgrades

1. Stop or drain workers/consumers (web can stay up briefly).
2. Pull the new image / `git pull` the self-hosted project.
3. Run **`sentry upgrade`** (applies Django migrations, bootstraps nodestore,
   creates Kafka topics with `--create-kafka-topics`).
4. Restart web, taskworkers, scheduler, and consumers.

The self-hosted project provides an `install.sh` that performs the upgrade steps
in order; prefer it for self-hosted.

> [!WARNING]
> Migrations must complete before the new application code serves traffic. Running
> a new web image against an un-migrated database will error. Always
> `sentry upgrade` first.

### 9.2 Backups & relocation

Use the `export`/`import` command families (§8). For full-fidelity moves, encrypt
with `--encrypt-with` (or GCP KMS) and keep the findings file. `sentry backup
compare` validates two exports; `sentry backup sanitize` scrubs sensitive fields
for sharing.

### 9.3 Data retention & cleanup

- `sentry cleanup --days N` prunes old rows in PostgreSQL/nodestore; use
  `--concurrency` and partitioned deletion for large installs.
- Event storage in **ClickHouse/Snuba** is governed by its own TTL/retention —
  configure retention there for the bulk of event volume.
- The `_SENTRY_CLEANUP` env flag and `--silent` support unattended runs.

### 9.4 Monitoring the platform itself

Watch:

- **Kafka consumer lag** and DLQ topics (`--enable-dlq`). Growing lag = ingestion
  falling behind.
- **Snuba/ClickHouse** health (search/stats depend on it).
- **Taskworker** throughput and queue depth; the scheduler must be running for
  periodic jobs.
- **Redis** memory (buffers/quotas/digests live here).
- Granian web worker saturation.
- The built-in health surface and `backpressure-monitor` for self-protection.

### 9.5 Performance tuning

| Lever | Where |
|---|---|
| Web concurrency | `SENTRY_GRANIAN_WORKERS`, `run web --workers` |
| Task concurrency | `run taskworker --concurrency`, replica count |
| Consumer throughput | per-consumer `--cluster`, parallelism, `--max-poll-interval-ms` |
| Ingestion sampling | **Dynamic sampling** + SDK `traces_sample_rate`; inbound filters in Relay |
| Cache | enable **memcached** in addition to Redis |
| Quotas | `SENTRY_QUOTAS` (Redis) + per-project rate limits and spike protection |
| Storage | move `filestore` to S3/GCS; tune ClickHouse retention |

---

## 10. Security

### 10.1 Authentication & authorization

- **Accounts:** username/email + password, with optional **2FA** (TOTP, WebAuthn/
  U2F via `fido2`/`python-u2flib-server`).
- **SSO/SAML:** `src/sentry/auth/providers/` ships SAML2 providers (Okta,
  OneLogin, Rippling, Auth0, generic), plus Google and GitHub auth. SSO session
  expiry via `SENTRY_SSO_EXPIRY_SECONDS`. SCIM endpoints provision users/groups.
- **Superuser & staff:** elevated access is gated by the superuser/staff flags and
  granular **permissions** (`sentry permissions`). `_SENTRY_SKIP_CONFIGURATION`
  and staff access map to the Django admin backend.
- **API tokens:** organization auth tokens, user tokens, and internal-integration
  tokens, each scope-limited. Tokens are replicated control→region via the outbox
  mechanism.

### 10.2 Secrets & encryption

- `SENTRY_SECRET_KEY` (≥ 32 chars) signs sessions and tokens; rotating it
  invalidates all sessions. The self-hosted config refuses to boot without it and
  warns if it is too short.
- Field-level at-rest encryption uses **Fernet** keys
  (`DATABASE_ENCRYPTION_FERNET_PRIMARY_KEY_ID` / `_KEYS_LOCATION`).
- Webhooks and internal RPC are HMAC-signed (`WEBHOOK_SIGNING_SECRET`,
  `SENTRY_SUBNET_SECRET`, taskworker `TASKWORKER_SHARED_SECRET`).

### 10.3 PII & data scrubbing

Defense-in-depth, two layers:

1. **Client-side** — SDK `send_default_pii=False`, `before_send` hooks, and
   per-SDK scrubbing.
2. **Server-side** — **Relay** applies data-scrubbing rules and **Enhanced
   Privacy** before storage; project **Security & Privacy** settings define
   scrubbing rules, sensitive-field lists, and IP-address storage policy.

> [!CAUTION]
> Inconsistent email scrubbing under Enhanced Privacy has been reported
> ([#110838](https://github.com/getsentry/sentry/issues/110838)). Do not rely on a
> single scrubbing layer for regulated data — configure both SDK-side and
> server-side rules and verify with test events.

### 10.4 Network & transport

- Put Sentry behind TLS termination and set the proxy SSL header + secure-cookie
  settings (§4.4).
- **CSP** is supported via `django-csp`. CSRF protection is Django's, with secure
  cookies recommended.
- Restrict Kafka, ClickHouse, PostgreSQL, and Redis to the internal network; only
  Relay's ingest endpoint and the web port need external exposure.

### 10.5 Licensing & compliance

Sentry is under **FSL-1.1-Apache-2.0** (`LICENSE.md`): source-available, free to
self-host and modify, with a non-compete restriction that lapses two years after
each release (then Apache-2.0). Confirm obligations before redistributing or
offering Sentry-as-a-service.

---

## 11. Developer Internals

### 11.1 Repository layout

```text
src/sentry/          Backend Django project (api, integrations, issues, models, …)
src/sentry/conf/     server.py — canonical settings + SENTRY_OPTIONS
src/sentry/runner/   The `sentry` CLI (commands/)
src/apigw/           SaaS API gateway (region routing)
src/flagpole/        Feature-flag evaluation engine
src/bitfield/, src/sudo/, src/social_auth/, src/django_picklefield/   vendored/support libs
static/app/          React SPA (views/, components/, router/, stores/)
static/gsApp/, static/gsAdmin/   SaaS billing & internal admin UIs
api-docs/            OpenAPI spec + build tooling
tests/               Python + JS + acceptance tests
migrations_lockfile.txt   Guards migration ordering
devservices/, devenv/     Local development service orchestration
self-hosted/         Dockerfile + default config for the self-hosted image
```

### 11.2 Local development

```bash
# Bootstrap the dev environment (devenv + devservices)
make bootstrap            # or: devenv sync
# Run the all-in-one dev server (backend + frontend proxy)
sentry devserver --workers
# Frontend-only hot reload
pnpm dev-ui
```

`devservices` brings up the supporting containers (PostgreSQL, Redis, Kafka,
ClickHouse/Snuba, Relay, Symbolicator) locally. `sentry devserver` runs the web
app, optionally ingest (`--ingest`), workers (`--workers`), and the task
scheduler (`--task-scheduler`).

### 11.3 Adding a REST endpoint

1. Subclass the appropriate `Endpoint` base in `src/sentry/api/endpoints/` (these
   add Sentry's auth, permissions, rate limiting, and pagination on top of DRF's
   `APIView`).
2. Implement `get`/`post`/… returning a `Response`; use serializers from
   `api/serializers/`.
3. Register the route in `src/sentry/api/urls.py`.
4. Add an OpenAPI schema annotation (drf-spectacular) for public endpoints.
5. Add tests under `tests/sentry/api/endpoints/`.

### 11.4 Background tasks

Define tasks with the taskworker decorators (namespaced); they execute under
`sentry run taskworker`. Periodic tasks are registered with the scheduler. The
older Celery-style tasks are being migrated to taskworker — follow the patterns in
`src/sentry/taskworker/` and `src/sentry/tasks/`.

### 11.5 Migrations

- Generate via `sentry django makemigrations` (through the `django` passthrough).
- `migrations_lockfile.txt` enforces a single migration head; conflicts must be
  resolved before merge.
- Zero-downtime patterns are enforced by `django-pg-zero-downtime-migrations`;
  destructive operations are gated by **safe-migration** checks.
- Apply with `sentry upgrade` (production) or `sentry django migrate` (dev).

### 11.6 Testing

| Target | Command |
|---|---|
| Python unit/integration | `make test-python-ci` (pytest) |
| Frontend | `pnpm test` / `make test-js` (Jest) |
| Acceptance (Selenium/Playwright) | `make test-acceptance` |
| API docs validation | `make test-api-docs` |
| Snapshot review | `make review-python-snapshots` / `accept-python-snapshots` |

Type checking: backend via mypy (`tools/mypy_helpers/`), frontend via
`pnpm typecheck` (tsgo). Linting via `eslint`, `oxfmt`, `stylelint` (frontend) and
flake8/ruff-style tooling (backend).

### 11.7 Feature flags

`src/flagpole/` evaluates feature flags from `flagpole.yaml` definitions
(`flagpole_eval` CLI). Use `sentry createflag` to scaffold a flag; features gate
both backend behavior and frontend surfaces.

---

## 12. Troubleshooting (Operations)

### Error: `SENTRY_SECRET_KEY is undefined`

**Cause:** No secret key configured.
**Solution:** `sentry config generate-secret-key`, then set `SENTRY_SECRET_KEY`
(env) or `system.secret-key` (config.yml). Must be ≥ 32 chars.
**Prevention:** Generate and store it in your secrets manager during initial setup.

### Error: `REDIS_PORT_6379_TCP_ADDR (or SENTRY_REDIS_HOST) is undefined`

**Cause:** Redis isn't linked/configured; the self-hosted config raises this hard.
**Solution:** Set `SENTRY_REDIS_HOST` (and port/password/db) or link a `redis`
container.
**Prevention:** Treat Redis as a hard dependency in your topology.

### Issue: search, dashboards, and stats are empty but issues still appear

**Cause:** Snuba/ClickHouse or the snuba-writer consumers are down; PostgreSQL-only
data still flows.
**Solution:** Check `docker compose ps`/logs for `snuba-*` and the relevant
`sentry run consumer` processes; restart and let them catch up on lag.
**Prevention:** Alert on consumer lag and Snuba health.

### Issue: events accepted by Relay but never grouped

**Cause:** Ingest consumers or post-processing tasks aren't running.
**Solution:** Verify `sentry run consumer ingest-events` (and friends) and
`sentry run taskworker` are healthy; inspect DLQ topics for rejected messages.
**Prevention:** Run consumers and workers under a supervisor with health checks
(`--healthcheck-file-path`).

### Issue: migrations fail on upgrade

**Cause:** Skipped versions, an interrupted previous upgrade, or migration-head
conflicts.
**Solution:** Re-run `sentry upgrade` (idempotent); check `migrations_lockfile.txt`
state; review the failing migration in `src/sentry/migrations/`.
**Prevention:** Upgrade sequentially through supported versions; never skip the
`sentry upgrade` step.

### Issue: emails are never delivered

**Cause:** The mail backend defaults to `dummy` until SMTP is configured.
**Solution:** Set `SENTRY_EMAIL_HOST` (and credentials/TLS), then test with
`sentry sendmail <file>`.
**Prevention:** Configure SMTP during install and verify with a test message.

### Issue: native/mobile stack traces are not symbolicated

**Cause:** Symbolicator isn't running, or debug files weren't uploaded.
**Solution:** Confirm the Symbolicator service is up and that dSYM/ProGuard/debug
files are uploaded for the matching release.
**Prevention:** Automate debug-file upload in the build pipeline.

---

## 13. Observations & Potential Improvements

Surfaced from a source-level read; these are observations for operators and
contributors, not defects in your deployment.

- **Operational complexity is the dominant cost.** A correct install requires
  PostgreSQL, Redis, Kafka, ClickHouse/Snuba, Relay, and Symbolicator running in
  concert. The hard dependency on Redis and the silent `dummy` mail backend are
  common first-boot stumbling blocks — front-load these in any internal runbook.
- **The analytics plane is a separate failure domain.** Because issues persist in
  PostgreSQL while search/stats live in Snuba, partial outages are easy to
  misdiagnose. Dashboards that surface Snuba and consumer-lag health pay for
  themselves quickly.
- **Two alerting systems coexist.** The legacy rule-based alerts and the newer
  `workflow_engine` (detectors/conditions/actions) overlap. When documenting
  internal alerting standards, pin which surface your org standardizes on.
- **Celery→taskworker migration is in flight.** Some background work still uses
  legacy task patterns while new code targets taskworker/taskbroker. Contributors
  should follow `taskworker/` for new tasks and avoid adding Celery-style tasks.
- **Pydantic v1 pin.** The `pydantic<2` constraint will eventually need a major
  migration; new code should not lean on v1-only behavior in ways that complicate
  it.
- **Frontend state is multi-paradigm.** Reflux, MobX, and TanStack Query coexist;
  new views should prefer TanStack Query for server state to ease the eventual
  retirement of Reflux.

---

## 14. Quick Reference

### Service start commands

```bash
sentry run web                       # web/API (Granian, :9000)
sentry run taskworker --concurrency 4
sentry run taskworker-scheduler
sentry run consumer ingest-events
sentry upgrade --create-kafka-topics # migrations + bootstrap
```

### Common admin commands

```bash
sentry createuser --email [email removed] --superuser
sentry createorg --name Acme --owner-email [email removed]
sentry config generate-secret-key
sentry config set mail.host smtp.acme.com
sentry cleanup --days 90 --concurrency 4
sentry export global backup.json --encrypt-with key.pub
```

### Key paths & ports

| Item | Value |
|---|---|
| Web UI / API port | `9000` (`SENTRY_WEB_PORT`) |
| Config directory | `$SENTRY_CONF` (default `/etc/sentry`) |
| Config files | `sentry.conf.py`, `config.yml` |
| Data volume | `/data` (files at `/data/files`) |
| API base path | `/api/0/` |
| SaaS API host | `https://{us|de}.sentry.io` |

### Required services checklist

- [ ] PostgreSQL  - [ ] Redis  - [ ] Kafka  - [ ] ClickHouse + Snuba
- [ ] Relay  - [ ] Symbolicator  - [ ] (optional) Memcached, SMTP, S3/GCS

---

_Generated June 2026 from getsentry/sentry at cd5d504. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._