# Daytona Administrator & Developer Guide

Daytona is a cloud development sandbox platform that provisions isolated, reproducible development environments at scale. This guide covers the complete operational surface: architecture, configuration, deployment, security, and internals for administrators managing Daytona infrastructure and developers extending the platform.

Daytona is a polyglot monorepo (Go + TypeScript) managed by Nx 21.4.1 and Yarn 4.13.0, comprising ten distinct services that coordinate to deliver on-demand sandboxes with full lifecycle management, snapshot persistence, multi-region support, and organization-level resource governance.

---

## 1. Architecture Overview

### 1.1 System Topology

Daytona follows a hub-and-spoke architecture. The API server is the central control plane; runners are distributed worker nodes that execute sandbox operations. All other services (proxy, SSH gateway, dashboard, daemon) connect through the API or directly to runners.

```
                                 Internet
                                    |
                       +------------+------------+
                       |                         |
                  Dashboard (React)         CLI (Go/Cobra)
                       |                         |
                       v                         v
                 +---------------------------+
                 |     API Server (NestJS)    |
                 |  - Job orchestration       |
                 |  - Auth (OIDC)             |
                 |  - Organization mgmt       |
                 |  - Runner scoring          |
                 |  - Webhook dispatch        |
                 +---------------------------+
                    |        |           |
           +-------+    +---+---+    +--+--------+
           |             |           |            |
      PostgreSQL      Redis       S3/MinIO    ClickHouse
                                                  |
           +----------------------------------+   |
           |          Runner (Go/Gin)         |   |
           |  - Sandbox CRUD                  |   |
           |  - Image build (BuildKit)        |   |
           |  - Snapshot management           |   |
           +----------------------------------+   |
              |              |            |       |
         +---+---+    +-----+----+  +----+----+  |
         | Proxy |    |SSH Gateway|  | Daemon  |  |
         | (Go)  |    |   (Go)   |  |  (Go)   |  |
         +-------+    +----------+  +---------+  |
                                                  |
                                    OTel Collector
```

### 1.2 Component Responsibilities

| Component | Technology | Port | Purpose |
|-----------|-----------|------|---------|
| **API Server** | NestJS 11, Express 5, TypeScript | 3000 | Central REST API, job orchestration, auth, billing integration, webhook dispatch |
| **CLI** | Go, Cobra | - | User-facing command-line interface for sandbox and organization management |
| **Runner** | Go, Gin | 8080 | Executes sandbox operations on worker nodes; manages images, builds, and backups |
| **Daemon** | Go | - | Runs inside each sandbox; exposes the Toolbox API for file, process, and git operations |
| **Proxy** | Go | - | Routes HTTP/WebSocket traffic to individual sandboxes by subdomain |
| **SSH Gateway** | Go | 22 | Provides SSH access into running sandboxes with key-based authentication |
| **Dashboard** | React 19, Vite, TanStack, Radix UI, Tailwind | 5173 (dev) | Web UI for sandbox management, organization admin, and monitoring |
| **Docs Site** | Astro | - | Public documentation website |
| **Snapshot Manager** | Go | - | Manages sandbox snapshot lifecycle (create, restore, prune) |
| **OTel Collector** | Go | - | Custom OpenTelemetry collector with Daytona-specific exporters |

### 1.3 SDK Ecosystem

Daytona provides first-party SDKs for programmatic sandbox management:

| SDK | Package | Language |
|-----|---------|----------|
| Python | `daytona` | Python 3.8+ |
| TypeScript | `@daytona/sdk` | TypeScript/Node.js |
| Go | `libs/sdk-go` | Go 1.21+ |
| Java | `libs/sdk-java` | Java 11+ |
| Ruby | `libs/sdk-ruby` | Ruby 3.0+ |

Each SDK has a corresponding auto-generated API client (both sync and async variants for Python). The clients are generated from the OpenAPI specification the API server exposes.

### 1.4 Communication Patterns

**API to Runner**: The API server dispatches jobs to runners via a polling mechanism. Runners poll the API at a configurable interval (`POLL_TIMEOUT`, default 30s) for pending jobs up to `POLL_LIMIT` (default 10) at a time.

**Runner to API**: Runners report job status, health metrics (CPU, memory, disk), and sandbox state transitions back to the API.

**Proxy Routing**: The proxy resolves sandbox-specific subdomains to the correct runner and sandbox, forwarding traffic transparently. The proxy domain and protocol are configured via `PROXY_DOMAIN` and `PROXY_PROTOCOL`.

**Webhook Delivery**: Webhook events are dispatched via Svix. Each organization manages its own webhook endpoints. Events cover sandbox lifecycle, snapshot operations, and organization changes.

**Telemetry Pipeline**: Sandbox-level metrics flow through the custom OTel collector, which exports to ClickHouse for analytics and optionally to PostHog for product telemetry.

> [!NOTE]
> The runner uses a pull-based job model, not push. The API never initiates connections to runners. This simplifies firewall rules: runners only need outbound HTTPS to the API server.

---

## 2. System Requirements

### 2.1 API Server Node

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| CPU | 2 cores | 4+ cores |
| Memory | 4 GB | 8+ GB |
| Disk | 20 GB | 50+ GB |
| Network | 100 Mbps | 1 Gbps |
| Node.js | 18+ | 20 LTS |
| OS | Linux (x86_64) | Ubuntu 22.04+ / Debian 12+ |

### 2.2 Runner Node

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| CPU | 4 cores | 8+ cores |
| Memory | 8 GB | 16+ GB |
| Disk | 50 GB SSD | 200+ GB NVMe |
| Network | 1 Gbps | 10 Gbps |
| Container Runtime | Docker 24+ with BuildKit | Docker 25+ |
| OS | Linux (x86_64) | Ubuntu 22.04+ |

Default runner resource allocation per sandbox: 4 CPU cores, 8 GB memory, 50 GB disk (configurable via `DEFAULT_RUNNER_CPU`, `DEFAULT_RUNNER_MEMORY`, `DEFAULT_RUNNER_DISK`).

### 2.3 Infrastructure Dependencies

| Service | Version | Required | Purpose |
|---------|---------|----------|---------|
| **PostgreSQL** | 14+ | Yes | Primary data store (TypeORM, WAL mode) |
| **Redis** | 7+ | Yes | Caching, rate limiting, throttling, pub/sub |
| **S3-compatible storage** | - | Yes | Object storage (MinIO, AWS S3, Cloudflare R2) |
| **OIDC Provider** | - | Yes | Authentication (Dex, Auth0, Keycloak) |
| **Docker Registry** | - | Yes | Transient + internal image registries |
| **Svix** | - | Optional | Webhook delivery |
| **ClickHouse** | - | Optional | Telemetry and analytics storage |
| **Kafka** | - | Optional | Audit event stream publishing |
| **OpenSearch** | - | Optional | Audit log storage and search |
| **PostHog** | - | Optional | Product analytics |
| **SMTP Server** | - | Optional | Email notifications (invitations, alerts) |

> [!IMPORTANT]
> PostgreSQL must run in WAL (Write-Ahead Logging) mode. TypeORM manages the connection pool, but WAL mode must be enabled at the PostgreSQL server level for the concurrent access patterns Daytona's job system generates.

### 2.4 Network Requirements

| Flow | Source | Destination | Port | Protocol |
|------|--------|-------------|------|----------|
| Client to API | Internet/VPN | API Server | 443 (TLS) | HTTPS |
| Client to Dashboard | Internet/VPN | Dashboard | 443 (TLS) | HTTPS |
| Client to Proxy | Internet/VPN | Proxy | 443 (TLS) | HTTPS/WSS |
| Client to SSH Gateway | Internet/VPN | SSH Gateway | 22 | SSH |
| Runner to API | Runner nodes | API Server | 443 | HTTPS |
| API to PostgreSQL | API Server | PostgreSQL | 5432 | TCP (TLS optional) |
| API to Redis | API Server | Redis | 6379 | TCP (TLS optional) |
| API to S3 | API Server | S3 endpoint | 443 | HTTPS |
| Runner to Registry | Runner nodes | Docker registries | 443 | HTTPS |
| API to Svix | API Server | Svix | 443 | HTTPS |
| API to ClickHouse | API Server | ClickHouse | 8443 | HTTPS |

### 2.5 Build Toolchain (Development)

| Tool | Version |
|------|---------|
| Nx | 21.4.1 |
| Yarn | 4.13.0 |
| Go | 1.22+ |
| Node.js | 20+ |
| Docker | 24+ |
| Protocol Buffers | 3.x (for runner-proto) |

---

## 3. Configuration Reference

### 3.1 Core Application

These variables control the API server's fundamental behavior.

| Variable | Default | Description |
|----------|---------|-------------|
| `PORT` | `3000` | HTTP port the API server listens on |
| `APP_URL` | - | Public-facing URL of the API (used for OIDC callbacks, email links) |
| `APP_ROLE` | - | Application role identifier for multi-instance deployments |
| `APP_MODE` | - | Application mode (affects feature flags and behavior) |
| `VERSION` | - | Application version string, injected at build time |
| `RUN_MIGRATIONS` | - | Set to `true` to run database migrations on startup |
| `MAINTENANCE_MODE` | - | Set to `true` to enable maintenance mode (rejects non-health requests) |
| `DISABLE_CRON_JOBS` | - | Set to `true` to disable all scheduled background tasks |
| `ENVIRONMENT` | - | Environment name (e.g., `production`, `staging`); used in tracing spans |
| `CRON_TIMEZONE` | - | IANA timezone for cron job scheduling (e.g., `America/New_York`) |
| `SKIP_CONNECTIONS` | - | Skip startup connection checks (useful for testing) |

> [!WARNING]
> Setting `RUN_MIGRATIONS=true` on every instance in a multi-replica deployment will cause migration race conditions. Designate exactly one instance to run migrations, or use a separate migration job before rolling out new instances.

### 3.2 Database (PostgreSQL)

| Variable | Default | Description |
|----------|---------|-------------|
| `DB_HOST` | - | PostgreSQL hostname or IP |
| `DB_PORT` | `5432` | PostgreSQL port |
| `DB_USERNAME` | - | Database user |
| `DB_PASSWORD` | - | Database password |
| `DB_DATABASE` | - | Database name |
| `DB_TLS_ENABLED` | `false` | Enable TLS for database connections |
| `DB_TLS_REJECT_UNAUTHORIZED` | `true` | Reject connections with untrusted TLS certificates |
| `DB_POOL_MAX` | - | Maximum number of connections in the pool |
| `DB_POOL_MIN` | - | Minimum number of connections maintained |
| `DB_POOL_IDLE_TIMEOUT_MS` | - | Time (ms) before idle connections are released |
| `DB_POOL_CONNECTION_TIMEOUT_MS` | - | Maximum wait time (ms) for acquiring a connection |

**Connection pool tuning guidelines:**

- `DB_POOL_MAX` should not exceed PostgreSQL's `max_connections` divided by the number of API replicas.
- For a single-replica deployment handling moderate load, `DB_POOL_MAX=20` and `DB_POOL_MIN=5` is a reasonable starting point.
- Set `DB_POOL_IDLE_TIMEOUT_MS` to `30000` (30s) to reclaim idle connections without excessive reconnection overhead.
- `DB_POOL_CONNECTION_TIMEOUT_MS` of `5000` (5s) surfaces connection exhaustion quickly rather than queueing indefinitely.

### 3.3 Redis

| Variable | Default | Description |
|----------|---------|-------------|
| `REDIS_HOST` | - | Redis hostname or IP |
| `REDIS_PORT` | `6379` | Redis port |
| `REDIS_USERNAME` | - | Redis ACL username (Redis 6+) |
| `REDIS_PASSWORD` | - | Redis password |
| `REDIS_TLS` | `false` | Enable TLS for Redis connections |

Redis is used for three distinct purposes: response caching, rate limiting counters, and activity throttling. A single Redis instance handles all three. For high-availability deployments, Redis Sentinel or Redis Cluster can be placed behind the same host/port configuration.

### 3.4 S3 / Object Storage

| Variable | Default | Description |
|----------|---------|-------------|
| `S3_ENDPOINT` | - | S3-compatible endpoint URL |
| `S3_STS_ENDPOINT` | - | STS endpoint for temporary credential generation |
| `S3_REGION` | - | S3 region (e.g., `us-east-1`) |
| `S3_ACCESS_KEY` | - | Access key ID |
| `S3_SECRET_KEY` | - | Secret access key |
| `S3_DEFAULT_BUCKET` | - | Default bucket for sandbox artifacts |
| `S3_ACCOUNT_ID` | - | Account ID (required for Cloudflare R2) |
| `S3_ROLE_NAME` | - | IAM role name for STS AssumeRole |

**Storage backend compatibility:**

| Provider | Endpoint Format | Notes |
|----------|----------------|-------|
| AWS S3 | `https://s3.{region}.amazonaws.com` | Native; STS supported |
| MinIO | `https://minio.example.com:9000` | Self-hosted; set `S3_REGION` to any valid string |
| Cloudflare R2 | `https://{account_id}.r2.cloudflarestorage.com` | Requires `S3_ACCOUNT_ID` |
| DigitalOcean Spaces | `https://{region}.digitaloceanspaces.com` | S3-compatible |

### 3.5 OIDC / Authentication

| Variable | Default | Description |
|----------|---------|-------------|
| `OIDC_CLIENT_ID` | - | OIDC client ID (alias: `OID_CLIENT_ID`) |
| `OIDC_ISSUER_BASE_URL` | - | OIDC issuer URL (alias: `OID_ISSUER_BASE_URL`) |
| `PUBLIC_OIDC_DOMAIN` | - | Public-facing OIDC domain |
| `OIDC_AUDIENCE` | - | OIDC audience claim (alias: `OID_AUDIENCE`) |
| `OIDC_MANAGEMENT_API_ENABLED` | - | Enable OIDC management API integration |
| `OIDC_MANAGEMENT_API_CLIENT_ID` | - | Management API client ID |
| `OIDC_MANAGEMENT_API_CLIENT_SECRET` | - | Management API client secret |
| `OIDC_MANAGEMENT_API_AUDIENCE` | - | Management API audience |
| `SKIP_USER_EMAIL_VERIFICATION` | - | Bypass email verification requirement |

> [!TIP]
> Both `OIDC_*` and `OID_*` prefixes are accepted for client ID, issuer URL, and audience. This dual-prefix support exists for backward compatibility. Prefer the `OIDC_*` variants in new deployments.

### 3.6 SMTP

| Variable | Default | Description |
|----------|---------|-------------|
| `SMTP_HOST` | - | SMTP server hostname |
| `SMTP_PORT` | `587` | SMTP port (587 for STARTTLS, 465 for implicit TLS) |
| `SMTP_USER` | - | SMTP authentication username |
| `SMTP_PASSWORD` | - | SMTP authentication password |
| `SMTP_SECURE` | - | Use implicit TLS (port 465) |
| `SMTP_EMAIL_FROM` | - | Sender address for all outgoing email |

SMTP is used for organization invitations and email notifications. If SMTP is not configured, invitation flows and email-dependent features are silently disabled.

### 3.7 Docker Registries

Daytona uses four registry types: **internal** (platform images), **transient** (ephemeral build artifacts), **organization** (user-managed), and **backup** (snapshot storage).

| Variable | Default | Description |
|----------|---------|-------------|
| `TRANSIENT_REGISTRY_URL` | - | Transient registry URL |
| `TRANSIENT_REGISTRY_ADMIN` | - | Transient registry admin username |
| `TRANSIENT_REGISTRY_PASSWORD` | - | Transient registry admin password |
| `TRANSIENT_REGISTRY_PROJECT_ID` | - | Transient registry project identifier |
| `INTERNAL_REGISTRY_URL` | - | Internal registry URL |
| `INTERNAL_REGISTRY_ADMIN` | - | Internal registry admin username |
| `INTERNAL_REGISTRY_PASSWORD` | - | Internal registry admin password |
| `INTERNAL_REGISTRY_PROJECT_ID` | - | Internal registry project identifier |

### 3.8 Proxy

| Variable | Default | Description |
|----------|---------|-------------|
| `PROXY_DOMAIN` | - | Base domain for sandbox proxy routing (e.g., `sandboxes.example.com`) |
| `PROXY_PROTOCOL` | - | Protocol for proxy URLs (`http` or `https`) |
| `PROXY_API_KEY` | - | API key for proxy-to-API authentication |
| `PROXY_TEMPLATE_URL` | - | URL template for constructing sandbox proxy URLs |
| `PROXY_TOOLBOX_BASE_URL` | - | Base URL the proxy uses to reach sandbox Toolbox APIs |

### 3.9 SSH Gateway

| Variable | Default | Description |
|----------|---------|-------------|
| `SSH_GATEWAY_API_KEY` | - | API key for SSH gateway authentication with the API server |
| `SSH_GATEWAY_COMMAND` | - | Default command executed on SSH connection |
| `SSH_GATEWAY_PUBLIC_KEY` | - | SSH gateway host public key |
| `SSH_GATEWAY_URL` | - | Public URL of the SSH gateway |

### 3.10 Webhooks (Svix)

| Variable | Default | Description |
|----------|---------|-------------|
| `SVIX_AUTH_TOKEN` | - | Svix authentication token |
| `SVIX_SERVER_URL` | - | Svix server URL |

Webhook endpoints are managed per-organization. When both variables are configured, the API server registers webhook event types with Svix and dispatches events for sandbox lifecycle transitions, snapshot operations, and organization changes.

### 3.11 Rate Limiting

Daytona implements five rate-limiting tiers, each with a TTL (time window in seconds) and a request limit within that window:

| Tier | TTL Variable | Limit Variable | Purpose |
|------|-------------|----------------|---------|
| Anonymous | `RATE_LIMIT_ANONYMOUS_TTL` | `RATE_LIMIT_ANONYMOUS_LIMIT` | Unauthenticated requests |
| Failed Auth | `RATE_LIMIT_FAILED_AUTH_TTL` | `RATE_LIMIT_FAILED_AUTH_LIMIT` | Failed authentication attempts |
| Authenticated | `RATE_LIMIT_AUTHENTICATED_TTL` | `RATE_LIMIT_AUTHENTICATED_LIMIT` | Authenticated general requests |
| Sandbox Create | `RATE_LIMIT_SANDBOX_CREATE_TTL` | `RATE_LIMIT_SANDBOX_CREATE_LIMIT` | Sandbox creation requests |
| Sandbox Lifecycle | `RATE_LIMIT_SANDBOX_LIFECYCLE_TTL` | `RATE_LIMIT_SANDBOX_LIFECYCLE_LIMIT` | Start/stop/destroy operations |

Rate limit state is stored in Redis. The `Failed Auth` tier is particularly important: it defends against credential-stuffing attacks by throttling repeated authentication failures per source IP.

### 3.12 Organization Quotas

Default quotas applied to new organizations. These can be overridden per-organization through the admin API.

| Variable | Default | Description |
|----------|---------|-------------|
| `DEFAULT_ORG_QUOTA_TOTAL_CPU_QUOTA` | `10` | Total CPU cores across all sandboxes |
| `DEFAULT_ORG_QUOTA_TOTAL_MEMORY_QUOTA` | - | Total memory (GB) across all sandboxes |
| `DEFAULT_ORG_QUOTA_TOTAL_DISK_QUOTA` | - | Total disk (GB) across all sandboxes |
| `DEFAULT_ORG_QUOTA_MAX_CPU_PER_SANDBOX` | - | Maximum CPU cores per individual sandbox |
| `DEFAULT_ORG_QUOTA_MAX_MEMORY_PER_SANDBOX` | - | Maximum memory (GB) per individual sandbox |
| `DEFAULT_ORG_QUOTA_MAX_DISK_PER_SANDBOX` | - | Maximum disk (GB) per individual sandbox |
| `DEFAULT_ORG_QUOTA_SNAPSHOT_QUOTA` | - | Maximum number of snapshots |
| `DEFAULT_ORG_QUOTA_MAX_SNAPSHOT_SIZE` | - | Maximum size per snapshot |
| `DEFAULT_ORG_QUOTA_VOLUME_QUOTA` | - | Maximum number of persistent volumes |

### 3.13 Admin Overrides

Admin quotas override organization-level limits for administrator accounts.

| Variable | Default | Description |
|----------|---------|-------------|
| `ADMIN_API_KEY` | - | API key for admin endpoints |
| `ADMIN_TOTAL_CPU_QUOTA` | - | Admin CPU quota |
| `ADMIN_TOTAL_MEMORY_QUOTA` | - | Admin memory quota |
| `ADMIN_TOTAL_DISK_QUOTA` | - | Admin disk quota |
| `ADMIN_MAX_CPU_PER_SANDBOX` | - | Admin max CPU per sandbox |
| `ADMIN_MAX_MEMORY_PER_SANDBOX` | - | Admin max memory per sandbox |
| `ADMIN_MAX_DISK_PER_SANDBOX` | - | Admin max disk per sandbox |
| `ADMIN_SNAPSHOT_QUOTA` | - | Admin snapshot quota |
| `ADMIN_MAX_SNAPSHOT_SIZE` | - | Admin max snapshot size |
| `ADMIN_VOLUME_QUOTA` | - | Admin volume quota |

### 3.14 Regions

| Variable | Default | Description |
|----------|---------|-------------|
| `DEFAULT_REGION_ID` | - | Identifier for the default region |
| `DEFAULT_REGION_NAME` | - | Human-readable name for the default region |
| `DEFAULT_REGION_ENFORCE_QUOTAS` | - | Whether to enforce resource quotas in the default region |

### 3.15 Runner Defaults (API-side)

These configure the default runner that the API server expects on initial setup.

| Variable | Default | Description |
|----------|---------|-------------|
| `DEFAULT_RUNNER_DOMAIN` | - | Default runner domain |
| `DEFAULT_RUNNER_API_KEY` | - | Default runner authentication key |
| `DEFAULT_RUNNER_PROXY_URL` | - | Proxy URL for the default runner |
| `DEFAULT_RUNNER_API_URL` | - | API URL for the default runner |
| `DEFAULT_RUNNER_CPU` | `4` | CPU cores allocated per sandbox |
| `DEFAULT_RUNNER_MEMORY` | `8` | Memory (GB) allocated per sandbox |
| `DEFAULT_RUNNER_DISK` | `50` | Disk (GB) allocated per sandbox |
| `DEFAULT_RUNNER_API_VERSION` | - | Runner API version expected |
| `DEFAULT_RUNNER_NAME` | - | Display name for the default runner |
| `BUILD_INFO_MAX_SANDBOXES_PER_RUNNER` | - | Maximum concurrent sandboxes per runner |

### 3.16 Audit and Logging

| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_LEVEL` | - | Log level (`debug`, `info`, `warn`, `error`) |
| `LOG_CONSOLE_DISABLED` | - | Disable console log output |
| `LOG_REQUESTS_ENABLED` | - | Log all HTTP request/response pairs |
| `AUDIT_TOOLBOX_REQUESTS_ENABLED` | - | Audit Toolbox API requests from sandboxes |
| `AUDIT_LOG_RETENTION_DAYS` | - | Days to retain audit log entries |
| `AUDIT_CONSOLE_LOG_ENABLED` | - | Output audit events to console |
| `AUDIT_PUBLISH_ENABLED` | - | Enable audit event publishing to external systems |
| `AUDIT_PUBLISH_BATCH_SIZE` | - | Number of events per publish batch |
| `AUDIT_PUBLISH_MODE` | - | Publish mode (sync/async) |
| `AUDIT_PUBLISH_STORAGE_ADAPTER` | - | Storage adapter: `kafka` or `opensearch` |
| `AUDIT_PUBLISH_OPENSEARCH_INDEX_NAME` | - | OpenSearch index name for audit events |

### 3.17 Kafka

| Variable | Default | Description |
|----------|---------|-------------|
| `KAFKA_ENABLED` | - | Enable Kafka integration |
| `KAFKA_BROKERS` | - | Comma-separated broker addresses |
| `KAFKA_CLIENT_ID` | - | Kafka client identifier |
| `KAFKA_SASL_MECHANISM` | - | SASL mechanism (`plain`, `scram-sha-256`, `scram-sha-512`) |
| `KAFKA_SASL_USERNAME` | - | SASL username |
| `KAFKA_SASL_PASSWORD` | - | SASL password |
| `KAFKA_TLS_ENABLED` | - | Enable TLS for Kafka connections |
| `KAFKA_TLS_REJECT_UNAUTHORIZED` | - | Reject untrusted TLS certificates |

### 3.18 OpenSearch

| Variable | Default | Description |
|----------|---------|-------------|
| `OPENSEARCH_NODES` | - | Comma-separated node URLs |
| `OPENSEARCH_USERNAME` | - | OpenSearch username |
| `OPENSEARCH_PASSWORD` | - | OpenSearch password |
| `OPENSEARCH_AWS_ROLE_ARN` | - | AWS IAM role ARN (for AWS-managed OpenSearch) |
| `OPENSEARCH_AWS_REGION` | - | AWS region for OpenSearch |
| `OPENSEARCH_TLS_REJECT_UNAUTHORIZED` | - | Reject untrusted TLS certificates |

### 3.19 Telemetry

| Variable | Default | Description |
|----------|---------|-------------|
| `OTEL_ENABLED` | - | Enable OpenTelemetry instrumentation |
| `OTEL_COLLECTOR_ENDPOINT_URL` | - | OTel collector gRPC endpoint (alias: `SANDBOX_OTEL_ENDPOINT_URL`) |
| `OTEL_COLLECTOR_API_KEY` | - | OTel collector authentication key |
| `POSTHOG_HOST` | - | PostHog instance URL |
| `POSTHOG_API_KEY` | - | PostHog project API key |
| `POSTHOG_ENVIRONMENT` | - | PostHog environment tag |

### 3.20 ClickHouse

| Variable | Default | Description |
|----------|---------|-------------|
| `CLICKHOUSE_HOST` | - | ClickHouse server hostname |
| `CLICKHOUSE_PORT` | - | ClickHouse HTTP port |
| `CLICKHOUSE_DATABASE` | - | Database name |
| `CLICKHOUSE_USERNAME` | - | Username |
| `CLICKHOUSE_PASSWORD` | - | Password |
| `CLICKHOUSE_PROTOCOL` | - | Protocol (`http` or `https`) |

### 3.21 Encryption and Security

| Variable | Default | Description |
|----------|---------|-------------|
| `ENCRYPTION_KEY` | - | Symmetric key for encrypting sensitive data at rest |
| `ENCRYPTION_SALT` | - | Salt for key derivation |
| `CERT_PATH` | - | Path to TLS certificate file |
| `CERT_KEY_PATH` | - | Path to TLS private key file |
| `HEALTH_CHECK_API_KEY` | - | API key required for health check endpoints |

### 3.22 Caching

| Variable | Default | Description |
|----------|---------|-------------|
| `API_KEY_VALIDATION_CACHE_TTL_SECONDS` | - | TTL for API key validation cache entries |
| `API_KEY_USER_CACHE_TTL_SECONDS` | - | TTL for API key to user mapping cache |

### 3.23 Sandbox Behavior

| Variable | Default | Description |
|----------|---------|-------------|
| `DEFAULT_SNAPSHOT` | - | Default sandbox snapshot/image to use |
| `MAX_AUTO_ARCHIVE_INTERVAL` | - | Maximum interval before auto-archiving idle sandboxes |
| `WARM_POOL_CANDIDATE_LIMIT` | - | Number of warm (pre-provisioned) sandboxes to maintain |
| `SANDBOX_ACTIVITY_THROTTLE_TTL_SECONDS` | - | Throttle window for sandbox activity reporting |
| `SANDBOX_ACTIVITY_FLUSH_BATCH_SIZE` | - | Batch size for flushing activity events |
| `FAILED_SNAPSHOT_RUNNER_RETENTION_HOURS` | - | Hours to retain failed snapshot data on runners |
| `BUILDINFO_SNAPSHOT_RUNNER_STALENESS_DAYS` | - | Days before snapshot build info is considered stale |
| `MAX_CONCURRENT_BACKUPS_PER_RUNNER` | - | Maximum concurrent backup operations per runner |
| `ORGANIZATION_SANDBOX_DEFAULT_LIMITED_NETWORK_EGRESS` | - | Default network egress policy for organization sandboxes |

### 3.24 Dashboard and UI

| Variable | Default | Description |
|----------|---------|-------------|
| `DASHBOARD_URL` | - | Public URL of the dashboard |
| `DASHBOARD_BASE_API_URL` | - | API URL the dashboard uses for backend calls |
| `NOTIFICATION_GATEWAY_DISABLED` | - | Disable the real-time notification gateway |

### 3.25 External Integrations

| Variable | Default | Description |
|----------|---------|-------------|
| `PYLON_APP_ID` | - | Pylon application ID (support integration) |
| `BILLING_API_URL` | - | External billing service URL |
| `ANALYTICS_API_URL` | - | External analytics service URL |

### 3.26 Runner Service Configuration

These variables are set on the runner nodes themselves (not the API server).

| Variable | Default | Required | Description |
|----------|---------|----------|-------------|
| `DAYTONA_API_URL` | - | Yes | API server URL the runner connects to |
| `DAYTONA_RUNNER_TOKEN` | - | Yes | Authentication token for runner-to-API communication |
| `SERVER_URL` | - | No | Runner's own externally reachable URL |
| `API_TOKEN` | - | No | Runner's local API token |
| `API_PORT` | `8080` | No | Runner's HTTP API port |
| `BUILD_ENGINE` | `buildkit` | No | Container build engine |
| `API_VERSION` | `2` | No | Runner API protocol version |
| `POLL_TIMEOUT` | `30s` | No | Interval between job polls to the API |
| `POLL_LIMIT` | `10` | No | Maximum jobs fetched per poll cycle |
| `BUILD_CPU_CORES` | `4` | No | CPU cores allocated for image builds |
| `BUILD_MEMORY_GB` | `8` | No | Memory allocated for image builds |

---

## 4. Deployment & Scaling

### 4.1 Deployment Architecture

A production Daytona deployment consists of three tiers:

1. **Control Plane**: API server instance(s), PostgreSQL, Redis, S3
2. **Worker Plane**: Runner nodes (one or more, horizontally scalable)
3. **Edge Plane**: Proxy, SSH gateway, dashboard (stateless, horizontally scalable)

```
                    Load Balancer (L7)
                    /        |         \
               API (x3)   Proxy (x2)  SSH GW (x2)
                 |           |            |
            PostgreSQL    Redis        (stateless)
                 |
                S3
                 |
        +--------+--------+--------+
        |        |        |        |
    Runner-1  Runner-2  Runner-3  Runner-N
```

### 4.2 Database Migrations: Expand and Contract

Daytona uses the **Expand and Contract** migration pattern for zero-downtime deployments. This is a critical operational detail.

#### Phase 1: Pre-deploy (Expand)

Pre-deploy migrations are strictly **additive**. They only add new columns, new tables, or new indexes. They never remove, rename, or alter existing columns. This means the old application code continues to function against the new schema.

```bash
# Run pre-deploy migrations before deploying new code
RUN_MIGRATIONS=true node dist/main.js --migrate-only
```

#### Phase 2: Deploy

Roll out the new application code. Both old and new code versions work against the expanded schema simultaneously during the rolling update.

#### Phase 3: Post-deploy (Contract)

After all instances are running the new code, run post-deploy migrations to clean up: remove deprecated columns, drop old tables, tighten constraints.

```bash
# Run post-deploy migrations after all instances are updated
RUN_MIGRATIONS=true APP_MODE=post-deploy node dist/main.js --migrate-only
```

> [!CAUTION]
> Never skip the expand phase or run contract migrations while old code is still serving traffic. Removing columns that old code depends on will cause immediate 500 errors across all affected endpoints.

### 4.3 Multi-Instance API Deployment

The API server supports horizontal scaling with these considerations:

**Stateless request handling**: All session and cache state lives in Redis. Any API instance can handle any request.

**Migration coordination**: Set `RUN_MIGRATIONS=true` on exactly one instance (or run migrations as a separate init container/job). All other instances should have `RUN_MIGRATIONS` unset or `false`.

**Cron job deduplication**: Set `DISABLE_CRON_JOBS=true` on all but one instance to prevent duplicate scheduled task execution. Alternatively, use `APP_ROLE` to designate which instance runs cron jobs.

**Maintenance mode**: Setting `MAINTENANCE_MODE=true` on any instance causes it to reject non-health-check traffic, useful for draining during rolling updates.

### 4.4 Runner Scaling

Runners scale horizontally. Each runner:

1. Registers with the API server using its `DAYTONA_RUNNER_TOKEN`
2. Polls the API for pending jobs
3. Reports health metrics (CPU, memory, disk utilization)
4. Executes sandbox operations locally

**Adding a runner:**

```bash
# On the new runner node
export DAYTONA_API_URL=https://api.daytona.example.com
export DAYTONA_RUNNER_TOKEN=<token-from-api>
export API_PORT=8080
export BUILD_ENGINE=buildkit

./daytona-runner
```

The API server's runner scoring system automatically distributes workload across available runners based on their reported resource utilization.

**Removing a runner**: Stop the runner process. The API server detects the runner as unhealthy after `RUNNER_HEALTH_TIMEOUT_SECONDS` and stops routing new jobs to it. Existing sandboxes on the runner continue to function but cannot be managed until the runner returns or sandboxes are recovered.

### 4.5 Warm Pool

The warm pool pre-provisions sandboxes to reduce cold-start latency. Configure `WARM_POOL_CANDIDATE_LIMIT` to set the number of pre-warmed sandboxes maintained. The API server's cron system replenishes the pool as sandboxes are claimed.

### 4.6 Auto-Archive

Idle sandboxes are automatically archived after `MAX_AUTO_ARCHIVE_INTERVAL`. Archived sandboxes retain their state in S3 but release runner resources. They can be restored on demand, though restoration incurs a cold-start penalty.

### 4.7 Backup and Recovery

Each runner supports concurrent backup operations, limited by `MAX_CONCURRENT_BACKUPS_PER_RUNNER`. Backups are stored in the backup registry.

The `RECOVER_SANDBOX` job type handles sandbox recovery. The runner exposes an `is-recoverable` endpoint to check whether a sandbox's state can be recovered before attempting the operation.

---

## 5. Operations

### 5.1 Sandbox Lifecycle State Machine

Every sandbox transitions through a well-defined state machine:

```
                                 +---> error
                                 |
  pending_build --> building_snapshot --> pulling_snapshot --> creating
                                                                |
                                                                v
                                          +---------- started <---+
                                          |              |        |
                                          v              v        |
                                       stopped      (running)    |
                                          |                       |
                                          v                       |
                                       archived                   |
                                          |                       |
                                          +----> restoring -------+
                                          |
                                          v
                                       destroying --> destroyed
```

**Primary states:**

| State | Description |
|-------|-------------|
| `creating` | Sandbox is being provisioned on a runner |
| `started` | Sandbox is running and accessible |
| `stopped` | Sandbox is stopped but retains local state on the runner |
| `archived` | Sandbox state is persisted to S3; runner resources released |
| `destroyed` | Sandbox is permanently deleted |
| `error` | Terminal error state |
| `build_failed` | Image build failed |

**Transitional states:**

| State | Description |
|-------|-------------|
| `pending_build` | Waiting for image build to start |
| `building_snapshot` | Building a snapshot image |
| `pulling_snapshot` | Pulling a snapshot image to the runner |
| `resizing` | Changing sandbox resource allocation |
| `snapshotting` | Creating a snapshot of the running sandbox |
| `forking` | Creating a copy of an existing sandbox |
| `restoring` | Restoring an archived sandbox |
| `destroying` | Destruction in progress |
| `starting` | Start operation in progress |
| `stopping` | Stop operation in progress |
| `archiving` | Archive operation in progress |
| `unknown` | State cannot be determined |

### 5.2 Job System

The job system is the mechanism by which the API server dispatches work to runners.

**Job types:**

| Job Type | Resource Type | Description |
|----------|--------------|-------------|
| `CREATE_SANDBOX` | SANDBOX | Provision a new sandbox |
| `START_SANDBOX` | SANDBOX | Start a stopped sandbox |
| `STOP_SANDBOX` | SANDBOX | Stop a running sandbox |
| `DESTROY_SANDBOX` | SANDBOX | Permanently delete a sandbox |
| `RESIZE_SANDBOX` | SANDBOX | Change resource allocation |
| `RECOVER_SANDBOX` | SANDBOX | Recover a sandbox from error state |
| `FORK_SANDBOX` | SANDBOX | Clone an existing sandbox |
| `SNAPSHOT_SANDBOX` | SANDBOX | Create a point-in-time snapshot |
| `PULL_SNAPSHOT` | SNAPSHOT | Pull a snapshot image to a runner |
| `BUILD_SNAPSHOT` | SNAPSHOT | Build a snapshot from a Dockerfile |
| `REMOVE_SNAPSHOT` | SNAPSHOT | Delete a snapshot |
| `CREATE_BACKUP` | BACKUP | Back up sandbox state |

**Job flow:**

1. API server creates a job record in PostgreSQL
2. Runner polls `GET /jobs` on the API
3. Runner claims a job (atomic update)
4. Runner executes the operation
5. Runner reports completion/failure back to the API
6. API updates sandbox state accordingly

### 5.3 Runner Internal API

The runner exposes 19 internal endpoints for sandbox and image management:

**Sandbox operations:**

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/:sandboxId` | Get sandbox status |
| `POST` | `/:sandboxId/destroy` | Destroy a sandbox |
| `POST` | `/:sandboxId/start` | Start a sandbox |
| `POST` | `/:sandboxId/stop` | Stop a sandbox |
| `POST` | `/:sandboxId/backup` | Create a backup |
| `POST` | `/:sandboxId/resize` | Resize sandbox resources |
| `POST` | `/:sandboxId/recover` | Recover a sandbox |
| `GET` | `/:sandboxId/is-recoverable` | Check if sandbox is recoverable |
| `GET` | `/:sandboxId/network-settings` | Get network configuration |

**Image operations:**

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/pull` | Pull an image |
| `POST` | `/build` | Build an image |
| `POST` | `/tag` | Tag an image |
| `POST` | `/remove` | Remove an image |
| `POST` | `/inspect` | Inspect an image |
| `GET` | `/exists` | Check if an image exists locally |
| `GET` | `/info` | Get image metadata |
| `GET` | `/logs` | Stream build/pull logs |

**Other:**

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/*any` | Proxy passthrough |
| `ANY` | `/callback` | Authentication callback |

### 5.4 Health Monitoring

The API server exposes a health endpoint (`GET /api/v1/health`) that reports:

- Database connectivity
- Redis connectivity
- S3 accessibility
- Runner health status (based on `RUNNER_HEALTH_TIMEOUT_SECONDS`)

Protect the health endpoint with `HEALTH_CHECK_API_KEY` in production to prevent information leakage.

### 5.5 Cron Jobs

The API server runs background cron jobs for:

- **Warm pool replenishment**: Maintains pre-provisioned sandboxes
- **Auto-archive**: Archives idle sandboxes past the threshold
- **Stale build cleanup**: Removes build info older than `BUILDINFO_SNAPSHOT_RUNNER_STALENESS_DAYS`
- **Failed snapshot cleanup**: Removes failed snapshot data after `FAILED_SNAPSHOT_RUNNER_RETENTION_HOURS`

Control cron execution with:

- `DISABLE_CRON_JOBS=true` to disable all cron jobs on a given instance
- `CRON_TIMEZONE` to set the scheduling timezone

### 5.6 Logging

Configure logging behavior with:

```bash
LOG_LEVEL=info              # debug, info, warn, error
LOG_CONSOLE_DISABLED=false  # Set true to suppress console output
LOG_REQUESTS_ENABLED=true   # Log HTTP request/response pairs
```

For structured log aggregation, keep console logging enabled and pipe stdout to your log collector (Fluentd, Filebeat, Vector).

### 5.7 Audit Trail

Daytona provides a comprehensive audit system with two storage backends:

**Kafka mode** (`AUDIT_PUBLISH_STORAGE_ADAPTER=kafka`): Publishes audit events to Kafka topics for downstream consumption.

**OpenSearch mode** (`AUDIT_PUBLISH_STORAGE_ADAPTER=opensearch`): Writes audit events directly to an OpenSearch index.

Both modes support batching (`AUDIT_PUBLISH_BATCH_SIZE`) and can run alongside console audit logging (`AUDIT_CONSOLE_LOG_ENABLED`).

Audit log retention is controlled by `AUDIT_LOG_RETENTION_DAYS`. Set this according to your compliance requirements.

> [!NOTE]
> When `AUDIT_TOOLBOX_REQUESTS_ENABLED=true`, all Toolbox API calls from sandboxes are logged. This generates significant volume in active deployments. Enable selectively or ensure your audit storage is sized accordingly.

### 5.8 Troubleshooting

#### Sandbox Creation Fails

| Error | Cause | Solution |
|-------|-------|----------|
| No runners available | All runners are unhealthy or at capacity | Check runner health; add runners; verify `RUNNER_HEALTH_TIMEOUT_SECONDS` |
| Quota exceeded | Organization has exhausted CPU/memory/disk quota | Increase quota via admin API; destroy unused sandboxes |
| Image pull failed | Snapshot image cannot be pulled from registry | Verify registry credentials; check network between runner and registry |
| Build failed | Dockerfile build error | Check build logs via runner `/logs` endpoint; verify `BUILD_CPU_CORES` and `BUILD_MEMORY_GB` are sufficient |

#### Runner Connectivity Issues

| Error | Cause | Solution |
|-------|-------|----------|
| Runner marked unhealthy | Runner failed to report within timeout | Check runner process; verify `DAYTONA_API_URL` reachability; check `RUNNER_HEALTH_TIMEOUT_SECONDS` |
| Job poll failures | Network issue or API authentication failure | Verify `DAYTONA_RUNNER_TOKEN`; check TLS certificates; review runner logs |
| High job latency | Runner overloaded | Check runner scoring thresholds; add capacity; review `BUILD_INFO_MAX_SANDBOXES_PER_RUNNER` |

#### Database Issues

| Error | Cause | Solution |
|-------|-------|----------|
| Connection pool exhausted | Too many concurrent requests for pool size | Increase `DB_POOL_MAX`; check for connection leaks; scale API horizontally |
| Migration failure | Schema conflict or partial migration | Check migration logs; verify only one instance runs migrations; restore from backup if needed |
| Slow queries | Missing indexes or large tables | Enable `LOG_REQUESTS_ENABLED`; analyze slow query log in PostgreSQL; check vacuum schedule |

#### Known Issues

- **go-git fails on small sandboxes** (GitHub [#4544](https://github.com/daytonaio/daytona/issues/4544)): Git operations via go-git can fail in resource-constrained sandboxes. Ensure minimum memory allocation is sufficient.
- **Docker-in-docker base image fails** (GitHub [#4461](https://github.com/daytonaio/daytona/issues/4461)): Certain DinD base images are incompatible. Check the issue for verified compatible images.
- **Runner image missing mount-s3** (GitHub [#3273](https://github.com/daytonaio/daytona/issues/3273)): The `mount-s3` utility may be missing from runner images, affecting S3 filesystem mounts.
- **Toolbox API proxy returns bare 502** (GitHub [#4545](https://github.com/daytonaio/daytona/issues/4545)): Proxy returns 502 without meaningful error body when the Toolbox API is unreachable.
- **listFiles trailing slash issue** (GitHub [#4542](https://github.com/daytonaio/daytona/issues/4542)): File listing API behaves differently with and without trailing slashes in the path.

---

## 6. Performance

### 6.1 Runner Scoring System

The API server uses a weighted scoring algorithm to distribute sandbox operations across runners. This is the core load-balancing mechanism.

#### Score Components

Each runner receives a composite score based on three categories of metrics:

**Resource utilization** (current usage):

| Metric | Weight Variable | Penalty Exponent | Penalty Threshold |
|--------|----------------|------------------|-------------------|
| CPU usage | `RUNNER_CPU_USAGE_WEIGHT` | `RUNNER_CPU_PENALTY_EXPONENT` | `RUNNER_CPU_PENALTY_THRESHOLD` |
| Memory usage | `RUNNER_MEMORY_USAGE_WEIGHT` | `RUNNER_MEMORY_PENALTY_EXPONENT` | `RUNNER_MEMORY_PENALTY_THRESHOLD` |
| Disk usage | `RUNNER_DISK_USAGE_WEIGHT` | `RUNNER_DISK_PENALTY_EXPONENT` | `RUNNER_DISK_PENALTY_THRESHOLD` |
| CPU load average | - | `RUNNER_CPU_LOAD_AVG_PENALTY_EXPONENT` | `RUNNER_CPU_LOAD_AVG_PENALTY_THRESHOLD` |

**Resource allocation** (committed but not necessarily used):

| Metric | Weight Variable |
|--------|----------------|
| Allocated CPU | `RUNNER_ALLOCATED_CPU_WEIGHT` |
| Allocated memory | `RUNNER_ALLOCATED_MEMORY_WEIGHT` |
| Allocated disk | `RUNNER_ALLOCATED_DISK_WEIGHT` |
| Started sandboxes | `RUNNER_STARTED_SANDBOXES_WEIGHT` |

**Threshold levels** (three tiers per metric):

| Tier | CPU | Memory | Disk | Allocated CPU | Allocated Memory | Allocated Disk | Started Sandboxes |
|------|-----|--------|------|--------------|-----------------|---------------|-------------------|
| Optimal | `RUNNER_OPTIMAL_CPU` | `RUNNER_OPTIMAL_MEMORY` | `RUNNER_OPTIMAL_DISK` | `RUNNER_OPTIMAL_ALLOC_CPU` | `RUNNER_OPTIMAL_ALLOC_MEM` | `RUNNER_OPTIMAL_ALLOC_DISK` | `RUNNER_OPTIMAL_STARTED_SANDBOXES` |
| Critical | `RUNNER_CRITICAL_CPU` | `RUNNER_CRITICAL_MEMORY` | `RUNNER_CRITICAL_DISK` | `RUNNER_CRITICAL_ALLOC_CPU` | `RUNNER_CRITICAL_ALLOC_MEM` | `RUNNER_CRITICAL_ALLOC_DISK` | `RUNNER_CRITICAL_STARTED_SANDBOXES` |

#### Scoring Thresholds

Three operation-specific thresholds determine whether a runner is eligible:

| Threshold Variable | Purpose |
|-------------------|---------|
| `RUNNER_DECLARATIVE_BUILD_SCORE_THRESHOLD` | Minimum score to accept image build jobs |
| `RUNNER_AVAILABILITY_SCORE_THRESHOLD` | Minimum score to accept general sandbox operations |
| `RUNNER_START_SCORE_THRESHOLD` | Minimum score to accept sandbox start operations |

#### How Scoring Works

1. The API server queries each runner's reported metrics
2. For each metric, the score starts at 1.0 (optimal) and degrades as usage approaches critical thresholds
3. When usage exceeds a penalty threshold, a penalty function applies: `penalty = (usage / threshold) ^ exponent`
4. The final score is a weighted sum of all component scores
5. Runners below the relevant threshold for the requested operation are excluded
6. Among eligible runners, the highest-scoring runner receives the job

> [!TIP]
> Start with conservative thresholds (e.g., optimal CPU at 40%, critical at 85%) and adjust based on observed performance. Aggressive thresholds lead to uneven distribution; too-conservative thresholds leave capacity unused. Monitor the runner scoring to understand how scores distribute across your fleet.

### 6.2 Connection Pool Tuning

#### PostgreSQL

The default TypeORM pool settings are conservative. For high-throughput deployments:

```bash
DB_POOL_MAX=50
DB_POOL_MIN=10
DB_POOL_IDLE_TIMEOUT_MS=30000
DB_POOL_CONNECTION_TIMEOUT_MS=5000
```

Ensure PostgreSQL's `max_connections` accommodates all API replicas: `max_connections >= (DB_POOL_MAX * replica_count) + 20` (reserve 20 for admin/monitoring).

#### Redis

The ioredis client manages its own connection pool. Ensure Redis `maxclients` is set high enough for all connected API instances plus any other Redis consumers.

### 6.3 Caching Strategy

Daytona caches two hot paths in Redis:

- **API key validation**: Cached for `API_KEY_VALIDATION_CACHE_TTL_SECONDS`. Reduces database round-trips for authenticated requests.
- **API key to user mapping**: Cached for `API_KEY_USER_CACHE_TTL_SECONDS`. Avoids repeated user lookups.

For deployments with high request volume, set both TTLs to 60-300 seconds. Shorter TTLs mean faster propagation of key revocations; longer TTLs mean fewer database queries.

### 6.4 Activity Throttling

Sandbox activity events (heartbeats, file changes) are throttled to prevent overwhelming the database:

- `SANDBOX_ACTIVITY_THROTTLE_TTL_SECONDS`: Minimum interval between activity events for the same sandbox
- `SANDBOX_ACTIVITY_FLUSH_BATCH_SIZE`: Number of activity events batched before flushing to the database

### 6.5 Build Performance

Image builds are resource-intensive. Tune per runner:

- `BUILD_CPU_CORES`: More cores speed up multi-stage builds
- `BUILD_MEMORY_GB`: Increase for builds that compile large projects
- `BUILD_ENGINE=buildkit`: BuildKit is the only supported engine; it provides layer caching and parallel stage execution

The runner shares build resources with running sandboxes. Size runner nodes to accommodate both build workloads and steady-state sandbox operation.

---

## 7. Security

### 7.1 Authentication Architecture

Daytona delegates authentication to an external OIDC provider (Auth0, Dex, Keycloak). The API server validates JWT tokens against the configured issuer.

**Authentication flow:**

1. Client redirects to OIDC provider (`OIDC_ISSUER_BASE_URL`)
2. User authenticates with the provider
3. Provider issues a JWT with the configured audience (`OIDC_AUDIENCE`)
4. Client presents the JWT to the Daytona API
5. API validates the token signature, issuer, audience, and expiration
6. API maps the token subject to a Daytona user account

**API key authentication** is available as an alternative for programmatic access. API keys are validated against the database (with Redis caching). The admin API key (`ADMIN_API_KEY`) bypasses normal authentication for administrative operations.

### 7.2 Authorization Model

Daytona uses an organization-based authorization model with role-based access control.

**Organization roles:**

| Role | Capabilities |
|------|-------------|
| `owner` | Full control over the organization, members, and all resources |
| `member` | Access to organization resources based on assigned permissions |

**Custom role permissions:**

| Permission | Scope |
|------------|-------|
| `write:registries` | Create and update container registries |
| `delete:registries` | Delete container registries |
| `write:snapshots` | Create and update snapshots |
| `delete:snapshots` | Delete snapshots |
| `write:sandboxes` | Create and manage sandboxes |
| `delete:sandboxes` | Delete sandboxes |
| `read:volumes` | View persistent volumes |
| `write:volumes` | Create and update volumes |
| `delete:volumes` | Delete volumes |
| `write:regions` | Create and update regions |
| `delete:regions` | Delete regions |
| `read:runners` | View runner status and configuration |
| `write:runners` | Register and configure runners |
| `delete:runners` | Deregister runners |
| `read:audit_logs` | View audit log entries |

**Personal organizations** are created automatically for each user. They cannot be deleted and do not support member invitations.

### 7.3 Rate Limiting

The five-tier rate-limiting system protects against abuse:

| Tier | Target | Rationale |
|------|--------|-----------|
| **Anonymous** | Unauthenticated endpoints | Prevents scraping and DDoS |
| **Failed Auth** | Authentication failures | Blocks credential stuffing |
| **Authenticated** | General authenticated requests | Fair-use enforcement |
| **Sandbox Create** | Sandbox creation | Prevents resource exhaustion |
| **Sandbox Lifecycle** | Start/stop/destroy | Prevents rapid state churn |

Rate limit counters are stored in Redis with per-tier TTLs. When a limit is exceeded, the API returns `429 Too Many Requests` with a `Retry-After` header.

> [!WARNING]
> The `Failed Auth` rate limit is your primary defense against credential-stuffing attacks. Set it conservatively (e.g., 5 attempts per 300 seconds). A lenient failed-auth limit combined with no IP-based blocking makes brute-force attacks feasible.

### 7.4 Data Encryption

- **At rest**: Sensitive fields (API keys, credentials, tokens) are encrypted using `ENCRYPTION_KEY` with `ENCRYPTION_SALT` for key derivation
- **In transit**: TLS for all external-facing endpoints (`CERT_PATH`, `CERT_KEY_PATH`); optional TLS for database (`DB_TLS_ENABLED`), Redis (`REDIS_TLS`), Kafka (`KAFKA_TLS_ENABLED`), and OpenSearch (`OPENSEARCH_TLS_REJECT_UNAUTHORIZED`)
- **S3**: Use HTTPS endpoints and ensure bucket policies enforce encryption

### 7.5 Network Security

**Egress control**: `ORGANIZATION_SANDBOX_DEFAULT_LIMITED_NETWORK_EGRESS` controls whether sandboxes have restricted outbound network access by default. This is critical for multi-tenant deployments where sandbox workloads are untrusted.

**Inter-service authentication**:

| Flow | Mechanism |
|------|-----------|
| Runner to API | `DAYTONA_RUNNER_TOKEN` (bearer token) |
| Proxy to API | `PROXY_API_KEY` |
| SSH Gateway to API | `SSH_GATEWAY_API_KEY` |
| Admin operations | `ADMIN_API_KEY` |
| Health checks | `HEALTH_CHECK_API_KEY` |

> [!CAUTION]
> Every inter-service API key (`PROXY_API_KEY`, `SSH_GATEWAY_API_KEY`, `DEFAULT_RUNNER_API_KEY`, `ADMIN_API_KEY`) should be unique, high-entropy, and rotated periodically. A compromised runner token grants full sandbox management capability on that runner.

### 7.6 Organization Suspension

Organizations can be suspended with a recorded reason. Suspended organizations cannot create sandboxes, manage resources, or perform any mutating operations. Only admin-level API access can suspend or unsuspend organizations.

### 7.7 Invitation Security

Organization invitations follow a strict state machine:

```
created --> pending --> accepted | declined | cancelled
```

Invitations are tied to email addresses and validated against OIDC identity. `SKIP_USER_EMAIL_VERIFICATION` bypasses email verification, which should only be enabled in development environments.

### 7.8 Secret Management Checklist

| Secret | Where | Rotation Strategy |
|--------|-------|-------------------|
| `ENCRYPTION_KEY` | API server | Rotate with re-encryption migration |
| `ADMIN_API_KEY` | API server | Rotate and update admin clients |
| `DAYTONA_RUNNER_TOKEN` | Each runner | Re-register runner with new token |
| `PROXY_API_KEY` | Proxy + API | Update both simultaneously |
| `SSH_GATEWAY_API_KEY` | SSH GW + API | Update both simultaneously |
| `DB_PASSWORD` | API server + PostgreSQL | Change in PostgreSQL first, then API |
| `REDIS_PASSWORD` | API server + Redis | Change in Redis first, then API |
| OIDC client secret | API server + OIDC provider | Rotate in provider first |
| Registry passwords | API server + registries | Change in registry first, then API |
| S3 credentials | API server | Rotate via IAM; prefer STS with `S3_ROLE_NAME` |
| `SVIX_AUTH_TOKEN` | API server | Rotate in Svix dashboard, then API |

---

## 8. Developer Internals

### 8.1 Monorepo Structure

```
apps/
  api/                 NestJS API server (TypeScript)
  cli/                 Go CLI (Cobra)
  runner/              Go runner service (Gin)
  daemon/              Go daemon (runs inside sandboxes)
  proxy/               Go traffic proxy
  ssh-gateway/         Go SSH gateway
  dashboard/           React dashboard (Vite)
  docs/                Astro documentation site
  snapshot-manager/    Go snapshot manager
  otel-collector/      Custom OpenTelemetry collector

libs/
  api-client/          Auto-generated API clients (Python, TS, Go, Java, Ruby)
  api-client-async/    Async API clients (Python)
  sdk-go/              Go SDK
  sdk-java/            Java SDK
  sdk-ruby/            Ruby SDK
  common-go/           Shared Go utilities
  computer-use/        Computer-use automation library
  runner-proto/        Protobuf definitions for runner communication
  opencode-plugin/     OpenCode plugin integration
```

The monorepo is managed by **Nx 21.4.1** for task orchestration and dependency graph management, with **Yarn 4.13.0** for JavaScript/TypeScript dependency management.

### 8.2 API Server Internals

The API server (`apps/api/`) is a NestJS 11 application running on Express 5.

**Configuration loading** (`apps/api/src/config/configuration.ts`): All environment variables are loaded and validated at startup. The configuration module provides typed access throughout the application via NestJS dependency injection.

**Module organization**: NestJS modules encapsulate domains (sandboxes, organizations, runners, snapshots, audit, etc.). Each module provides its own controllers, services, and entities.

**Database layer**: TypeORM entities define the schema. Migrations are generated with TypeORM CLI and follow the Expand and Contract pattern described in Section 4.2.

**Job dispatch**: When a sandbox operation is requested, the API creates a job record and assigns it to the best-scoring runner. The runner picks up the job on its next poll cycle.

### 8.3 Runner Internals

The runner (`apps/runner/`) is a Go service built with the Gin HTTP framework.

**Job polling**: The runner's main loop polls the API server every `POLL_TIMEOUT` (default 30s) for up to `POLL_LIMIT` (default 10) pending jobs. Claimed jobs are executed concurrently.

**Build engine**: BuildKit is the sole supported build engine (`BUILD_ENGINE=buildkit`). The runner manages a BuildKit daemon for container image builds, allocating `BUILD_CPU_CORES` and `BUILD_MEMORY_GB` to the build process.

**Sandbox management**: The runner creates, starts, stops, and destroys sandboxes using the container runtime. Each sandbox runs the Daemon process internally for Toolbox API access.

**Image operations**: The runner handles image pull, build, tag, remove, and inspect operations. Build logs are streamed through the `/logs` endpoint.

### 8.4 Daemon (Toolbox API)

The daemon (`apps/daemon/`) runs inside each sandbox and exposes the Toolbox API. This API provides:

- File system operations (read, write, list, search)
- Process management (execute, signal, list)
- Git operations (clone, pull, push, status)
- Port forwarding management
- Environment variable management

The Toolbox API is accessed via the proxy, which routes requests based on sandbox ID. When `AUDIT_TOOLBOX_REQUESTS_ENABLED=true`, all Toolbox API calls are logged.

### 8.5 Proxy Internals

The proxy (`apps/proxy/`) routes HTTP and WebSocket traffic to sandboxes. It resolves sandbox-specific subdomains using the `PROXY_TEMPLATE_URL` pattern and forwards traffic to the appropriate runner and sandbox.

The proxy authenticates with the API server using `PROXY_API_KEY` to resolve sandbox routing information.

### 8.6 SSH Gateway Internals

The SSH gateway (`apps/ssh-gateway/`) provides SSH access to running sandboxes. It authenticates users via the API server (using `SSH_GATEWAY_API_KEY`), maps SSH connections to sandbox instances, and proxies the SSH session to the sandbox's daemon.

### 8.7 Protobuf Communication

The `libs/runner-proto/` package defines Protocol Buffer message types for structured communication between the API server and runners. This provides type-safe serialization for job payloads and status updates.

### 8.8 Auto-generated API Clients

API clients for all five languages are auto-generated from the OpenAPI specification. The generation process:

1. API server exports its OpenAPI spec
2. Code generators produce client libraries in `libs/api-client/` (sync) and `libs/api-client-async/` (Python async)
3. Generated clients are versioned alongside the API

Do not manually edit generated client code. Changes must go through the OpenAPI spec and regeneration pipeline.

### 8.9 Testing

The monorepo uses Nx to orchestrate test execution across projects:

```bash
# Run all tests
yarn nx run-many --target=test

# Run tests for a specific project
yarn nx test api

# Run Go tests
yarn nx test runner
yarn nx test cli
```

### 8.10 Building

```bash
# Build all projects
yarn nx run-many --target=build

# Build specific project
yarn nx build api
yarn nx build dashboard

# Build Go binaries
yarn nx build runner
yarn nx build cli
```

---

## 9. Integration Points

### 9.1 Webhook Events (Svix)

Daytona dispatches webhook events via Svix for real-time integration with external systems. Webhooks are managed per-organization.

**Prerequisites:**

- `SVIX_AUTH_TOKEN` configured
- `SVIX_SERVER_URL` pointing to a Svix instance (self-hosted or cloud)

**Event categories:**

| Category | Events |
|----------|--------|
| Sandbox lifecycle | Created, started, stopped, archived, destroyed, error |
| Snapshot | Created, built, removed |
| Organization | Member invited, member joined, member removed |

Webhook endpoints are registered through the API. Svix handles delivery, retries, and endpoint health monitoring.

### 9.2 OpenTelemetry

The custom OTel collector (`apps/otel-collector/`) aggregates telemetry from all components:

**Traces**: Distributed traces across API server, runners, and proxy. Each span is tagged with `ENVIRONMENT` for filtering.

**Metrics**: Resource utilization, job latency, sandbox lifecycle timing, error rates.

**Export targets**: ClickHouse (primary analytics store), plus any OTLP-compatible backend.

Enable with `OTEL_ENABLED=true` and point `OTEL_COLLECTOR_ENDPOINT_URL` to the collector's gRPC endpoint.

### 9.3 PostHog Analytics

Optional product analytics integration:

```bash
POSTHOG_HOST=https://app.posthog.com
POSTHOG_API_KEY=phc_...
POSTHOG_ENVIRONMENT=production
```

PostHog receives anonymized usage events for product development insights. This is optional and can be disabled by leaving the variables unset.

### 9.4 Billing Integration

External billing is supported via `BILLING_API_URL`. The API server communicates with the billing service for:

- Usage metering (CPU-hours, storage, bandwidth)
- Quota enforcement
- Subscription management

### 9.5 Analytics Integration

External analytics via `ANALYTICS_API_URL` supplements the built-in ClickHouse analytics for business intelligence dashboards and reporting.

### 9.6 Pylon Integration

`PYLON_APP_ID` enables Pylon integration for customer support workflows, connecting sandbox issues to support tickets.

### 9.7 SDK Integration Patterns

**Python SDK:**

```python
from daytona import Daytona

daytona = Daytona(api_key="your-api-key", api_url="https://api.daytona.example.com")

# Create a sandbox
sandbox = daytona.create()

# Execute a command
response = sandbox.process.exec("echo 'Hello from sandbox'")
print(response.output)

# Clean up
sandbox.delete()
```

**TypeScript SDK:**

```typescript
import { Daytona } from '@daytona/sdk';

const daytona = new Daytona({
  apiKey: 'your-api-key',
  apiUrl: 'https://api.daytona.example.com',
});

const sandbox = await daytona.create();
const result = await sandbox.process.exec('echo "Hello from sandbox"');
console.log(result.output);
await sandbox.delete();
```

**Go SDK:**

```go
package main

import (
    "fmt"
    daytona "github.com/daytonaio/daytona/libs/sdk-go"
)

func main() {
    client := daytona.NewClient("your-api-key", "https://api.daytona.example.com")
    sandbox, _ := client.Create()
    result, _ := sandbox.Process.Exec("echo 'Hello from sandbox'")
    fmt.Println(result.Output)
    sandbox.Delete()
}
```

> [!TIP]
> The Python SDK provides both synchronous and asynchronous clients. For high-throughput automation, use the async client from `libs/api-client-async/` to avoid blocking on sandbox operations.

### 9.8 Docker Registry Types

Daytona manages four types of container registries:

| Type | Purpose | Configuration |
|------|---------|---------------|
| **Internal** | Platform-managed images and base snapshots | `INTERNAL_REGISTRY_*` variables |
| **Transient** | Ephemeral build artifacts, automatically cleaned | `TRANSIENT_REGISTRY_*` variables |
| **Organization** | User-managed registries per organization | Configured via API |
| **Backup** | Sandbox backup storage | Managed internally |

### 9.9 Multi-Region Architecture

Daytona supports multi-region deployments where runners in different regions serve sandboxes with region affinity:

- `DEFAULT_REGION_ID` and `DEFAULT_REGION_NAME` define the primary region
- `DEFAULT_REGION_ENFORCE_QUOTAS` controls whether resource quotas are enforced per-region
- Additional regions are created via the API and assigned runners

Region-aware scheduling routes sandbox creation to runners in the requested region, falling back to other regions based on runner scoring.

> [!IMPORTANT]
> In multi-region deployments, ensure S3 storage is either replicated across regions or uses a globally accessible endpoint. Sandbox snapshots stored in one region must be accessible from runners in other regions for cross-region restore operations.

---

_Generated May 2026 from daytonaio/daytona at 4c6a681. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._