# Twitter's "The Algorithm" — Administrator & Developer Guide

This guide covers the architecture, internals, configuration, deployment patterns, and code structure of Twitter's open-sourced recommendation algorithm. It is written for platform engineers, ML engineers, and architects who need to understand how these systems were designed and operated, either for study, adaptation, or reimplementation.

## 1. Architecture Overview

### What the System Does

Twitter's recommendation algorithm is a distributed system of ~20 interconnected microservices that collectively decide what content users see across Home Timeline, Search, Notifications, and Who to Follow. The system processes billions of tweets and hundreds of millions of user interactions to produce personalized feeds in real time.

The architecture follows a multi-stage candidate generation → ranking → filtering pattern, where each stage progressively narrows and scores content. Services communicate primarily via Thrift RPC (using Twitter's Finatra/Finagle stack), with Kafka for event streaming and HDFS/Manhattan for persistent storage.

### System Architecture

```text
┌──────────────────────────────────────────────────────────────────┐
│                        PRODUCT LAYER                             │
│  ┌─────────────┐  ┌─────────────┐  ┌────────────┐  ┌─────────┐ │
│  │ Home Mixer   │  │ Tweet Mixer │  │ Push Svc   │  │ Search  │ │
│  │ (For You/    │  │ (content    │  │ (notif     │  │ Root    │ │
│  │  Following)  │  │  blending)  │  │  ranking)  │  │         │ │
│  └──────┬───────┘  └──────┬──────┘  └─────┬──────┘  └────┬────┘ │
│         │                 │               │               │      │
│  ┌──────┴─────────────────┴───────────────┴───────────────┘      │
│  │           Product Mixer Framework                             │
│  │     (pipeline orchestration, feature hydration, scoring)      │
│  └──────┬────────────────────────────────────────────────────┘   │
└─────────┼────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼          RECOMMENDATION LAYER                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Timeline     │  │ CR-Mixer     │  │ Follow Recs  │           │
│  │ Ranker       │  │ (content     │  │ Service      │           │
│  │ (scoring)    │  │  candidates) │  │ (who to      │           │
│  │              │  │              │  │  follow)     │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ SimClusters  │  │ Topic Social │  │ Recos        │           │
│  │ ANN          │  │ Proof        │  │ Injector     │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
└──────────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼           FEATURE & SIGNAL LAYER                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Graph Feature│  │ User Signal  │  │ Representation│          │
│  │ Service      │  │ Service      │  │ Manager/Scorer│          │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
│  ┌──────────────┐  ┌──────────────┐                              │
│  │ Unified User │  │ VisibilityLib│                              │
│  │ Actions      │  │ (moderation) │                              │
│  └──────────────┘  └──────────────┘                              │
└──────────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼           DATA & ML LAYER                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Earlybird    │  │ TweetyPie    │  │ ANN Service  │           │
│  │ (search      │  │ (tweet       │  │ (nearest     │           │
│  │  index)      │  │  storage)    │  │  neighbor)   │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Navi (model  │  │ twml (ML     │  │ Trust &      │           │
│  │  serving)    │  │  training)   │  │ Safety Models│           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
└──────────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼           INFRASTRUCTURE                               │
│  Aurora (scheduling) │ Kafka (streaming) │ HDFS (storage)        │
│  Manhattan (KV)      │ Strato (data)     │ ZooKeeper (coord)     │
│  Thrift (RPC)        │ Finagle (network) │ Finatra (framework)   │
└──────────────────────────────────────────────────────────────────┘
```

> [!NOTE]
> All Scala/Java services in this repository use Twitter's Finatra framework for HTTP/Thrift server lifecycle, dependency injection (via Google Guice), and request filtering. The Finagle library provides the underlying RPC, load balancing, and connection pooling.

### Component Inventory

| Component | Path | Language | Role |
|---|---|---|---|
| Home Mixer | `home-mixer/` | Scala | Home timeline assembly (For You, Following) |
| Timeline Ranker | `timelineranker/` | Scala | Tweet scoring and ranking |
| CR-Mixer | `cr-mixer/` | Scala | Content-based recommendation candidates |
| Follow Recs Service | `follow-recommendations-service/` | Scala | Who to Follow candidate gen + ranking |
| Product Mixer | `product-mixer/` | Scala | Generic pipeline framework |
| TweetyPie | `tweetypie/` | Scala | Tweet storage, hydration, CRUD |
| Push Service | `pushservice/` | Scala + Python | Notification ranking (heavy/light) |
| Earlybird | `src/java/com/twitter/search/` | Java | Real-time tweet search (Lucene-based) |
| VisibilityLib | `visibilitylib/` | Scala | Content moderation rules engine |
| Graph Feature Service | `graph-feature-service/` | Scala | User-user graph feature computation |
| User Signal Service | `user-signal-service/` | Scala | User behavioral signal aggregation |
| Unified User Actions | `unified_user_actions/` | Scala | Cross-product event tracking |
| SimClusters | `src/scala/com/twitter/simclusters_v2/` | Scala | Topic embedding generation (batch) |
| SimClusters ANN | `simclusters-ann/` | Scala | Topic-based nearest neighbor |
| ANN Service | `ann/` | Scala + Java | Approximate nearest neighbor (HNSW/Annoy/FAISS) |
| Representation Manager | `representation-manager/` | Scala | Embedding storage and versioning |
| Representation Scorer | `representation-scorer/` | Scala | Embedding similarity computation |
| Navi | `navi/` | Rust | ML model serving (TF/PyTorch/ONNX) |
| twml | `twml/` | Python + C++ | ML training framework (on TF) |
| Trust & Safety Models | `trust_and_safety_models/` | Python | NSFW, toxicity, abusive content models |
| Recos Injector | `recos-injector/` | Scala | Recommendation event injection |
| Topic Social Proof | `topic-social-proof/` | Scala | Topic relevance scoring |
| Tweet Mixer | `tweet-mixer/` | Scala | Tweet blending for mixed feeds |
| Interaction Graph | `src/scala/com/twitter/interaction_graph/` | Scala | User interaction graph processing |
| Timelines | `timelines/` | Python + Scala | Timeline data processing and model training |

### Technology Stack

| Layer | Technology |
|---|---|
| Primary Languages | Scala 2.12, Java 11, Python 3.7, Rust |
| Build System | Bazel (Scala/Java), Cargo (Rust), pip/setuptools (Python) |
| RPC Framework | Apache Thrift via Finagle |
| Service Framework | Finatra (HTTP + Thrift), mTLS |
| ML Training | TensorFlow (via twml), Keras |
| ML Serving | Navi (TF, PyTorch, ONNX via gRPC) |
| Search Engine | Apache Lucene (custom Earlybird) |
| Stream Processing | Apache Kafka |
| Batch Processing | Apache Beam, Scalding, Scio (on Hadoop/HDFS) |
| Coordination | Apache ZooKeeper |
| Storage | Manhattan (KV), HDFS, Feature Store |
| Deployment | Aurora (Mesos scheduler) |
| Graph Processing | GraphJet (real-time), Scalding (batch) |
| Embeddings | SimClusters, Word2Vec (Follow2Vec), WHIMP (cosine similarity) |
| ANN Algorithms | HNSW, Annoy, FAISS, brute-force |
| Monitoring | Prometheus (Navi), Stats (Finagle), Decider (feature flags) |

### Data Flow: Home Timeline Request

```text
1. User opens For You tab
2. Home Mixer receives request
3. Product Mixer orchestrates pipeline:
   a. Earlybird → in-network tweet candidates (recent tweets from followed accounts)
   b. CR-Mixer → out-of-network candidates (content recs from multiple signals)
   c. SimClusters ANN → topic-relevant candidates
4. Feature hydration:
   a. Graph Feature Service → user-user graph features
   b. User Signal Service → engagement history, behavioral signals
   c. Representation Manager → user/tweet embeddings
5. Timeline Ranker scores all candidates with heavy ranking model
6. VisibilityLib filters by safety rules, content policy
7. Home Mixer assembles final timeline with mixing rules
8. Response returned to client
```

## 2. System Requirements

### Runtime Infrastructure Dependencies

| Dependency | Purpose | Required For |
|---|---|---|
| Apache Aurora | Job scheduling and deployment | All Scala/Java services |
| Apache Kafka | Event streaming | UUA, Earlybird indexing, feature updates |
| HDFS | Distributed file storage | Batch jobs, model storage, embeddings |
| Manhattan | Key-value store | Feature serving, embedding lookup |
| Strato | Data access layer | Feature Store, various data sources |
| Apache ZooKeeper | Service coordination | Dataset pointers, config distribution |
| Finagle/Finatra | RPC and service framework | All Scala/Java services |
| Feature Store | ML feature serving | Model feature hydration |

### Hardware Requirements (Per Service)

Based on Aurora job configurations in the ground truth:

| Service | CPU | RAM | Disk | Notes |
|---|---|---|---|---|
| ANN Index Builder | 32 cores | 150 GB | 60 GB | GPU optional, CUDA 11.2 for ML |
| Navi Model Serving | 8+ cores | 16+ GB | Varies | GPU recommended for TF/PyTorch |
| twml Training | 2+ cores | 8+ GB | Varies | GPU required for deep models |
| Earlybird | High | High | SSD | Lucene index in memory/SSD |
| Graph Feature Service | Moderate | High | Moderate | Graph data in memory |

> [!IMPORTANT]
> The ANN Dataflow worker Dockerfile specifies CUDA 11.2.2 with cuDNN 8 on Ubuntu 20.04 as the reference GPU environment. Python 3.7 is pinned explicitly.

## 3. Configuration Reference

### Navi Model Serving Configuration

Navi reads a JSON configuration file specifying model locations and runtime parameters:

```json
{
  "model_dir": ["/path/to/model1", "/path/to/model2"],
  "inter_op_parallelism": 1,
  "intra_op_parallelism": 4
}
```

Key configuration functions (from `navi/navi/src/lib.rs`):

| Function | Purpose |
|---|---|
| `read_config(meta_file)` | Parse JSON config from file |
| `get_config_or_else(config, key, default_fn)` | Read config key with computed default |
| `get_config_or(config, key, default)` | Read config key with string default |
| `get_meta_dir()` | Get model metadata directory path |

Navi CLI arguments (from `navi/navi/src/cli_args.rs`):

| Argument | Type | Description |
|---|---|---|
| `--model-dir` | `Vec<String>` | Model directory paths |
| `--config` | `String` | Path to JSON configuration file |

### twml Training Configuration (Complete CLI Reference)

All 70 CLI parameters extracted from `twml/twml/argument_parser.py`:

#### Core Training Parameters

| Flag | Type | Default | Description |
|---|---|---|---|
| `--save_dir` | str | (required) | Training result directory (local or `hdfs://`) |
| `--export_dir` | str | None | SavedModel export directory for prediction serving |
| `--train.batch_size` | int | 32 | Samples per training batch |
| `--eval.batch_size` | int | 32 | Samples per evaluation batch |
| `--train.learning_rate` | float | 0.002 | Learning rate (scales gradient update) |
| `--train.steps` | int | -1 | Training batches before evaluation (-1 = full dataset) |
| `--eval.steps` | int | -1 | Steps per evaluation (-1 = full dataset) |
| `--train.max_steps` | int | None | Global step limit (None = one train/eval cycle) |
| `--optimizer` | str | SGD | Optimizer: SGD, Adagrad, Adam, Ftrl, Momentum, RMSProp, LazyAdam, DGC |
| `--data_threads` | int | 2 | Dataset loading threads |
| `--num_workers` | int | None | Hogwild workers for single-node training |

#### Evaluation & Early Stopping

| Flag | Type | Default | Description |
|---|---|---|---|
| `--eval.period` | int | 600 | Seconds between evaluations in `train_and_evaluate` |
| `--eval.delay` | int | 120 | Seconds before first evaluation |
| `--train.early_stop_patience` | int | -1 | Max evaluations without improvement (-1 = disabled) |
| `--train.early_stop_tolerance` | float | 0 | Minimum improvement threshold |
| `--train.log_metrics` | flag | False | Log metrics during training (10-20% perf hit) |

> [!WARNING]
> Early stopping (`--train.early_stop_patience`) cannot be used with `--distributed` mode. Attempting to combine them will produce undefined behavior.

#### Distributed Training

| Flag | Type | Default | Description |
|---|---|---|---|
| `--distributed` | flag | False | Enable distributed `train_and_evaluate` |
| `--distributed_training_cleanup` | flag | False | Stop TwitterSetDeployment on restart (GKE) |
| `--disable_auto_ps_shutdown` | flag | False | Disable auto parameter server shutdown |
| `--train.dataset_shards` | int | None | Number of data partitions for codistillation |
| `--train.dataset_shard_index` | int | None | Which data partition to use (0-indexed) |

#### Checkpointing & Monitoring

| Flag | Type | Default | Description |
|---|---|---|---|
| `--config.save_checkpoints_secs` | int | 600 | Checkpoint save interval (seconds) |
| `--config.keep_checkpoint_max` | int | 20 | Maximum checkpoints to retain |
| `--config.tf_random_seed` | int | None | TensorFlow random seed |
| `--overwrite_save_dir` | flag | False | Delete existing save_dir contents |
| `--disable_tensorboard` | flag | False | Don't start TensorBoard server |
| `--tensorboard_port` | int | None | TensorBoard port (ignored if disabled) |
| `--health_port` | int | None | Health endpoint port (set by twml_cli) |
| `--stats_port` | int | None | Stats endpoint port |
| `--experiment_tracking_path` | str | None | ML Dashboard tracking path (`user:project:experiment:run`) |
| `--disable_experiment_tracking` | flag | False | Disable experiment tracking |

#### Gradient & Optimization

| Flag | Type | Default | Description |
|---|---|---|---|
| `--gradient_noise_scale` | float | None | 0-mean normal noise scale |
| `--clip_gradients` | float | None | Global gradient norm clipping |
| `--dgc.density` | float | 0.1 | Deep gradient compression density (top 10%) |
| `--dgc.density_decay` | bool | True | Enable exponential density decay |
| `--dgc.density_decay_steps` | int | 10000 | Decay step interval |
| `--dgc.density_decay_rate` | float | 0.5 | Decay rate |
| `--dgc.min_density` | float | 0.1 | Minimum density after decay |
| `--dgc.accumulation` | bool | False | Accumulate small gradients |
| `--show_optimizer_summaries` | flag | False | Display gradients/LR in TensorBoard |
| `--num_mkl_threads` | int | 1 | MKL parallelism threads |

#### Learning Rate Decay

| Flag | Type | Default | Description |
|---|---|---|---|
| `--decay_steps` | float | None | Steps for exponential/polynomial/inverse/cosine decay |
| `--exponential_decay_rate` | float | None | Rate for exponential decay |
| `--end_learning_rate` | float | 0.0001 | End rate for polynomial decay |

Supported decay schedules: `exponential`, `polynomial`, `piecewise_constant`, `inverse`, `cosine_decay`, `cosine_restarts_decay`.

#### Feature Importance

| Flag | Type | Default | Description |
|---|---|---|---|
| `--feature_importance.algorithm` | str | (none) | `serial` or `tree` |
| `--feature_importance.sensitivity` | float | 0.03 | Max metric drop before tree expansion stops |
| `--feature_importance.example_count` | int | 10000 | Examples for importance computation |
| `--feature_importance.data_dir` | str | None | Dataset path (defaults to eval dir) |
| `--feature_importance.metric` | str | roc_auc | Metric for tree algorithm stop criterion |
| `--feature_importance.is_metric_larger_the_better` | flag | False | Metric direction (larger = better) |
| `--feature_importance.is_metric_smaller_the_better` | flag | False | Metric direction (smaller = better) |
| `--feature_importance.dont_build_tree` | flag | False | Skip trie building for tree algorithm |
| `--feature_importance.split_feature_group_on_period` | flag | False | Split groups on period vs. optimal prefix |

### ANN Index Builder Configuration

The ANN index builder accepts these Aurora profile parameters:

| Parameter | Type | Description |
|---|---|---|
| `profile.name` | string | Aurora job name |
| `profile.role` | string | Aurora role |
| `profile.hadoop_cluster` | string | Hadoop cluster (e.g., `dw2-smf1`) |
| `profile.entity_kind` | enum | Entity type: `user`, `tweet`, `word`, `url`, `tfwId` |
| `profile.num_dimensions` | int | Embedding dimensionality |
| `profile.metric` | enum | Distance metric: `InnerProduct`, `Cosine`, `L2` |
| `profile.concurrency_level` | int | Parallel index inserts (set to CPU count) |
| `profile.algo` | enum | Algorithm: `hnsw`, `annoy`, `brute_force` |
| `profile.output_dir` | string | HDFS output path for serialized index |

**HNSW-specific:**

| Parameter | Default | Description |
|---|---|---|
| `profile.ef_construction` | — | Build-time recall parameter (higher = better recall, slower build) |
| `profile.max_m` | — | Max connections per node (6-48 optimal, 16 recommended) |
| `profile.expected_elements` | — | Approximate index size |

**Annoy-specific:**

| Parameter | Default | Description |
|---|---|---|
| `profile.annoy_num_trees` | — | Number of trees (more = better accuracy, larger index) |

**FAISS-specific (load testing only):**

| Parameter | Default | Description |
|---|---|---|
| `profile.faiss_nprobe` | `1` | Number of clusters to probe |
| `profile.faiss_quantizerEf` | `0` | Quantizer ef parameter |
| `profile.faiss_quantizerKfactorRF` | `0` | Quantizer k-factor |
| `profile.faiss_quantizerNprobe` | `0` | Quantizer nprobe |
| `profile.faiss_ht` | `0` | Hamming threshold |

> [!NOTE]
> FAISS parameters accept comma-separated lists for grid search during load testing (e.g., `--bind=profile.faiss_nprobe=1,3,9,27,81,128`).

### Environment Variables

| Variable | Source | Description |
|---|---|---|
| `TWML_JOB_CLUSTER` | `twml/twml/util.py:917` | Cluster for distributed training |
| `TWML_JOB_ROLE` | `twml/twml/util.py:918` | Job role identifier |
| `TWML_JOB_ENV` | `twml/twml/util.py:919` | Environment (staging/production) |
| `TWML_DISTRIBUTED_BASE_JOBNAME` | `twml/twml/util.py:920` | Base name for distributed jobs |
| `HADOOP_CONF_DIR` | — | HDFS configuration (required for `hdfs://` paths) |

## 4. Deployment & Scaling

### Deployment Model

All production services deploy on **Apache Aurora** (a Mesos scheduler). Aurora job configurations are defined in `.aurora` files using Thermos templates.

Deployment pattern:
1. Build service bundle with Bazel: `./bazel bundle <target>:deploy --bundle-jvm-archive=zip`
2. Upload bundle to Packer (Twitter's artifact storage)
3. Create Aurora job: `aurora job create <cluster>/<role>/<env>/<name> <config>.aurora`

### Service Architecture Patterns

All Scala services follow a consistent pattern using Finatra:

```scala
object Main extends ServerMain

class ServerMain extends ThriftServer with Mtls {
  override val name = "service-name"
  override val modules: Seq[Module] = Seq(/* Guice modules */)

  override def configureThrift(router: ThriftRouter): Unit = {
    router
      .filter[LoggingMDCFilter]
      .filter[TraceIdMDCFilter]
      .filter[ThriftMDCFilter]
      .filter[StatsFilter]
      .add[ServiceController]
  }

  override protected def warmup(): Unit = {
    handle[WarmupHandler]()
  }
}
```

Standard filter chain: `LoggingMDCFilter` → `TraceIdMDCFilter` → `ThriftMDCFilter` → `AccessLoggingFilter` → `StatsFilter` → controller.

### Earlybird Cluster Topology

Earlybird splits the tweet index across three cluster types:

| Cluster | Content | Freshness | Shard Strategy |
|---|---|---|---|
| Realtime | Public tweets | Last ~7 days | Time-based partitioning |
| Protected | Protected tweets | Last ~7 days | Time-based partitioning |
| Archive | All tweets | All time (up to ~2 days ago) | Time-based partitioning |

Each cluster uses multiple partitions with replication. **Earlybird roots** fan out queries across partitions and merge responses.

**Indexing pipeline:**
- Ingesters consume from Kafka, extract fields/features, write to intermediate Kafka topics
- Earlybird instances consume from intermediate topics and update in-memory Lucene indexes
- Feature Update Service pushes real-time engagement counts (likes, retweets, replies)

### Navi Deployment

Navi compiles to three separate binaries depending on ML backend:

| Binary | Feature Flag | Backend |
|---|---|---|
| `navi` | `tf` | TensorFlow SavedModel |
| `navi_torch` | `torch` | PyTorch TorchScript |
| `navi_onnx` | `onnx` | ONNX Runtime |

Each binary serves predictions via the TensorFlow Serving gRPC protocol (`prediction_service.proto`), with support for the KFServing v2 inference protocol.

> [!TIP]
> Navi supports hot model reloading. When a new model version directory appears under the configured `model_dir`, Navi automatically detects the version timestamp and loads the new model without restart.

## 5. Operations

### Monitoring

**Navi** exposes Prometheus metrics via an HTTP endpoint (Warp-based). Key metrics:

| Metric | Type | Description |
|---|---|---|
| Prediction latency | Histogram | Per-model inference time distribution |
| Batch size | Histogram | Dynamic batching size distribution |
| Model load time | Gauge | Time to load model version |
| Request count | Counter | Total prediction requests |

**Scala services** use Finagle's built-in StatsReceiver for metrics, exposed via the standard Finagle admin interface.

> [!NOTE]
> All Scala services include `StatsFilter` in their Thrift filter chain, which automatically tracks request rate, latency percentiles, and error rates per endpoint.

### Feature Flags (Deciders)

Services use Twitter's **Decider** system for feature flags and gradual rollouts. Decider modules are injected via Guice:

```scala
override val modules: Seq[Module] = Seq(
  DeciderModule,
  // ...
)
```

Deciders control traffic routing, experiment allocation, and feature availability without redeployment.

### Health Checks

Finatra services expose health endpoints via the admin HTTP interface. twml training jobs expose health on a configurable `--health_port` for graceful shutdown coordination.

> [!TIP]
> For twml distributed training, the `--disable_auto_ps_shutdown` flag prevents the parameter server from shutting down prematurely when individual workers fail. Use this for fault-tolerant long-running jobs.

### Batch Processing Operations

SimClusters, interaction graphs, and embedding generation run as batch Scalding/Scio jobs on Hadoop:

```bash
# Example: Run embedding sampling job
oscar hdfs \
  --screen --tee log.txt \
  --hadoop-client-memory 6000 \
  --bundle embeddingformat-deploy \
  --host hadoopnest1.smf1.twitter.com \
  --tool com.twitter.scalding.entityembeddings.util.EmbeddingFormat.EmbeddingSamplingJob -- \
  --entity_kind user \
  --input.embedding_path $INPUT_PATH \
  --input.embedding_format usertensor \
  --output.embedding_path $OUTPUT_PATH \
  --output.embedding_format tab \
  --sample_percent 0.1
```

Supported embedding formats: `usertensor`, `usercontinuous`, `comma`, `tab`.

## 6. Performance

### Earlybird Optimizations

- **Incremental indexing:** New tweets are indexed in real-time as they arrive from Kafka, maintaining index freshness
- **Single-writer, multiple-reader:** Eliminates write contention while supporting high query concurrency
- **Partitioned index:** Horizontal scaling across shards, with roots handling fan-out and merge
- **Column-stride field indexes:** Optimized for per-document feature lookups (engagement counts, timestamps)

### Navi Batching

Navi implements dynamic batching to maximize GPU/CPU utilization:

- Incoming requests are buffered in a `BatchPredictor`
- Batches are flushed when either the size limit or timeout is reached (`duration_past` check)
- Responses are demultiplexed back to individual request channels

### Product Mixer Quality Factors

Product Mixer pipelines support **quality factors** — runtime parameters that trade off result quality for latency:

```scala
// From pipeline framework
class QualityFactorStatus {
  // Dynamically adjust pipeline behavior based on latency targets
}
```

When latency increases, quality factors can reduce candidate counts, simplify feature hydration, or skip expensive scoring stages.

## 7. Security

### Authentication & Authorization

- All Scala services use **mTLS** (mutual TLS) via Finatra's `Mtls` trait
- Navi supports mTLS with x509 certificate verification (`x509-parser` crate)
- Inter-service communication is authenticated via Thrift client IDs (`ThriftClientIdModule`)

> [!WARNING]
> The open-source release does not include the internal certificate infrastructure. Services expect valid mTLS certificates for production operation. For local development, you would need to set up a local CA or disable mTLS.

### Content Safety

VisibilityLib is the centralized content moderation engine. It evaluates rules against:

- **Content labels** — safety classifications applied to tweets and users
- **Visibility actions** — outcomes like hide, filter, tombstone, interstitial
- **Rule conditions** — composable predicates combining labels, user state, and context

The `visibilitylib/src/main/scala/com/twitter/visibility/models/` directory (24 files, 126K chars) defines the complete vocabulary of content labels and actions.

### Trust & Safety ML Models

| Model | Path | Purpose |
|---|---|---|
| NSFW Text Classifier | `trust_and_safety_models/nsfw/` | Text-based NSFW detection |
| Toxicity Classifier | `trust_and_safety_models/toxicity/` | Toxic content detection |
| Abusive Content Model | `trust_and_safety_models/abusive/` | Abusive behavior classification |

The toxicity trainer supports multiple loss functions: `BCE`, `masked BCE`, `CCE`, `sCCE`, `Focal_BCE`, `inv_KL_loss`, with configurable optimizers and multi-fold training.

> [!CAUTION]
> The trust and safety models are trained on sensitive data. The model architectures and training code are included, but training data is not part of the open-source release.

## 8. Developer Internals

### Repository Structure

```text
the-algorithm/                          # Root (7,651 files, 12.3M source chars)
├── home-mixer/                         # Home timeline product pipeline
│   └── server/src/main/scala/          # Scala service implementation
├── timelineranker/                     # Tweet ranking service
│   ├── server/                         # Server with config and Scala source
│   ├── client/                         # Client library
│   └── common/                         # Shared models
├── follow-recommendations-service/     # Who to Follow
│   ├── server/                         # Server implementation
│   ├── common/                         # 54+ model files, candidate sources
│   └── thrift/                         # Thrift service definitions
├── cr-mixer/                           # Content recommendation mixer
│   ├── server/                         # Server with 9 core modules
│   └── thrift/                         # Thrift definitions
├── product-mixer/                      # Generic pipeline framework
│   ├── core/                           # Pipeline, model, feature, controller
│   ├── component-library/              # Reusable components (scorers, features)
│   └── shared-library/                 # Shared utilities (observers)
├── tweetypie/                          # Tweet storage service
│   ├── server/                         # 49 handler files, 22 config files
│   ├── common/                         # Shared types
│   └── servo/                          # Request/repo/json/decider subsystems
├── pushservice/                        # Push notification service
│   └── src/main/python/models/         # Heavy/light ranking models
├── visibilitylib/                      # Content moderation
│   └── src/main/scala/                 # 24 model files, features, rules, builders
├── src/                                # Core shared source
│   ├── java/com/twitter/search/        # Earlybird search (Java)
│   ├── scala/com/twitter/              # SimClusters, interaction graph, recos
│   ├── python/                         # Deepbird timeline models
│   └── thrift/                         # Shared Thrift definitions
├── twml/                               # Twitter ML library
│   ├── twml/                           # Python package (17 files, 180K chars)
│   ├── twml_common/                    # Common utilities
│   └── libtwml/                        # C++ native library (21 files, 112K chars)
├── navi/                               # ML model serving (Rust)
│   ├── navi/                           # Main server (9 src files, 100K chars)
│   ├── dr_transform/                   # Data record transformer
│   ├── segdense/                       # Feature densification
│   └── thrift_bpr_adapter/             # Thrift BPR parsing (127K chars)
├── ann/                                # Approximate Nearest Neighbor
│   └── src/main/scala/com/twitter/ann/ # Common, query server, scalding
├── graph-feature-service/              # Graph features
│   └── src/main/scala/                 # Server + worker architecture
├── user-signal-service/                # User signals
│   ├── server/                         # 18 signal types
│   └── thrift/                         # Service definitions
├── unified_user_actions/               # Event tracking
│   ├── adapter/                        # Event adapters
│   ├── kafka/                          # Kafka serialization
│   ├── enricher/                       # Event enrichment
│   ├── service/                        # UUA service
│   └── client/                         # Client library
├── representation-manager/             # Embedding management
│   ├── server/                         # Migration, config, common
│   └── client/                         # Client with config
├── representation-scorer/              # Embedding scoring
├── recos-injector/                     # Recommendation injection
├── trust_and_safety_models/            # Safety ML models
│   ├── nsfw/                           # NSFW classifier
│   ├── toxicity/                       # Toxicity classifier
│   └── abusive/                        # Abusive content model
├── topic-social-proof/                 # Topic relevance
├── simclusters-ann/                    # SimCluster nearest neighbor
├── tweet-mixer/                        # Tweet blending (20 feature files)
├── timelines/                          # Timeline data processing
├── science/                            # Science/search tools
└── ci/                                 # CI (minimal: exit 0)
```

### Architecture Deep Dive: Product Mixer Framework

Product Mixer is the foundational framework for all product pipelines. Key abstractions:

**Pipeline types:**
- `MixerPipeline` — top-level pipeline combining multiple candidate sources
- `CandidatePipeline` — produces candidates from a single source
- `RecommendationPipeline` — end-to-end recommendation pipeline
- `ScoringPipeline` — scoring-only pipeline

**Pipeline steps (from `product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/`):**
- `Step` trait — base for all pipeline steps
- `FailOpenPolicy` — determines behavior on candidate pipeline failure (Always, Never, or custom category set)
- `InvalidStepStateException` — thrown when a step's required data is missing
- `NewPipelineArrowBuilder` — constructs Stitch arrows for pipeline execution

**Component types (from `product-mixer/component-library/`):**
- Candidate sources
- Feature hydrators (query-level and candidate-level)
- Scorers (with common scorer abstractions)
- Filters
- Decorators (URT builders for promoted content, articles, etc.)
- Selectors and sorters
- Marshallers (URT, RTF, slice response formats)

**Observation:** The `product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer/` (3 files, 10.5K chars) provides standardized observability wrappers for pipeline components.

### Architecture Deep Dive: ANN Service

The ANN service (`ann/src/main/scala/com/twitter/ann/common/`) provides a generic interface for approximate nearest neighbor operations:

**Core abstractions (from `Api.scala`):**
- `EntityEmbedding[T]` — entity with associated embedding vector
- `Queryable[T, P, D]` — query interface with `query()` and `queryWithDistance()`
- `Appendable[T, P, D]` — mutable index with `append()` and `toQueryable()`
- `RuntimeParams` — algorithm-specific runtime parameters (e.g., HNSW ef, Annoy num_trees_to_search)
- `Distance[D]` — distance metric abstraction

**ID serialization (from `AnnInjections.scala`):**
- `LongInjection`, `StringInjection`, `IntInjection` — and their Java boxed variants
- Used for entity ID → byte array conversion in index storage

**Query server (from `ann/src/main/scala/com/twitter/ann/service/query_server/common/`, 9 files, 27K chars):**
- Finatra ThriftServer with standard filter chain
- Warmup handler for index pre-loading
- Supports multiple index types per server instance

### Architecture Deep Dive: Navi Model Serving

Navi is a Rust-based high-performance model serving system. Key components:

**Core types (from `navi/navi/src/`):**

| Module | Key Types | Purpose |
|---|---|---|
| `batch.rs` | `BatchPredictor` | Dynamic request batching |
| `bootstrap.rs` | `TensorInput`, `TensorInputEnum` | Model bootstrap and input types |
| `cli_args.rs` | `Args` | Command-line argument parsing (clap) |
| `lib.rs` | `TensorReturnEnum`, `PredictResult`, `TensorScores`, `PredictMessage`, `Callback` | Core prediction types |
| `predict_service.rs` | `PredictService` | Model management and routing |
| `tf_model.rs` | `TFModel`, `TFTensorEnum` | TensorFlow backend |
| `torch_model.rs` | `TorchModel` | PyTorch backend |
| `onnx_model.rs` | `OnnxModel` | ONNX Runtime backend |
| `metrics.rs` | `register_custom_metrics`, `metrics_handler` | Prometheus metrics |

**Data processing pipeline (from `navi/dr_transform/` and `navi/segdense/`):**

| Crate | Purpose |
|---|---|
| `dr_transform` | Converts Thrift `BatchPredictionRequest` to PyTorch tensors |
| `segdense` | Segment-dense feature transform (schema-driven densification) |
| `bpr_thrift` | Thrift parser for Batch Prediction Request protocol |

**Thrift data types (from `navi/thrift_bpr_adapter/thrift/src/`, 6 files, 127K chars):**
- `DataRecord` — feature container (binary, continuous, discrete, string, sparse features)
- `CompactDataRecord` — space-optimized variant
- `TensorRecord` — tensor-based feature container
- `BatchPredictionRequest` / `BatchPredictionResponse` — request/response envelopes
- Tensor types: `StringTensor`, `Int32Tensor`, `Int64Tensor`, `FloatTensor`, `DoubleTensor`, `BoolTensor`, `RawTypedTensor`, `BinaryTensor`
- Sparse tensor types: `COOSparseTensor`, `SparseTensor` (enum)

### Architecture Deep Dive: twml

twml is Twitter's ML library, split into Python and C++ layers:

**Python layer (`twml/twml/`, 17 files, 181K chars):**

| Module | Key Classes/Functions | Purpose |
|---|---|---|
| `argument_parser.py` | `SortingHelpFormatter`, `DefaultSubcommandArgParse` | CLI argument parsing (70 args) |
| `dataset.py` | `BlockFormatDataset` | TF Dataset from Twitter's block format |
| `hooks.py` | `StepProgressHook`, `EarlyStopHook`, `MetricsUpdateHook`, `EarlyStopDuration`, `StopAtStepHook` | TF SessionRunHooks for training lifecycle |
| `feature_config.py` | `FeatureConfig`, `FeatureConfigBuilder` | Feature specification and validation |
| `input_fns.py` | `default_input_fn` | TF Estimator input function builders |
| `learning_rate_decay.py` | (6 decay strategies) | Learning rate scheduling |
| `metrics.py` | (comprehensive metric library) | Evaluation metrics |
| `array.py` | `Array` | C++ array wrapper |
| `block_format_writer.py` | `BlockFormatWriter` | Block format file writer |
| `tensorio.py` | `TensorIO` | Tensor serialization I/O |
| `util.py` | (utility functions) | Checkpoint mgmt, feature conversion |
| `errors.py` | `EarlyStopError`, `CheckpointNotFoundError` | Custom exceptions |

**C++ layer (`twml/libtwml/`, 21 files, 112K chars + 8 internal files, 34K chars):**
- Native implementations for data format parsing, array operations, and performance-critical paths
- Exposed to Python via the `Array` and `BlockFormatWriter` wrapper classes

**Common utilities (`twml/twml_common/`, 3 files, 1.8K chars):**
- `PartitionInitializer` — Keras initializer from numpy arrays
- `serialize` / `deserialize` — object serialization helpers
- `create_sparse_tensor` / `create_reference_input` — sparse tensor construction

### Architecture Deep Dive: Follow Recommendations Service

The Follow Recommendations Service is the largest service by model complexity:

**Common models (`follow-recommendations-service/common/src/main/scala/.../common/models/`, 54 files, 78K chars):**
Define the data model for recommendation candidates, scores, signals, and display contexts.

**Candidate sources (16+ algorithms):** Each in its own directory under `common/candidate_sources/`, with dedicated README explaining the algorithm, data sources, and parameters.

**Ranking pipeline:**
1. Candidates from all sources are unified
2. Feature hydration from RealGraph, Feature Store, social graph
3. Multi-tower MLP model scores candidates (trained weekly on Profile Sidebar impression data)
4. Filtering by safety, dedup, business rules
5. Results published to Manhattan for serving

**Controllers (`follow-recommendations-service/server/src/main/scala/.../controllers/`, 5 files, 8.5K chars):**
Handle request routing, response assembly, and flow orchestration.

### Architecture Deep Dive: TweetyPie

TweetyPie is the tweet storage and hydration service — one of the largest services in the codebase:

- **Handlers** (`tweetypie/server/src/main/scala/.../handler/`, 49 files, 91K chars): Tweet CRUD operations, media handling, URL resolution, safety checks
- **Config** (`tweetypie/server/src/main/scala/.../config/`, 22 files, 245K chars): Service configuration, feature switches, dependency wiring
- **Core** (`tweetypie/server/src/main/scala/.../core/`, 15 files, 33K chars): Core tweet model, hydration, filtering
- **Observers** (`tweetypie/server/src/main/scala/.../service/observer/`, 11 files, 30K chars): Per-operation metrics and tracing
- **Servo** subsystem: Request handling, repository pattern, JSON serialization, decider integration

### Architecture Deep Dive: VisibilityLib

VisibilityLib evaluates content visibility rules:

- **Models** (`visibilitylib/src/main/scala/.../visibility/models/`, 24 files, 126K chars): Content labels, visibility actions, rule outcomes, abuse types, media labels, safety levels
- **Features** (`visibilitylib/src/main/scala/.../visibility/features/`, 4 files, 15K chars): Feature definitions for rule evaluation
- **Rules** (`visibilitylib/src/main/scala/.../visibility/rules/`): Rule definitions and evaluation logic
- **Builders** (`visibilitylib/src/main/scala/.../visibility/builder/`): Rule context construction

### Architecture Deep Dive: Push Service (Notifications)

The notification system uses a four-stage pipeline:

1. **Candidate generation** — multiple sources produce notification candidates
2. **Light ranking** — fast MLP pre-filter (`pushservice/src/main/python/models/light_ranking/`)
3. **Heavy ranking** — ClemNet multi-task model (`pushservice/src/main/python/models/heavy_ranking/`)
4. **Quality control** — policy predicates

**Heavy ranker architecture (ClemNet):**

```python
class ClemNet(tf.keras.layers.Layer):
    # Multi-block architecture with:
    # - KerasConv1D layers (1D convolution on feature groups)
    # - ChannelWiseDense layers (per-channel dense projection)
    # - ResidualLayer connections
    # - Block2 modules combining conv + dense + residual
```

Configuration via Pydantic-style params: `ClemNetParams`, `BlockParams`, `ConvParams`, `DenseParams`, `TopLayerParams`.

**Predicate system** (`pushservice/src/main/scala/.../pushservice/predicate/event/`, 1 file, 6.3K chars): Event-based filtering predicates for notification quality control.

**Config** (`pushservice/src/main/scala/.../pushservice/config/`, 5 files, 119K chars): The largest config module in the codebase, containing comprehensive notification pipeline configuration.

### Architecture Deep Dive: SimClusters

SimClusters maps users and content into a shared topic embedding space:

- **Summingbird common** (`src/scala/com/twitter/simclusters_v2/summingbird/common/`, 14 files, 51K chars): Real-time aggregation of cluster membership
- **Scalding embedding** (`src/scala/com/twitter/simclusters_v2/scalding/embedding/common/`, 4 files, 37K chars): Batch embedding generation
- **Scalding common** (`src/scala/com/twitter/simclusters_v2/scalding/common/`, 4 files, 14.6K chars): Shared Scalding utilities
- **Common** (`src/scala/com/twitter/simclusters_v2/common/`, 11 files, 43K chars): Core cluster models and utilities
- **Scio pipelines**: BigQuery generation and multi-type graph processing

### Architecture Deep Dive: Earlybird Internals

Earlybird's search index architecture (from `src/java/com/twitter/search/`):

- **Core** (`core/earlybird/`): Inverted index, posting lists, term dictionaries (FST and MPH), column-stride field indexes, facet counting
- **Config** (`earlybird/config/`, 7 files, 26K chars + `earlybird/common/config/`, 2 files, 25K chars): Schema definitions, tier configurations
- **Ingester** (`search/ingester/pipeline/app/`, 4 files, 13K chars): Kafka-to-index pipeline
- **Earlybird Root** (`earlybird_root/common/`, 9 files, 38K chars + `earlybird_root/config/`, 1 file, 1.5K chars): Request routing and response merging across partitions
- **Relevance** (`search/common/relevance/`, config + features): Search result scoring and feature extraction
- **Schema** (`search/common/schema/`, 9 files, 62K chars): Tweet field definitions, analyzers, and index mappings

### Code Patterns and Conventions

**Dependency injection:** All services use Google Guice via Finatra's module system. Services declare dependencies as `Module` sequences.

**RPC pattern:** Thrift-based RPC with Finagle. Services define Thrift IDL in `thrift/` directories, generate Scala/Java stubs, and implement controllers.

**Pipeline pattern:** Product Mixer's step-based pipeline with fail-open policies, quality factors, and arrow-based execution (Twitter's Stitch library).

**Configuration pattern:** YAML configs in `resources/config/` directories, loaded by service-specific config modules (Scala objects/classes).

**Testing pattern:** Test files co-located with source in parallel `test/` directories. Unified User Actions includes test configs for client validation.

**Error handling:**
- Custom exception types (`EarlyStopError`, `CheckpointNotFoundError`, `InvalidStepStateException`)
- Fail-open policies for non-critical pipeline stages
- Stitch-based error propagation in Product Mixer

### Testing Strategy

- **Unit tests:** Standard Scala/Java test frameworks, co-located in `src/test/`
- **Integration tests:** Service-level tests using Finatra's testing framework
- **Load tests:** ANN load test framework with configurable QPS, duration, and parameter grid search
- **ML model evaluation:** twml's `eval.steps` and `eval.period` for online evaluation during training
- **CI:** Minimal (`ci/ci.sh` exits 0); real validation through Bazel test targets

### Potential Improvements

Based on code analysis:

1. **CI pipeline** — The `ci/ci.sh` script is a no-op. A real CI pipeline running `./bazel test //...` on key targets would prevent regressions.
2. **Documentation** — `repo_docs.cbf` returned 0 files at the root level. A root-level README summarizing the architecture would help new contributors.
3. **Dependency versions** — Navi pins specific Git tags for `ort` crate; upgrading to ONNX Runtime 1.15+ would bring performance improvements.
4. **Python version** — twml and the Dockerfile pin Python 3.7, which reached end-of-life in June 2023. Migration to 3.10+ is recommended.
5. **Configuration** — Service configs are scattered across 12+ locations with no unified schema. A shared configuration framework would reduce duplication.
6. **Feature Store coupling** — Multiple services depend on the internal Feature Store client. Abstracting this behind an interface would improve testability.

### Contributing

The repository is read-only (archived). To study or adapt the code:

1. Fork the repository
2. Build individual services using Bazel or Cargo
3. Service-specific READMEs exist in many subsystem directories (especially `follow-recommendations-service/common/src/main/scala/.../candidate_sources/`)
4. The ANN subsystem has the most complete internal documentation with README.rst and README.md files

## 9. Integration Points

### Inter-Service Communication

All services communicate via **Thrift RPC** over Finagle with mTLS:

| From | To | Protocol | Purpose |
|---|---|---|---|
| Home Mixer | Earlybird | Thrift | In-network tweet retrieval |
| Home Mixer | CR-Mixer | Thrift | Out-of-network candidates |
| Home Mixer | Timeline Ranker | Thrift | Candidate scoring |
| Home Mixer | VisibilityLib | Library | Content filtering |
| Timeline Ranker | Graph Feature Service | Thrift | Graph features |
| Timeline Ranker | User Signal Service | Thrift | User signals |
| Follow Recs | RealGraph | Thrift/Data | Engagement graph |
| Follow Recs | Feature Store | Thrift | Feature hydration |
| Push Service | Navi | gRPC | Model inference |
| All Services | Manhattan | Thrift | Key-value storage |
| All Services | Kafka | Protocol | Event streaming |
| Batch Jobs | HDFS | Protocol | Data storage |

### Event Streaming (Kafka Topics)

Unified User Actions (UUA) standardizes event tracking across all surfaces:

- **Adapters** (`unified_user_actions/adapter/`): Convert product-specific events to UUA format
- **Kafka serialization** (`unified_user_actions/kafka/`): Custom Kafka SerDe with internal type handling
- **Enrichment** (`unified_user_actions/enricher/`): Post-processing enrichment of UUA events
- **Client** (`unified_user_actions/client/`): Client library with environment-specific config (staging, production)

### Embedding Infrastructure

| Component | Role |
|---|---|
| Representation Manager | Stores and versions embeddings, handles migration between versions |
| Representation Scorer | Computes similarity between embedding pairs |
| Feature Store | Serves embeddings as features for model inference |
| SimClusters | Generates topic embeddings via batch Scalding/Scio jobs |
| ANN Service | Indexes embeddings for nearest-neighbor queries |

The Representation Manager's migration module (`representation-manager/server/src/main/scala/.../migration/`, 1 file, 34K chars) handles version transitions for the embedding storage layer.

---

_Generated May 2026 from twitter/the-algorithm at c54bec0. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._