Twitter's "The Algorithm" — User Guide

Twitter's recommendation algorithm is the system that decides what content appears in your Home timeline, who shows up in "Who to Follow," what tweets surface in Search, and which notifications you receive. This repository is the open-sourced codebase behind those systems — a massive Scala/Java/Python/Rust monorepo containing the services, ML models, and data pipelines that power Twitter's core product experience.

1. Quick Start

What This Repository Contains

Twitter's "The Algorithm" is not a single program you run — it is the collection of interconnected backend services that together form Twitter's recommendation engine. The major subsystems are:

• Home Timeline Ranking — decides which tweets appear in "For You" and "Following" tabs
• Follow Recommendations — powers "Who to Follow" suggestions using graph algorithms and ML
• Tweet Search (Earlybird) — real-time search indexing and retrieval built on Apache Lucene
• Content Recommendations (CR-Mixer) — content-based tweet recommendations
• Push Notifications — the heavy/light ranking models that decide which notifications to send
• Trust & Safety Models — NSFW detection, toxicity classification, abusive content filtering
• ML Infrastructure — twml (Twitter ML library) and Navi (Rust model serving)

Key Features

• Real-time tweet indexing and retrieval across billions of tweets
• Multi-stage candidate generation → ranking → filtering pipeline
• Graph-based recommendation algorithms (RealGraph, SimClusters, SALSA)
• Approximate Nearest Neighbor (ANN) service supporting HNSW, Annoy, FAISS, and brute-force algorithms
• Content visibility rules engine with hundreds of safety and policy rules
• Distributed ML training framework with TensorFlow integration
• Multi-backend model serving (TensorFlow, PyTorch, ONNX) via Navi
• Unified user action tracking across all product surfaces

Prerequisites

• Java 11+ and Scala 2.12
• Bazel build system
• Python 3.7+ (for ML components)
• Rust toolchain (for Navi model serving)
• Access to internal Twitter infrastructure services (Aurora, HDFS, Manhattan, Kafka) for full operation

Important

This codebase was designed to run within Twitter's internal infrastructure. Many services depend on internal systems (Aurora for scheduling, Manhattan for storage, Kafka for streaming, Strato for data access) that are not publicly available. You can study the code and build individual components, but running the full system requires these backend dependencies.

Build a Component

./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder:indexbuilder-deploy --bundle-jvm-archive=zip

INFO: Build completed successfully, 1 total action

Tip

Start with a self-contained subsystem like navi/ (Rust, builds with Cargo) or twml/ (Python, installs with pip) if you want to run code rather than just study it.

What to Explore Next

• Home timeline ranking — start at home-mixer/ and timelineranker/
• Follow recommendations — start at follow-recommendations-service/
• Search — start at src/java/com/twitter/search/
• ML training — start at twml/
• Model serving — start at navi/

2. Installation

System Requirements

Clone the Repository

git clone https://github.com/twitter/the-algorithm.git
cd the-algorithm

Cloning into 'the-algorithm'...

Building Scala/Java Services

All Scala and Java services use Bazel. Each service has its own build target:

# Build the Follow Recommendations Service
./bazel build follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations:follow_recommendations

# Build the Graph Feature Service server
./bazel build graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server:server

Warning

This is a very large repository (7,651 files). Building the full repository with ./bazel build //... will require substantial memory and time. Build specific targets instead.

Building Navi (Rust Model Serving)

Navi supports three ML backends via Cargo feature flags:

cd navi/navi

# Build with TensorFlow backend
cargo build --release --features tf

# Build with PyTorch backend
cargo build --release --features torch

# Build with ONNX backend
cargo build --release --features onnx

Note

On Linux, the ONNX and TensorFlow builds automatically enable CUDA and TensorRT support. On other platforms, GPU acceleration features are excluded.

Building twml (Twitter ML Library)

cd twml
pip install -e .

The native C++ components in twml/libtwml/ require a C++ compiler:

cd twml/libtwml
make

Post-Install Verification

Verify Bazel can resolve the dependency graph:

./bazel query 'deps(//home-mixer/server/...)' --output label 2>/dev/null | head -5

Install Troubleshooting

3. Configuration

Configuration in this repository is service-specific. Each service maintains its own configuration files, typically in YAML format within server/src/main/resources/config/ directories.

Service Configuration Locations

twml Environment Variables

The ML training framework reads these environment variables:

Tip

For full configuration reference with every setting and default value, see the Admin Guide's Configuration Reference section.

4. Core Usage

Understanding the Recommendation Pipeline

Twitter's recommendation system follows a multi-stage pipeline pattern across all product surfaces:

User Request
    │
    ▼
┌─────────────────────┐
│ Candidate Generation │  ← Multiple sources produce thousands of candidates
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   Light Ranking     │  ← Fast model filters to hundreds
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   Heavy Ranking     │  ← Full model scores remaining candidates
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Filtering & Rules  │  ← Visibility rules, dedup, policy enforcement
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   Product Mixing    │  ← Final assembly for the product surface
└─────────────────────┘

Exploring the Home Timeline

The Home timeline ("For You" tab) pipeline is orchestrated by Home Mixer and uses Timeline Ranker for scoring:

1. Candidate generation — Earlybird retrieves in-network tweets; CR-Mixer provides out-of-network recommendations; SimClusters ANN finds topic-relevant content
2. Feature hydration — Graph Feature Service computes user-user graph features; User Signal Service provides behavioral signals
3. Ranking — Timeline Ranker applies the scoring model using features from multiple sources
4. Filtering — VisibilityLib applies safety rules and content policies
5. Assembly — Home Mixer assembles the final timeline using Product Mixer framework

Note

The Product Mixer framework (product-mixer/) is Twitter's generic pipeline framework. Home Mixer, Tweet Mixer, and other product surfaces are all built on top of it. The framework provides pipeline orchestration, feature hydration, scoring, and response marshalling.

Exploring Follow Recommendations ("Who to Follow")

The Follow Recommendations Service generates account suggestions through multiple candidate sources:

After candidate generation, candidates flow through ranking (multi-tower MLP model trained on Profile Sidebar impression data) and filtering before display.

Tip

Each candidate source has its own README in follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/. These contain detailed algorithm explanations with academic references.

Exploring Tweet Search (Earlybird)

Earlybird is Twitter's real-time search system, built on Apache Lucene:

• Realtime cluster — indexes all public tweets from the past ~7 days
• Protected cluster — indexes protected tweets for the same timeframe
• Archive cluster — indexes all tweets ever posted, up to ~2 days ago

Each cluster is partitioned and replicated. Earlybird roots fan out requests to partitions and merge responses.

Key data flow:

Tweets → Kafka → Ingesters → Extract fields/features → Kafka → Earlybird (index + serve)
                                                                     ↑
Feature Update Service → engagement counts (likes, retweets, replies) ┘

Note

The Earlybird system uses a distributed inverted index with incremental (real-time) indexing. Its single-writer, multiple-reader architecture enables concurrent updates and queries while maintaining low latency. See the original Earlybird paper for the foundational design.

Using the twml Training Framework

twml is Twitter's ML training library built on TensorFlow. It provides a Trainer class with standardized training, evaluation, and export workflows.

python my_model.py \
  --save_dir hdfs:///user/$USER/my_model \
  --train.batch_size 32 \
  --train.learning_rate 0.002 \
  --train.steps 10000 \
  --eval.steps 1000 \
  --optimizer Adam

Warning

The --distributed flag enables distributed training but is incompatible with early stopping (--train.early_stop_patience). Do not use both simultaneously.

Using Navi for Model Serving

Navi is a high-performance Rust-based model inference server that supports TensorFlow, PyTorch, and ONNX models. It serves predictions via gRPC using the TensorFlow Serving API protocol.

Run Navi with a specific backend:

# TensorFlow backend
./navi --model-dir /path/to/models --config /path/to/config.json

# PyTorch backend
./navi_torch --model-dir /path/to/models --config /path/to/config.json

# ONNX backend
./navi_onnx --model-dir /path/to/models --config /path/to/config.json

Navi supports:

• Automatic model version detection and hot-reloading
• Batched prediction with configurable batch sizes and timeouts
• Prometheus metrics export via HTTP
• mTLS with x509 certificate verification
• Multiple model serving from a single instance

5. Common Tasks

Navigate the Codebase by Subsystem

Goal: Find the code responsible for a specific product feature.

1. Identify the subsystem from the top-level directories:

2. Within each subsystem, follow the standard directory structure:
   • server/src/main/scala/ — service implementation
   • server/src/main/resources/config/ — configuration
   • common/ — shared models and utilities
   • thrift/ — Thrift IDL definitions
   • client/ — client libraries

Build and Bundle a Service for Deployment

Goal: Create a deployable artifact for a service.

./bazel bundle follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations:follow_recommendations-deploy --bundle-jvm-archive=zip

INFO: Build completed successfully

The output bundle appears in dist/.

Build an ANN Index

Goal: Build an Approximate Nearest Neighbor index from embeddings on HDFS.

export JOB_NAME=ann_index_builder
export OUTPUT_PATH=hdfs:///user/$USER/${JOB_NAME}_test

aurora job create smf1/$USER/devel/$JOB_NAME \
  ann/src/main/aurora/index_builder/aurora_builder.aurora \
  --bind=profile.name=$JOB_NAME \
  --bind=profile.role=$USER \
  --bind=profile.output_dir=$OUTPUT_PATH \
  --bind=profile.entity_kind=user \
  --bind=profile.embedding_args='--input.embedding_format tab --input.embedding_path /path/to/embeddings' \
  --bind=profile.num_dimensions=300 \
  --bind=profile.algo=hnsw \
  --bind=profile.ef_construction=200 \
  --bind=profile.max_m=16 \
  --bind=profile.expected_elements=10000000 \
  --bind=profile.metric=InnerProduct \
  --bind=profile.concurrency_level=32 \
  --bind=profile.hadoop_cluster=dw2-smf1

Note

ANN supports three index algorithms: HNSW (Cosine, L2, InnerProduct), Annoy (Cosine, L2), and brute_force (Cosine, L2, InnerProduct). FAISS is also available for load testing.

Train an ML Model with twml

Goal: Train a ranking model using the twml framework.

1. Define your model class extending twml's Trainer
2. Configure training parameters:

python train.py \
  --save_dir hdfs:///user/$USER/model_output \
  --train.batch_size 64 \
  --train.learning_rate 0.001 \
  --train.max_steps 50000 \
  --eval.batch_size 64 \
  --eval.period 600 \
  --eval.delay 120 \
  --optimizer Adam \
  --config.save_checkpoints_secs 600 \
  --config.keep_checkpoint_max 20

3. Monitor training via TensorBoard (enabled by default on an auto-assigned port)
4. Export the trained model for serving

If training stalls: Check --train.early_stop_patience and --train.early_stop_tolerance settings. Use --train.log_metrics for debugging (note: this slows training by 10-20%).

Run a Load Test on an ANN Query Service

Goal: Benchmark ANN query service performance.

aurora job create smf1/$USER/staging/ann-loadtest \
  ann/src/main/aurora/loadtest/loadtest.aurora \
  --bind=profile.name=ann-loadtest \
  --bind=profile.role=$USER \
  --bind=profile.duration_sec=10 \
  --bind=profile.number_of_neighbors=10 \
  --bind=profile.qps=200 \
  --bind=profile.algo=hnsw \
  --bind=profile.metric=Cosine \
  --bind=profile.index_id_type=int \
  --bind=profile.hnsw_ef=400,600,800 \
  --bind=profile.embedding_dimension=3 \
  --bind=profile.concurrency_level=8 \
  --bind=profile.loadtest_type=remote \
  --bind=profile.service_destination=/srv#/staging/local/$USER/ann-server-test \
  --bind=profile.with_random_queries=True \
  --bind=profile.random_queries_count=50000

Load test results are printed to the Aurora job's stdout.

Compute Feature Importance

Goal: Determine which features matter most in a trained model.

python my_model.py \
  --feature_importance.algorithm tree \
  --feature_importance.sensitivity 0.03 \
  --feature_importance.example_count 10000 \
  --feature_importance.metric roc_auc \
  --feature_importance.is_metric_larger_the_better \
  --feature_importance.data_dir hdfs:///user/$USER/eval_data

Tip

The tree algorithm is faster for models with many features — it groups features by name prefix and only expands groups whose removal impacts the metric above the sensitivity threshold. Use serial only when you need exact per-feature importance values.

Generate Sample Query Sets for ANN Benchmarking

Goal: Create query, index, and truth sets for recall benchmarking.

./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline:ann-offline-deploy

export QUERY_EMBEDDINGS_PATH=/user/cortex-mlx/official_examples/ann/embeddings
export INDEX_EMBEDDINGS_PATH=/user/cortex-mlx/official_examples/ann/embeddings
export METRIC=InnerProduct
export NEIGHBOURS=10

oscar hdfs \
  --screen --tee log.txt \
  --bundle ann-offline-deploy \
  --host hadoopnest1.smf1.twitter.com \
  --tool com.twitter.ann.scalding.offline.KnnTruthSetGenerator -- \
  --neighbors $NEIGHBOURS \
  --metric $METRIC \
  --query_entity_kind user \
  --query.embedding_path $QUERY_EMBEDDINGS_PATH \
  --query.embedding_format tab \
  --query_sample_percent 50.0 \
  --index_entity_kind user \
  --index.embedding_path $INDEX_EMBEDDINGS_PATH \
  --index.embedding_format tab \
  --index_sample_percent 90.0 \
  --query_set_output.embedding_path /user/$USER/query_set \
  --query_set_output.embedding_format tab \
  --index_set_output.embedding_path /user/$USER/index_set \
  --index_set_output.embedding_format tab \
  --truth_set_output_path /user/$USER/truth_set \
  --reducers 100

6. Troubleshooting

Error: ValueError: Path for zookeeper dataset pointer is None

Cause: twml's BlockFormatDataset requires a ZooKeeper path for the dataset pointer, but none was specified.

Solution:

1. Ensure you pass a valid ZooKeeper dataset path in your data pipeline configuration
2. If running locally without ZooKeeper, provide data via a local file path instead

Prevention: Always specify --data_dir or the appropriate dataset path argument when running training.

Error: CheckpointNotFoundError: ... not found

Cause: The training framework cannot locate a checkpoint at the specified --save_dir path.

Solution:

1. Verify the --save_dir path exists and is accessible (check HDFS permissions if using HDFS)
2. If starting fresh, use --overwrite_save_dir to skip checkpoint restoration

Prevention: Ensure HDFS paths use the correct prefix format (hdfs:///user/...).

Error: ModuleNotFoundError: Tensorflow has not found the GPU

Cause: The trust and safety model training scripts require GPU access via TensorFlow, but CUDA is not properly configured.

Solution:

1. Verify CUDA toolkit installation: nvidia-smi
2. Ensure TensorFlow GPU package is installed: pip install tensorflow-gpu
3. Check LD_LIBRARY_PATH includes CUDA libraries

Prevention: Use the provided Dockerfile at ann/src/main/python/dataflow/worker_harness/Dockerfile as a reference for a working CUDA environment (CUDA 11.2, cuDNN 8, Ubuntu 20.04).

Error: PEM parsing failed (Navi)

Cause: Navi's mTLS bootstrap cannot parse the provided PEM certificate file.

Solution:

1. Verify the certificate file is valid PEM format: openssl x509 -in cert.pem -text -noout
2. Ensure the file path in the Navi configuration is correct
3. Check file permissions allow the Navi process to read the certificate

Prevention: Always validate certificates before deploying model-serving instances.

Error: model sync cli empty (Navi)

Cause: Navi's predict service could not find a model sync command in the configuration.

Solution:

1. Verify the model configuration JSON includes the model_dir field with valid paths
2. Ensure model directories contain versioned subdirectories with the expected model artifacts

Prevention: Use the config validation in navi/navi/src/lib.rs (read_config) to check configuration before starting the service.

Error: unsupport input tensor type (Navi)

Cause: The input tensor type does not match what the model expects.

Solution:

1. Check the model's input signature to verify expected tensor types
2. Ensure the Thrift BatchPredictionRequest contains features in the correct format (see DataRecord structure in navi/thrift_bpr_adapter/)

Warning

Navi's error messages use Rust's {:?} debug formatting. The actual error will include the mismatched type information — read the full message carefully.

Error: Unsupported params.learning_rate_decay

Cause: An invalid learning rate decay schedule name was passed to twml.

Solution: Use one of the supported schedules:

• exponential (requires --decay_steps, --exponential_decay_rate)
• polynomial (requires --decay_steps, --end_learning_rate)
• piecewise_constant (requires boundary and value lists)
• inverse (requires --decay_steps, --decay_rate, --min_learning_rate)
• cosine_decay (requires --decay_steps, --alpha)
• cosine_restarts_decay (requires --first_decay_steps, --t_mul, --m_mul, --alpha)

Prevention: Verify the schedule name and all required companion arguments before launching training.

Bazel Build Fails with Memory Errors

Cause: The repository is very large (7,651 files) and Bazel's analysis phase can exceed available memory.

Solution:

1. Increase Bazel's JVM heap: add startup --host_jvm_args=-Xmx8g to .bazelrc
2. Build individual targets rather than entire directory trees
3. Use --local_ram_resources to limit parallel actions

Prevention: Build specific service targets rather than using broad patterns like //....

Service Fails at Startup: Missing Internal Dependencies

Cause: Services depend on internal Twitter infrastructure (Aurora, Manhattan, Strato, Kafka, ZooKeeper) that is not available in the open-source distribution.

Solution:

1. This is expected — the repository was open-sourced for transparency, not for standalone deployment
2. Study the service code and configuration to understand the architecture
3. For individual components (Navi, twml, trust_and_safety_models), local execution is possible with appropriate data substitution

Caution

Do not attempt to run the full service mesh without the internal infrastructure dependencies. The services will fail at startup when they cannot connect to Aurora, Manhattan, or internal Kafka clusters.

Error: This loss name is not valid (Trust & Safety)

Cause: An invalid loss function name was passed to the toxicity model trainer.

Solution: Use one of: BCE, masked BCE, CCE, sCCE, Focal_BCE, inv_KL_loss.

Prevention: Check trust_and_safety_models/toxicity/load_model.py for the current list of supported loss functions.

Error: This is not an experiment id as defined in the settings file

Cause: The toxicity trainer received an experiment ID that does not exist in the settings configuration.

Solution:

1. Check trust_and_safety_models/toxicity/settings/ for available experiment definitions
2. Ensure the experiment ID string matches exactly

Prevention: List available experiment IDs from the settings file before launching training.

7. FAQ

Q: Can I run these services locally?

Most services cannot run standalone because they depend on Twitter's internal infrastructure (Aurora for deployment, Manhattan for storage, Kafka for streaming, Strato for data access). However, individual components like Navi (model serving), twml (ML training), and the trust_and_safety_models can be run independently with appropriate data inputs.

Q: What build system does this use?

Bazel for all Scala/Java services. Rust components (Navi, segdense, dr_transform) use Cargo. Python components (twml, trust_and_safety_models, pushservice models) use standard pip/setuptools.

Q: What is the Product Mixer framework?

Product Mixer (product-mixer/) is Twitter's internal framework for building product recommendation pipelines. It provides standardized abstractions for pipeline orchestration, candidate generation, feature hydration, scoring, filtering, and response marshalling. Home Mixer, Tweet Mixer, and other product surfaces are built on top of it.

Q: How does the "For You" tab work at a high level?

The For You tab combines in-network tweets (from accounts you follow, retrieved by Earlybird) with out-of-network recommendations (from CR-Mixer and SimClusters ANN). These candidates pass through feature hydration (Graph Feature Service, User Signal Service), heavy ranking (Timeline Ranker), visibility filtering (VisibilityLib), and final assembly (Home Mixer).

Q: What are SimClusters?

SimClusters is Twitter's topic clustering system. It maps users and tweets into a shared embedding space based on engagement patterns, enabling topic-based recommendations. The simclusters_v2 directory contains embedding generation pipelines (Scalding/Scio batch jobs) and the simclusters-ann service provides nearest-neighbor lookups on these embeddings.

Q: What distance metrics does the ANN service support?

InnerProduct, Cosine, and L2. Not all algorithms support all metrics: HNSW and brute_force support all three; Annoy supports only Cosine and L2.

Q: What ML frameworks are used?

TensorFlow is the primary ML framework, accessed through Twitter's twml library. PyTorch and ONNX are supported via Navi's runtime backends. The trust and safety models use Keras (TensorFlow). The push notification heavy ranker uses a custom ClemNet architecture built with tf.keras.layers.

Q: What is the VisibilityLib?

VisibilityLib (visibilitylib/) is Twitter's content moderation rules engine. It evaluates hundreds of visibility rules against content and users to determine what should be shown, filtered, or hidden. Rules cover safety policies, legal requirements, and product-specific display logic. The visibilitylib/src/main/scala/com/twitter/visibility/models/ directory contains 24 model files defining content labels, actions, and rule outcomes.

Q: What is RealGraph?

RealGraph is a user-user graph dataset that measures relationship strength between users based on real-time and historical engagement signals (favorites, replies, retweets, clicks, follows, mutes, blocks). It powers follow recommendations, timeline ranking, and various feature hydration pipelines. Based on the paper "Realgraph: User interaction prediction at Twitter" (KDD'14 UEO workshop).

Q: What entity types does the ANN service support?

The ANN service supports embeddings indexed by: user, tweet, word, url, and tfwId entity types. The entity kind determines how IDs are serialized and stored in the index.

Q: How do push notifications get ranked?

Push notifications use a four-stage pipeline: candidate generation → light ranking (fast MLP to pre-filter) → heavy ranking (ClemNet multi-task model predicting open + engage probabilities) → quality control. The models are in pushservice/src/main/python/models/.