Twitter's "The Algorithm" — Administrator & Developer Guide

This guide covers the architecture, internals, configuration, deployment patterns, and code structure of Twitter's open-sourced recommendation algorithm. It is written for platform engineers, ML engineers, and architects who need to understand how these systems were designed and operated, either for study, adaptation, or reimplementation.

1. Architecture Overview

What the System Does

Twitter's recommendation algorithm is a distributed system of ~20 interconnected microservices that collectively decide what content users see across Home Timeline, Search, Notifications, and Who to Follow. The system processes billions of tweets and hundreds of millions of user interactions to produce personalized feeds in real time.

The architecture follows a multi-stage candidate generation → ranking → filtering pattern, where each stage progressively narrows and scores content. Services communicate primarily via Thrift RPC (using Twitter's Finatra/Finagle stack), with Kafka for event streaming and HDFS/Manhattan for persistent storage.

System Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        PRODUCT LAYER                             │
│  ┌─────────────┐  ┌─────────────┐  ┌────────────┐  ┌─────────┐ │
│  │ Home Mixer   │  │ Tweet Mixer │  │ Push Svc   │  │ Search  │ │
│  │ (For You/    │  │ (content    │  │ (notif     │  │ Root    │ │
│  │  Following)  │  │  blending)  │  │  ranking)  │  │         │ │
│  └──────┬───────┘  └──────┬──────┘  └─────┬──────┘  └────┬────┘ │
│         │                 │               │               │      │
│  ┌──────┴─────────────────┴───────────────┴───────────────┘      │
│  │           Product Mixer Framework                             │
│  │     (pipeline orchestration, feature hydration, scoring)      │
│  └──────┬────────────────────────────────────────────────────┘   │
└─────────┼────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼          RECOMMENDATION LAYER                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Timeline     │  │ CR-Mixer     │  │ Follow Recs  │           │
│  │ Ranker       │  │ (content     │  │ Service      │           │
│  │ (scoring)    │  │  candidates) │  │ (who to      │           │
│  │              │  │              │  │  follow)     │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ SimClusters  │  │ Topic Social │  │ Recos        │           │
│  │ ANN          │  │ Proof        │  │ Injector     │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
└──────────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼           FEATURE & SIGNAL LAYER                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Graph Feature│  │ User Signal  │  │ Representation│          │
│  │ Service      │  │ Service      │  │ Manager/Scorer│          │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
│  ┌──────────────┐  ┌──────────────┐                              │
│  │ Unified User │  │ VisibilityLib│                              │
│  │ Actions      │  │ (moderation) │                              │
│  └──────────────┘  └──────────────┘                              │
└──────────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼           DATA & ML LAYER                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Earlybird    │  │ TweetyPie    │  │ ANN Service  │           │
│  │ (search      │  │ (tweet       │  │ (nearest     │           │
│  │  index)      │  │  storage)    │  │  neighbor)   │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │ Navi (model  │  │ twml (ML     │  │ Trust &      │           │
│  │  serving)    │  │  training)   │  │ Safety Models│           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
└──────────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────────────────────────────────────────────┐
│         ▼           INFRASTRUCTURE                               │
│  Aurora (scheduling) │ Kafka (streaming) │ HDFS (storage)        │
│  Manhattan (KV)      │ Strato (data)     │ ZooKeeper (coord)     │
│  Thrift (RPC)        │ Finagle (network) │ Finatra (framework)   │
└──────────────────────────────────────────────────────────────────┘

Note

All Scala/Java services in this repository use Twitter's Finatra framework for HTTP/Thrift server lifecycle, dependency injection (via Google Guice), and request filtering. The Finagle library provides the underlying RPC, load balancing, and connection pooling.

Component Inventory

Technology Stack

Data Flow: Home Timeline Request

1. User opens For You tab
2. Home Mixer receives request
3. Product Mixer orchestrates pipeline:
   a. Earlybird → in-network tweet candidates (recent tweets from followed accounts)
   b. CR-Mixer → out-of-network candidates (content recs from multiple signals)
   c. SimClusters ANN → topic-relevant candidates
4. Feature hydration:
   a. Graph Feature Service → user-user graph features
   b. User Signal Service → engagement history, behavioral signals
   c. Representation Manager → user/tweet embeddings
5. Timeline Ranker scores all candidates with heavy ranking model
6. VisibilityLib filters by safety rules, content policy
7. Home Mixer assembles final timeline with mixing rules
8. Response returned to client

2. System Requirements

Runtime Infrastructure Dependencies

Hardware Requirements (Per Service)

Based on Aurora job configurations in the ground truth:

Important

The ANN Dataflow worker Dockerfile specifies CUDA 11.2.2 with cuDNN 8 on Ubuntu 20.04 as the reference GPU environment. Python 3.7 is pinned explicitly.

3. Configuration Reference

Navi Model Serving Configuration

Navi reads a JSON configuration file specifying model locations and runtime parameters:

{
  "model_dir": ["/path/to/model1", "/path/to/model2"],
  "inter_op_parallelism": 1,
  "intra_op_parallelism": 4
}

Key configuration functions (from navi/navi/src/lib.rs):

Navi CLI arguments (from navi/navi/src/cli_args.rs):

twml Training Configuration (Complete CLI Reference)

All 70 CLI parameters extracted from twml/twml/argument_parser.py:

Core Training Parameters

Evaluation & Early Stopping

Warning

Early stopping (--train.early_stop_patience) cannot be used with --distributed mode. Attempting to combine them will produce undefined behavior.

Distributed Training

Checkpointing & Monitoring

Gradient & Optimization

Learning Rate Decay

Supported decay schedules: exponential, polynomial, piecewise_constant, inverse, cosine_decay, cosine_restarts_decay.

Feature Importance

ANN Index Builder Configuration

The ANN index builder accepts these Aurora profile parameters:

HNSW-specific:

Annoy-specific:

FAISS-specific (load testing only):

Note

FAISS parameters accept comma-separated lists for grid search during load testing (e.g., --bind=profile.faiss_nprobe=1,3,9,27,81,128).

Environment Variables

4. Deployment & Scaling

Deployment Model

All production services deploy on Apache Aurora (a Mesos scheduler). Aurora job configurations are defined in .aurora files using Thermos templates.

Deployment pattern:

1. Build service bundle with Bazel: ./bazel bundle <target>:deploy --bundle-jvm-archive=zip
2. Upload bundle to Packer (Twitter's artifact storage)
3. Create Aurora job: aurora job create <cluster>/<role>/<env>/<name> <config>.aurora

Service Architecture Patterns

All Scala services follow a consistent pattern using Finatra:

object Main extends ServerMain

class ServerMain extends ThriftServer with Mtls {
  override val name = "service-name"
  override val modules: Seq[Module] = Seq(/* Guice modules */)

  override def configureThrift(router: ThriftRouter): Unit = {
    router
      .filter[LoggingMDCFilter]
      .filter[TraceIdMDCFilter]
      .filter[ThriftMDCFilter]
      .filter[StatsFilter]
      .add[ServiceController]
  }

  override protected def warmup(): Unit = {
    handle[WarmupHandler]()
  }
}

Standard filter chain: LoggingMDCFilter → TraceIdMDCFilter → ThriftMDCFilter → AccessLoggingFilter → StatsFilter → controller.

Earlybird Cluster Topology

Earlybird splits the tweet index across three cluster types:

Each cluster uses multiple partitions with replication. Earlybird roots fan out queries across partitions and merge responses.

Indexing pipeline:

• Ingesters consume from Kafka, extract fields/features, write to intermediate Kafka topics
• Earlybird instances consume from intermediate topics and update in-memory Lucene indexes
• Feature Update Service pushes real-time engagement counts (likes, retweets, replies)

Navi Deployment

Navi compiles to three separate binaries depending on ML backend:

Each binary serves predictions via the TensorFlow Serving gRPC protocol (prediction_service.proto), with support for the KFServing v2 inference protocol.

Tip

Navi supports hot model reloading. When a new model version directory appears under the configured model_dir, Navi automatically detects the version timestamp and loads the new model without restart.

5. Operations

Monitoring

Navi exposes Prometheus metrics via an HTTP endpoint (Warp-based). Key metrics:

Scala services use Finagle's built-in StatsReceiver for metrics, exposed via the standard Finagle admin interface.

Note

All Scala services include StatsFilter in their Thrift filter chain, which automatically tracks request rate, latency percentiles, and error rates per endpoint.

Feature Flags (Deciders)

Services use Twitter's Decider system for feature flags and gradual rollouts. Decider modules are injected via Guice:

override val modules: Seq[Module] = Seq(
  DeciderModule,
  // ...
)

Deciders control traffic routing, experiment allocation, and feature availability without redeployment.

Health Checks

Finatra services expose health endpoints via the admin HTTP interface. twml training jobs expose health on a configurable --health_port for graceful shutdown coordination.

Tip

For twml distributed training, the --disable_auto_ps_shutdown flag prevents the parameter server from shutting down prematurely when individual workers fail. Use this for fault-tolerant long-running jobs.

Batch Processing Operations

SimClusters, interaction graphs, and embedding generation run as batch Scalding/Scio jobs on Hadoop:

# Example: Run embedding sampling job
oscar hdfs \
  --screen --tee log.txt \
  --hadoop-client-memory 6000 \
  --bundle embeddingformat-deploy \
  --host hadoopnest1.smf1.twitter.com \
  --tool com.twitter.scalding.entityembeddings.util.EmbeddingFormat.EmbeddingSamplingJob -- \
  --entity_kind user \
  --input.embedding_path $INPUT_PATH \
  --input.embedding_format usertensor \
  --output.embedding_path $OUTPUT_PATH \
  --output.embedding_format tab \
  --sample_percent 0.1

Supported embedding formats: usertensor, usercontinuous, comma, tab.

6. Performance

Earlybird Optimizations

• Incremental indexing: New tweets are indexed in real-time as they arrive from Kafka, maintaining index freshness
• Single-writer, multiple-reader: Eliminates write contention while supporting high query concurrency
• Partitioned index: Horizontal scaling across shards, with roots handling fan-out and merge
• Column-stride field indexes: Optimized for per-document feature lookups (engagement counts, timestamps)

Navi Batching

Navi implements dynamic batching to maximize GPU/CPU utilization:

• Incoming requests are buffered in a BatchPredictor
• Batches are flushed when either the size limit or timeout is reached (duration_past check)
• Responses are demultiplexed back to individual request channels

Product Mixer Quality Factors

Product Mixer pipelines support quality factors — runtime parameters that trade off result quality for latency:

// From pipeline framework
class QualityFactorStatus {
  // Dynamically adjust pipeline behavior based on latency targets
}

When latency increases, quality factors can reduce candidate counts, simplify feature hydration, or skip expensive scoring stages.

7. Security

Authentication & Authorization

• All Scala services use mTLS (mutual TLS) via Finatra's Mtls trait
• Navi supports mTLS with x509 certificate verification (x509-parser crate)
• Inter-service communication is authenticated via Thrift client IDs (ThriftClientIdModule)

Warning

The open-source release does not include the internal certificate infrastructure. Services expect valid mTLS certificates for production operation. For local development, you would need to set up a local CA or disable mTLS.

Content Safety

VisibilityLib is the centralized content moderation engine. It evaluates rules against:

• Content labels — safety classifications applied to tweets and users
• Visibility actions — outcomes like hide, filter, tombstone, interstitial
• Rule conditions — composable predicates combining labels, user state, and context

The visibilitylib/src/main/scala/com/twitter/visibility/models/ directory (24 files, 126K chars) defines the complete vocabulary of content labels and actions.

Trust & Safety ML Models

The toxicity trainer supports multiple loss functions: BCE, masked BCE, CCE, sCCE, Focal_BCE, inv_KL_loss, with configurable optimizers and multi-fold training.

Caution

The trust and safety models are trained on sensitive data. The model architectures and training code are included, but training data is not part of the open-source release.

8. Developer Internals

Repository Structure

the-algorithm/                          # Root (7,651 files, 12.3M source chars)
├── home-mixer/                         # Home timeline product pipeline
│   └── server/src/main/scala/          # Scala service implementation
├── timelineranker/                     # Tweet ranking service
│   ├── server/                         # Server with config and Scala source
│   ├── client/                         # Client library
│   └── common/                         # Shared models
├── follow-recommendations-service/     # Who to Follow
│   ├── server/                         # Server implementation
│   ├── common/                         # 54+ model files, candidate sources
│   └── thrift/                         # Thrift service definitions
├── cr-mixer/                           # Content recommendation mixer
│   ├── server/                         # Server with 9 core modules
│   └── thrift/                         # Thrift definitions
├── product-mixer/                      # Generic pipeline framework
│   ├── core/                           # Pipeline, model, feature, controller
│   ├── component-library/              # Reusable components (scorers, features)
│   └── shared-library/                 # Shared utilities (observers)
├── tweetypie/                          # Tweet storage service
│   ├── server/                         # 49 handler files, 22 config files
│   ├── common/                         # Shared types
│   └── servo/                          # Request/repo/json/decider subsystems
├── pushservice/                        # Push notification service
│   └── src/main/python/models/         # Heavy/light ranking models
├── visibilitylib/                      # Content moderation
│   └── src/main/scala/                 # 24 model files, features, rules, builders
├── src/                                # Core shared source
│   ├── java/com/twitter/search/        # Earlybird search (Java)
│   ├── scala/com/twitter/              # SimClusters, interaction graph, recos
│   ├── python/                         # Deepbird timeline models
│   └── thrift/                         # Shared Thrift definitions
├── twml/                               # Twitter ML library
│   ├── twml/                           # Python package (17 files, 180K chars)
│   ├── twml_common/                    # Common utilities
│   └── libtwml/                        # C++ native library (21 files, 112K chars)
├── navi/                               # ML model serving (Rust)
│   ├── navi/                           # Main server (9 src files, 100K chars)
│   ├── dr_transform/                   # Data record transformer
│   ├── segdense/                       # Feature densification
│   └── thrift_bpr_adapter/             # Thrift BPR parsing (127K chars)
├── ann/                                # Approximate Nearest Neighbor
│   └── src/main/scala/com/twitter/ann/ # Common, query server, scalding
├── graph-feature-service/              # Graph features
│   └── src/main/scala/                 # Server + worker architecture
├── user-signal-service/                # User signals
│   ├── server/                         # 18 signal types
│   └── thrift/                         # Service definitions
├── unified_user_actions/               # Event tracking
│   ├── adapter/                        # Event adapters
│   ├── kafka/                          # Kafka serialization
│   ├── enricher/                       # Event enrichment
│   ├── service/                        # UUA service
│   └── client/                         # Client library
├── representation-manager/             # Embedding management
│   ├── server/                         # Migration, config, common
│   └── client/                         # Client with config
├── representation-scorer/              # Embedding scoring
├── recos-injector/                     # Recommendation injection
├── trust_and_safety_models/            # Safety ML models
│   ├── nsfw/                           # NSFW classifier
│   ├── toxicity/                       # Toxicity classifier
│   └── abusive/                        # Abusive content model
├── topic-social-proof/                 # Topic relevance
├── simclusters-ann/                    # SimCluster nearest neighbor
├── tweet-mixer/                        # Tweet blending (20 feature files)
├── timelines/                          # Timeline data processing
├── science/                            # Science/search tools
└── ci/                                 # CI (minimal: exit 0)

Architecture Deep Dive: Product Mixer Framework

Product Mixer is the foundational framework for all product pipelines. Key abstractions:

Pipeline types:

• MixerPipeline — top-level pipeline combining multiple candidate sources
• CandidatePipeline — produces candidates from a single source
• RecommendationPipeline — end-to-end recommendation pipeline
• ScoringPipeline — scoring-only pipeline

Pipeline steps (from product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/):

• Step trait — base for all pipeline steps
• FailOpenPolicy — determines behavior on candidate pipeline failure (Always, Never, or custom category set)
• InvalidStepStateException — thrown when a step's required data is missing
• NewPipelineArrowBuilder — constructs Stitch arrows for pipeline execution

Component types (from product-mixer/component-library/):

• Candidate sources
• Feature hydrators (query-level and candidate-level)
• Scorers (with common scorer abstractions)
• Filters
• Decorators (URT builders for promoted content, articles, etc.)
• Selectors and sorters
• Marshallers (URT, RTF, slice response formats)

Observation: The product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer/ (3 files, 10.5K chars) provides standardized observability wrappers for pipeline components.

Architecture Deep Dive: ANN Service

The ANN service (ann/src/main/scala/com/twitter/ann/common/) provides a generic interface for approximate nearest neighbor operations:

Core abstractions (from Api.scala):

• EntityEmbedding[T] — entity with associated embedding vector
• Queryable[T, P, D] — query interface with query() and queryWithDistance()
• Appendable[T, P, D] — mutable index with append() and toQueryable()
• RuntimeParams — algorithm-specific runtime parameters (e.g., HNSW ef, Annoy num_trees_to_search)
• Distance[D] — distance metric abstraction

ID serialization (from AnnInjections.scala):

• LongInjection, StringInjection, IntInjection — and their Java boxed variants
• Used for entity ID → byte array conversion in index storage

Query server (from ann/src/main/scala/com/twitter/ann/service/query_server/common/, 9 files, 27K chars):

• Finatra ThriftServer with standard filter chain
• Warmup handler for index pre-loading
• Supports multiple index types per server instance

Architecture Deep Dive: Navi Model Serving

Navi is a Rust-based high-performance model serving system. Key components:

Core types (from navi/navi/src/):

Data processing pipeline (from navi/dr_transform/ and navi/segdense/):

Thrift data types (from navi/thrift_bpr_adapter/thrift/src/, 6 files, 127K chars):

• DataRecord — feature container (binary, continuous, discrete, string, sparse features)
• CompactDataRecord — space-optimized variant
• TensorRecord — tensor-based feature container
• BatchPredictionRequest / BatchPredictionResponse — request/response envelopes
• Tensor types: StringTensor, Int32Tensor, Int64Tensor, FloatTensor, DoubleTensor, BoolTensor, RawTypedTensor, BinaryTensor
• Sparse tensor types: COOSparseTensor, SparseTensor (enum)

Architecture Deep Dive: twml

twml is Twitter's ML library, split into Python and C++ layers:

Python layer (twml/twml/, 17 files, 181K chars):

C++ layer (twml/libtwml/, 21 files, 112K chars + 8 internal files, 34K chars):

• Native implementations for data format parsing, array operations, and performance-critical paths
• Exposed to Python via the Array and BlockFormatWriter wrapper classes

Common utilities (twml/twml_common/, 3 files, 1.8K chars):

• PartitionInitializer — Keras initializer from numpy arrays
• serialize / deserialize — object serialization helpers
• create_sparse_tensor / create_reference_input — sparse tensor construction

Architecture Deep Dive: Follow Recommendations Service

The Follow Recommendations Service is the largest service by model complexity:

Common models (follow-recommendations-service/common/src/main/scala/.../common/models/, 54 files, 78K chars): Define the data model for recommendation candidates, scores, signals, and display contexts.

Candidate sources (16+ algorithms): Each in its own directory under common/candidate_sources/, with dedicated README explaining the algorithm, data sources, and parameters.

Ranking pipeline:

1. Candidates from all sources are unified
2. Feature hydration from RealGraph, Feature Store, social graph
3. Multi-tower MLP model scores candidates (trained weekly on Profile Sidebar impression data)
4. Filtering by safety, dedup, business rules
5. Results published to Manhattan for serving

Controllers (follow-recommendations-service/server/src/main/scala/.../controllers/, 5 files, 8.5K chars): Handle request routing, response assembly, and flow orchestration.

Architecture Deep Dive: TweetyPie

TweetyPie is the tweet storage and hydration service — one of the largest services in the codebase:

• Handlers (tweetypie/server/src/main/scala/.../handler/, 49 files, 91K chars): Tweet CRUD operations, media handling, URL resolution, safety checks
• Config (tweetypie/server/src/main/scala/.../config/, 22 files, 245K chars): Service configuration, feature switches, dependency wiring
• Core (tweetypie/server/src/main/scala/.../core/, 15 files, 33K chars): Core tweet model, hydration, filtering
• Observers (tweetypie/server/src/main/scala/.../service/observer/, 11 files, 30K chars): Per-operation metrics and tracing
• Servo subsystem: Request handling, repository pattern, JSON serialization, decider integration

Architecture Deep Dive: VisibilityLib

VisibilityLib evaluates content visibility rules:

• Models (visibilitylib/src/main/scala/.../visibility/models/, 24 files, 126K chars): Content labels, visibility actions, rule outcomes, abuse types, media labels, safety levels
• Features (visibilitylib/src/main/scala/.../visibility/features/, 4 files, 15K chars): Feature definitions for rule evaluation
• Rules (visibilitylib/src/main/scala/.../visibility/rules/): Rule definitions and evaluation logic
• Builders (visibilitylib/src/main/scala/.../visibility/builder/): Rule context construction

Architecture Deep Dive: Push Service (Notifications)

The notification system uses a four-stage pipeline:

1. Candidate generation — multiple sources produce notification candidates
2. Light ranking — fast MLP pre-filter (pushservice/src/main/python/models/light_ranking/)
3. Heavy ranking — ClemNet multi-task model (pushservice/src/main/python/models/heavy_ranking/)
4. Quality control — policy predicates

Heavy ranker architecture (ClemNet):

class ClemNet(tf.keras.layers.Layer):
    # Multi-block architecture with:
    # - KerasConv1D layers (1D convolution on feature groups)
    # - ChannelWiseDense layers (per-channel dense projection)
    # - ResidualLayer connections
    # - Block2 modules combining conv + dense + residual

Configuration via Pydantic-style params: ClemNetParams, BlockParams, ConvParams, DenseParams, TopLayerParams.

Predicate system (pushservice/src/main/scala/.../pushservice/predicate/event/, 1 file, 6.3K chars): Event-based filtering predicates for notification quality control.

Config (pushservice/src/main/scala/.../pushservice/config/, 5 files, 119K chars): The largest config module in the codebase, containing comprehensive notification pipeline configuration.

Architecture Deep Dive: SimClusters

SimClusters maps users and content into a shared topic embedding space:

• Summingbird common (src/scala/com/twitter/simclusters_v2/summingbird/common/, 14 files, 51K chars): Real-time aggregation of cluster membership
• Scalding embedding (src/scala/com/twitter/simclusters_v2/scalding/embedding/common/, 4 files, 37K chars): Batch embedding generation
• Scalding common (src/scala/com/twitter/simclusters_v2/scalding/common/, 4 files, 14.6K chars): Shared Scalding utilities
• Common (src/scala/com/twitter/simclusters_v2/common/, 11 files, 43K chars): Core cluster models and utilities
• Scio pipelines: BigQuery generation and multi-type graph processing

Architecture Deep Dive: Earlybird Internals

Earlybird's search index architecture (from src/java/com/twitter/search/):

• Core (core/earlybird/): Inverted index, posting lists, term dictionaries (FST and MPH), column-stride field indexes, facet counting
• Config (earlybird/config/, 7 files, 26K chars + earlybird/common/config/, 2 files, 25K chars): Schema definitions, tier configurations
• Ingester (search/ingester/pipeline/app/, 4 files, 13K chars): Kafka-to-index pipeline
• Earlybird Root (earlybird_root/common/, 9 files, 38K chars + earlybird_root/config/, 1 file, 1.5K chars): Request routing and response merging across partitions
• Relevance (search/common/relevance/, config + features): Search result scoring and feature extraction
• Schema (search/common/schema/, 9 files, 62K chars): Tweet field definitions, analyzers, and index mappings

Code Patterns and Conventions

Dependency injection: All services use Google Guice via Finatra's module system. Services declare dependencies as Module sequences.

RPC pattern: Thrift-based RPC with Finagle. Services define Thrift IDL in thrift/ directories, generate Scala/Java stubs, and implement controllers.

Pipeline pattern: Product Mixer's step-based pipeline with fail-open policies, quality factors, and arrow-based execution (Twitter's Stitch library).

Configuration pattern: YAML configs in resources/config/ directories, loaded by service-specific config modules (Scala objects/classes).

Testing pattern: Test files co-located with source in parallel test/ directories. Unified User Actions includes test configs for client validation.

Error handling:

• Custom exception types (EarlyStopError, CheckpointNotFoundError, InvalidStepStateException)
• Fail-open policies for non-critical pipeline stages
• Stitch-based error propagation in Product Mixer

Testing Strategy

• Unit tests: Standard Scala/Java test frameworks, co-located in src/test/
• Integration tests: Service-level tests using Finatra's testing framework
• Load tests: ANN load test framework with configurable QPS, duration, and parameter grid search
• ML model evaluation: twml's eval.steps and eval.period for online evaluation during training
• CI: Minimal (ci/ci.sh exits 0); real validation through Bazel test targets

Potential Improvements

Based on code analysis:

1. CI pipeline — The ci/ci.sh script is a no-op. A real CI pipeline running ./bazel test //... on key targets would prevent regressions.
2. Documentation — repo_docs.cbf returned 0 files at the root level. A root-level README summarizing the architecture would help new contributors.
3. Dependency versions — Navi pins specific Git tags for ort crate; upgrading to ONNX Runtime 1.15+ would bring performance improvements.
4. Python version — twml and the Dockerfile pin Python 3.7, which reached end-of-life in June 2023. Migration to 3.10+ is recommended.
5. Configuration — Service configs are scattered across 12+ locations with no unified schema. A shared configuration framework would reduce duplication.
6. Feature Store coupling — Multiple services depend on the internal Feature Store client. Abstracting this behind an interface would improve testability.

Contributing

The repository is read-only (archived). To study or adapt the code:

1. Fork the repository
2. Build individual services using Bazel or Cargo
3. Service-specific READMEs exist in many subsystem directories (especially follow-recommendations-service/common/src/main/scala/.../candidate_sources/)
4. The ANN subsystem has the most complete internal documentation with README.rst and README.md files

9. Integration Points

Inter-Service Communication

All services communicate via Thrift RPC over Finagle with mTLS:

Event Streaming (Kafka Topics)

Unified User Actions (UUA) standardizes event tracking across all surfaces:

• Adapters (unified_user_actions/adapter/): Convert product-specific events to UUA format
• Kafka serialization (unified_user_actions/kafka/): Custom Kafka SerDe with internal type handling
• Enrichment (unified_user_actions/enricher/): Post-processing enrichment of UUA events
• Client (unified_user_actions/client/): Client library with environment-specific config (staging, production)

Embedding Infrastructure

The Representation Manager's migration module (representation-manager/server/src/main/scala/.../migration/, 1 file, 34K chars) handles version transitions for the embedding storage layer.