# Pathway Administrator & Developer Guide

## 1. Architecture Overview

Pathway is a Python framework for building real-time data processing pipelines, powered by a high-performance Rust computation engine rooted in differential dataflow and timely dataflow paradigms. The system is a hybrid Python+Rust architecture bridged via PyO3, designed for streaming-first incremental computation.

### Layered Architecture

| Layer | Location | Role |
|-------|----------|------|
| Python API | `python/pathway/` | User-facing SDK, connectors, schema system, LLM xpacks |
| Python Internals | `python/pathway/internals/` (56 files, 666K chars) | Core Table, Schema, Expression, Join, and Run APIs |
| PyO3 Bridge | `src/python_api/` (17K chars) | Rust↔Python interop via PyO3 0.25.0 (abi3-py310) |
| Rust Engine | `src/engine/` (515K chars, 18 files) | Dataflow computation operators |
| Rust Connectors | `src/connectors/` (511K chars, 18 files) | High-performance I/O |
| Rust Persistence | `src/persistence/` (117K chars, 8 files) | State checkpointing and replay |
| External Integration | `src/external_integration/` (47K chars) | Third-party service bindings |
| Vendorized Forks | `external/` | differential-dataflow + timely-dataflow (modified) |

### Rust Crate Structure

```
src/
├── lib.rs               # Crate root — module declarations + jemalloc setup
├── async_runtime.rs     # Tokio runtime creation (rt-multi-thread)
├── deepcopy.rs          # S3Bucket deep copy utilities
├── env.rs               # Environment variable parsing
├── fs_helpers.rs        # Directory management
├── mat_mul.rs           # N-dimensional matrix multiplication (ndarray)
├── pipe.rs              # Named pipe support
├── retry.rs             # Retry logic with backoff
├── timestamp.rs         # Timestamp handling
├── connectors/
│   ├── mod.rs           # Connector orchestration
│   ├── data_lake/       # Delta Lake, Iceberg, Arrow integration
│   ├── scanner/         # File system scanning
│   ├── aws/             # S3, DynamoDB, Kinesis
│   └── metadata/        # Connector metadata
├── engine/
│   ├── dataflow/        # Differential/timely dataflow operators
│   │   └── operators/   # Individual operator implementations
│   └── telemetry/       # OpenTelemetry integration
├── persistence/
│   └── backends/        # Filesystem, S3 backends (5 files, 17K chars)
├── external_integration/
└── python_api/          # PyO3 bindings
```

> [!IMPORTANT]
> The computation engine uses **vendorized forks** of differential-dataflow and timely-dataflow (located in `external/`). These are not upstream crates — they contain Pathway-specific modifications. Do not replace them with upstream versions.

### Python Package Structure

```
python/pathway/
├── __init__.py          # Main namespace exports
├── __main__.py          # Entry point → pathway.cli.main
├── cli.py               # Click CLI (spawn, replay, web_dashboard, airbyte)
├── _engine_finder.py    # MagicCargoFinder for dev builds
├── internals/           # Core API (Table, Schema, expressions, joins)
├── io/                  # 35+ connector modules
├── stdlib/              # Standard library (temporal, graphs, ML, stats)
├── xpacks/              # Extension packs (LLM/RAG, SharePoint)
├── persistence/         # State persistence Python API
├── demo/                # Demo stream generators
├── debug/               # Debug utilities
└── web_dashboard/       # Monitoring dashboard (frontend + backend)
```

### Data Flow

```
Input Connectors → Python API (Table operations) → Rust Engine (differential dataflow) → Output Connectors
                                                         ↕
                                                   Persistence Layer
```

All table operations are compiled into a dataflow graph executed by the Rust engine. The engine processes data incrementally — only changes (diffs) propagate through the graph, making it efficient for streaming workloads.

---

## 2. System Requirements

### Runtime Requirements

| Requirement | Minimum | Recommended |
|-------------|---------|-------------|
| Python | 3.10+ (abi3-py310) | 3.12+ |
| OS | Linux (primary), macOS | Linux x86_64 |
| Memory | Depends on data volume | jemalloc-managed |
| Disk | For persistence backends | SSD for checkpointing |

### Build Requirements (from source)

| Requirement | Version |
|-------------|---------|
| Rust toolchain | 1.93 |
| Python | 3.10+ |
| CMake | Required (for rdkafka ssl-vendored build) |
| C compiler | Required (for native dependencies) |

### Installation

```bash
pip install -U pathway
```

> [!NOTE]
> The pip package ships a pre-compiled Rust engine. Building from source requires the full Rust 1.93 toolchain and CMake for the rdkafka dependency (ssl-vendored, zstd, curl features).

### Memory Allocator

Pathway uses **jemalloc** by default (via the `jemallocator` crate 0.5.4). This provides better performance for the allocation patterns in dataflow computation. To override:

- Compile with the `standard-allocator` Cargo feature to use the system allocator instead.

---

## 3. Configuration Reference

### Environment Variables

#### Core Runtime

| Variable | Purpose | Values/Notes |
|----------|---------|--------------|
| `PATHWAY_LICENSE_KEY` | License key for Scale/Enterprise features | String; also settable via `pw.set_license_key()` |
| `PATHWAY_PERSISTENCE_MODE` | Persistence mode | Controls state checkpointing behavior |
| `PATHWAY_REPLAY_MODE` | Replay mode | For replaying recorded connector data |
| `PATHWAY_SNAPSHOT_ACCESS` | Snapshot access mode | `record` or `replay` |
| `PATHWAY_MONITORING_HTTP_PORT` | HTTP port for monitoring endpoint | Integer |
| `PATHWAY_SKIP_START_LOG` | Suppress startup log output | Boolean |
| `PATHWAY_YOLO_RARE_WAKEUPS` | Reduce wakeup frequency | Boolean; performance tuning |
| `PATHWAY_THREADS` | Thread count | Set automatically by `pathway spawn` |
| `PATHWAY_PROCESSES` | Process count | Set automatically by `pathway spawn` |
| `PATHWAY_PROCESS_ID` | Current process ID | Set automatically by `pathway spawn` |
| `PATHWAY_FIRST_PORT` | First communication port | Set automatically by `pathway spawn` |
| `PATHWAY_ADDRESSES` | Address list for multi-machine mode | Set automatically by `pathway spawn` |
| `PATHWAY_RUN_ID` | Run identifier | Set automatically by `pathway spawn` |
| `PATHWAY_SUPPRESS_OTHER_WORKER_ERRORS` | Suppress sibling worker error output | Boolean |
| `PATHWAY_SPAWN_ARGS` | JSON args for `spawn_from_env` command | JSON string |

#### Build-time Variables

| Variable | Purpose |
|----------|---------|
| `PATHWAY_PROFILE` | Cargo build profile (default: `dev`) |
| `PATHWAY_QUIET` | Suppress cargo build output |
| `PATHWAY_FEATURES` | Cargo features to enable |

#### AWS/Cloud Credentials

| Variable | Purpose |
|----------|---------|
| `AWS_S3_ACCESS_KEY` | S3 access key |
| `AWS_S3_SECRET_ACCESS_KEY` | S3 secret key |
| `AWS_PROFILE` | AWS profile name |

#### Debugging

| Variable | Purpose |
|----------|---------|
| `DIFFERENTIAL_LOG_ADDR` | Differential dataflow logging address |

### Licensing Configuration

```python
import pathway as pw

# Set license key programmatically
pw.set_license_key("your-license-key-here")

# Or via environment variable
# export PATHWAY_LICENSE_KEY=your-license-key-here
```

| Tier | Features | Enforcement |
|------|----------|-------------|
| Community | Basic features, limited workers | Default |
| Scale | Multi-machine, advanced features | `PATHWAY_LICENSE_KEY` required |
| Enterprise | Unlimited workers | `unlimited-workers` compile feature |

Entitlement checks are performed by `_check_entitlements()`. For example, the `"multiple-machines"` entitlement requires Scale tier or above.

### Persistence Configuration

```python
import pathway as pw

# Filesystem backend
persistence_config = pw.persistence.Config(
    backend=pw.persistence.Backend.filesystem(path="/data/checkpoints")
)

# S3 backend
persistence_config = pw.persistence.Config(
    backend=pw.persistence.Backend.s3(
        bucket="my-bucket",
        root_path="pathway-state/"
    )
)
```

### YAML Pipeline Configuration

Pathway supports YAML-based pipeline definitions loaded via `pw.load_yaml()`:

```yaml
# Custom YAML tags supported:
# !pw.schema_from_types — define schemas inline
# !pw.io.kafka.read — Kafka source
# !pw.io.postgres.write — PostgreSQL sink
# !pw.persistence.Config — persistence configuration
# !pw.persistence.Backend.filesystem — filesystem backend

# Environment variable interpolation:
kafka_brokers: $KAFKA_BROKERS
```

> [!TIP]
> YAML configurations support environment variable interpolation using `$VAR_NAME` syntax. This allows the same YAML file to work across environments without modification.

### Monitoring Configuration

```python
import pathway as pw

# Set monitoring level
pw.set_monitoring_config(level=pw.MonitoringLevel.ALL)
```

The monitoring HTTP endpoint is exposed on the port specified by `PATHWAY_MONITORING_HTTP_PORT`. The web dashboard runs separately on port 8088 via `pathway web_dashboard`.

---

## 4. Deployment & Scaling

### Single-Process Deployment

```bash
# Basic execution
python my_pipeline.py

# With persistence
PATHWAY_PERSISTENCE_MODE=... python my_pipeline.py
```

### Multi-Process/Thread Deployment via CLI

The `pathway spawn` command manages multi-process and multi-thread execution:

```bash
# Multi-threaded (single process, multiple threads)
pathway spawn --threads 4 python my_pipeline.py

# Multi-process
pathway spawn --processes 4 python my_pipeline.py

# Combined
pathway spawn --processes 2 --threads 4 python my_pipeline.py
```

#### Auto-Scaling Mechanism

The spawn system supports dynamic scaling via process exit codes:

| Signal | Exit Code | Constant | Effect |
|--------|-----------|----------|--------|
| SIGUSR1 | 10 | `EXIT_CODE_DOWNSCALE` | Scale down workers |
| SIGUSR2 | 12 | `EXIT_CODE_UPSCALE` | Scale up workers |

Scaling factors:
- `UPSCALING_FACTOR = 2.0` — doubles worker count on upscale signal
- `DOWNSCALING_FACTOR = 0.5` — halves worker count on downscale signal

> [!WARNING]
> The multi-machine feature (`PATHWAY_ADDRESSES`) requires a Scale or Enterprise license. The `_check_entitlements()` function enforces this at runtime. Community tier is limited to single-machine deployment.

### Multi-Machine Deployment

For distributed execution across multiple hosts:

1. Set `PATHWAY_ADDRESSES` to a list of all participating machine addresses
2. Set `PATHWAY_FIRST_PORT` for inter-process communication
3. Each machine gets a unique `PATHWAY_PROCESS_ID`
4. Requires Scale license (`"multiple-machines"` entitlement)

The `pathway spawn_from_env` command reads configuration from `PATHWAY_SPAWN_ARGS` (JSON), useful in orchestrated/containerized deployments.

### Cargo Build Profiles

| Profile | opt-level | Debug Info | Use Case |
|---------|-----------|------------|----------|
| `dev` | 3 | line-tables-only | Development (still optimized) |
| `release` | 3 | none | Production |
| `profiling` | 3 (inherits release) | line-tables-only | Performance analysis |
| `debugging` | 3 (inherits dev) | full | Debugging with symbols |

> [!NOTE]
> The `dev` profile uses `opt-level = 3` (full optimization), which is unusual. This means development builds are nearly as fast as release builds, but compile times are longer.

### Cargo Features

| Feature | Effect |
|---------|--------|
| `unlimited-workers` | Remove worker count limits |
| `standard-allocator` | Use system allocator instead of jemalloc |
| `yolo-id64` | 64-bit identifiers |
| `yolo-id32` | 32-bit identifiers |
| `enterprise` | Enables `unlimited-workers` |

---

## 5. Operations

### CLI Reference

| Command | Purpose |
|---------|---------|
| `pathway spawn` | Multi-process/thread execution with auto-scaling |
| `pathway replay` | Replay recorded connector data |
| `pathway spawn_from_env` | Read spawn args from `PATHWAY_SPAWN_ARGS` env var |
| `pathway web_dashboard` | Start monitoring UI (default port 8088) |
| `pathway airbyte create_source` | Create Docker-based Airbyte connector source |

### Monitoring Stack

Pathway integrates with multiple observability systems:

**OpenTelemetry** (traces + metrics):
- Uses OTLP/gRPC export
- `opentelemetry` 0.30.0 with SDK
- `opentelemetry-otlp` for export
- Configured in `src/engine/telemetry/`

**Prometheus**:
- `prometheus-client` 0.23.1 for metrics export
- Exposes standard process and pipeline metrics

**Web Dashboard**:
- Built-in monitoring UI at port 8088
- Start with `pathway web_dashboard`
- Frontend + backend in `python/pathway/web_dashboard/`

**HTTP Monitoring Endpoint**:
- Configured via `PATHWAY_MONITORING_HTTP_PORT`
- Provides runtime metrics and health information

### Persistence Operations

The persistence system supports:
- **Recording**: Capture connector data and state for replay
- **Replay**: Re-execute pipelines from recorded state
- **Checkpointing**: Periodic state snapshots for recovery
- **Backends**: Filesystem (local disk) and S3

State is stored in configurable paths. The `PATHWAY_SNAPSHOT_ACCESS` variable controls whether the system is recording new snapshots or replaying existing ones.

### Running Pipelines

```python
import pathway as pw

# Define pipeline...

# Execute (blocking)
pw.run()

# Execute all registered pipelines
pw.run_all()
```

> [!TIP]
> Use `pw.run()` for single-pipeline scripts. `pw.run_all()` executes all pipelines that have been constructed in the current process, useful when composing multiple independent dataflows.

---

## 6. Performance

### Memory Management

- **Default allocator**: jemalloc (via `jemallocator` 0.5.4)
- jemalloc provides better fragmentation behavior for dataflow workloads
- Override with `standard-allocator` feature if profiling shows jemalloc overhead

### Engine Optimization

The Rust engine leverages:
- **Differential dataflow**: Only processes changes (diffs), not full datasets
- **Timely dataflow**: Parallel, pipelined execution of operators
- **Incremental computation**: Maintains operator state; re-computes only affected outputs

### Concurrency Model

- **Tokio** (1.45.1, `rt-multi-thread`): Async runtime for I/O-bound connectors
- **Timely dataflow workers**: CPU-bound computation threads
- **Separation**: I/O connectors use async Tokio tasks; computation uses timely workers

### Connector Performance

High-throughput connectors use native Rust implementations:
- `rdkafka` 0.39.0 (librdkafka via cmake-build) for Kafka
- Native S3, PostgreSQL, MongoDB clients
- Bulk operations where supported (Elasticsearch bulk API, etc.)

### Known Performance Issues

- GitHub Issue #21: 1M row processing takes >10 minutes (open issue)

> [!CAUTION]
> The `PATHWAY_YOLO_RARE_WAKEUPS` environment variable reduces wakeup frequency to lower CPU usage in idle periods, but may increase latency for processing new data. Only enable in scenarios where latency tolerance is high.

---

## 7. Security

### TLS Configuration

The `TLSSettings` class (in `python/pathway/internals/_io_helpers.py`) provides comprehensive TLS configuration:

```python
from pathway.internals._io_helpers import TLSSettings

tls = TLSSettings(
    mode="verify-full",         # disable|allow|prefer|require|verify-ca|verify-full
    root_cert_path="/certs/ca.pem",
    client_cert_path="/certs/client.pem",
    client_key_path="/certs/client.key",
    trust_certificates=True
)
```

| TLS Mode | Behavior |
|----------|----------|
| `disable` | No TLS |
| `allow` | TLS if server supports it |
| `prefer` | Try TLS, fall back to plaintext |
| `require` | TLS required, no certificate verification |
| `verify-ca` | TLS required, verify CA signature |
| `verify-full` | TLS required, verify CA + hostname |

### Schema Registry Authentication

The Schema Registry connector supports multiple auth methods:

- Token-based authentication
- Username/password (Basic auth)
- Custom headers
- HTTP proxy configuration

### S3/Cloud Credentials

```python
from pathway.internals._io_helpers import AwsS3Settings

s3 = AwsS3Settings(
    access_key="...",
    secret_access_key="...",
    session_token="...",           # Optional, for temporary credentials
    endpoint="...",                # Custom endpoint (MinIO, etc.)
    path_style=True               # Path-style access (for MinIO)
)
```

> [!WARNING]
> Never hardcode credentials in pipeline code. Use environment variables (`AWS_S3_ACCESS_KEY`, `AWS_S3_SECRET_ACCESS_KEY`, `AWS_PROFILE`) or instance roles for cloud deployments.

### License Key Security

The `PATHWAY_LICENSE_KEY` grants access to Scale/Enterprise features. Treat it as a secret:
- Set via environment variable, not in source code
- The `pw.set_license_key()` API is for interactive/notebook use only

---

## 8. Developer Internals

### PyO3 Bridge Architecture

The Python↔Rust bridge uses PyO3 0.25.0 with `abi3-py310` (stable ABI from Python 3.10+):

- **Binding location**: `src/python_api/` (17K chars)
- **ABI stability**: Single compiled extension works across Python 3.10, 3.11, 3.12, etc.
- **Dev builds**: `_engine_finder.py` implements `MagicCargoFinder` to locate dev-built Rust libraries

### Core Python Internals

| Module | Lines/Size | Responsibility |
|--------|-----------|----------------|
| `internals/table.py` | Core | Table class, groupby, select, filter, join entry points |
| `internals/schema.py` | Core | Schema class, SchemaProperties, type system |
| `internals/expression.py` | Core | ColumnExpression, ColumnReference, operator overloading |
| `internals/joins.py` | Core | Join implementations (inner, left, right, outer) |
| `internals/run.py` | Core | `run()`, `run_all()` — pipeline execution entry points |
| `internals/api.py` | Core | Engine bridge types (Python↔Rust interface) |
| `internals/config.py` | Core | License, persistence, monitoring config objects |
| `internals/custom_reducers.py` | Core | BaseCustomAccumulator for stateful aggregations |
| `internals/interactive.py` | Core | LiveTable, enable_interactive_mode for notebooks |
| `internals/monitoring.py` | Core | MonitoringLevel enum |
| `internals/_io_helpers.py` | Core | TLSSettings, AwsS3Settings, SchemaRegistrySettings |

### Design Patterns

#### Schema-First Design

All tables require explicit schemas:

```python
import pathway as pw

class SensorReading(pw.Schema):
    sensor_id: str
    timestamp: int
    value: float
    metadata: dict
```

Schemas are Python classes inheriting from `pw.Schema` with typed attributes. The framework enforces type safety at the dataflow boundary.

#### Connector Subject/Observer Pattern

Connectors follow a unified read/write interface:

```python
# Reading (subject — produces data)
table = pw.io.kafka.read(
    rdkafka_settings={"bootstrap.servers": "localhost:9092"},
    topic="input-topic",
    schema=SensorReading
)

# Writing (observer — consumes data)
pw.io.kafka.write(
    table,
    rdkafka_settings={"bootstrap.servers": "localhost:9092"},
    topic_name="output-topic"
)
```

#### UDF Decorator Pattern

```python
@pw.udf
def normalize(value: float, max_val: float) -> float:
    return value / max_val if max_val != 0 else 0.0

# Used in table operations
result = table.select(normalized=normalize(pw.this.value, pw.this.max_value))
```

#### AsyncTransformer Pattern

For async operations (LLM calls, HTTP requests):

```python
from pathway.stdlib.utils import AsyncTransformer

class LLMTransformer(AsyncTransformer):
    async def invoke(self, row):
        response = await call_llm(row.prompt)
        return {"result": response}
```

#### Custom Reducers

```python
from pathway.internals.custom_reducers import BaseCustomAccumulator

class RunningMean(BaseCustomAccumulator):
    def __init__(self):
        self.sum = 0.0
        self.count = 0

    def update(self, value):
        self.sum += value
        self.count += 1

    def compute(self):
        return self.sum / self.count if self.count > 0 else 0.0
```

### Rust Engine Internals

The engine (`src/engine/`, 515K chars) implements:

- **Dataflow operators**: Map, filter, reduce, join, temporal operations
- **Operator fusion**: Combines compatible operators for efficiency
- **Progress tracking**: Timely dataflow frontier tracking for correctness
- **Worker coordination**: Multi-worker synchronization via timely infrastructure

The `src/engine/dataflow/operators/` directory contains individual operator implementations that map to Python API operations.

### Vendorized External Dependencies

Located in `external/`:
- **differential-dataflow**: Incremental computation framework (Pathway fork)
- **timely-dataflow**: Parallel dataflow computation (Pathway fork with `timely/` subpath)

These forks enable Pathway-specific optimizations and fixes that are not in upstream repositories.

---

## 9. Integration Points

### Connector Inventory

#### Input Connectors (Python `pw.io.*`)

| Connector | Module | Rust Backend |
|-----------|--------|-------------|
| Kafka | `pw.io.kafka` | rdkafka 0.39.0 |
| S3 | `pw.io.s3` | rust-s3 0.34.0 |
| Filesystem | `pw.io.fs` | Native |
| PostgreSQL | `pw.io.postgres` | postgres 0.19.10 |
| MongoDB | `pw.io.mongodb` | mongodb 3.2.2 |
| MySQL | `pw.io.mysql` | — |
| MSSQL | `pw.io.mssql` | tiberius 0.12 |
| DynamoDB | `pw.io.dynamodb` | aws-sdk-dynamodb |
| Elasticsearch | `pw.io.elasticsearch` | elasticsearch 8.17.0-alpha.1 |
| Kinesis | `pw.io.kinesis` | aws-sdk-kinesis |
| NATS | `pw.io.nats` | async-nats 0.41.0 |
| MQTT | `pw.io.mqtt` | rumqttc 0.24.0 |
| RabbitMQ | `pw.io.rabbitmq` | rabbitmq-stream-client 0.11.0 |
| Google Drive | `pw.io.gdrive` | — |
| HTTP | `pw.io.http` | hyper 0.14 |
| CSV | `pw.io.csv` | — |
| JSON Lines | `pw.io.jsonlines` | — |
| Plaintext | `pw.io.plaintext` | — |
| Python | `pw.io.python` | — |
| Airbyte | `pw.io.airbyte` | Docker-based |
| Debezium | `pw.io.debezium` | — |
| PubSub | `pw.io.pubsub` | — |
| Redpanda | `pw.io.redpanda` | — |
| MinIO | `pw.io.minio` | — |
| Iceberg | `pw.io.iceberg` | iceberg 0.8.0 |
| Delta Lake | `pw.io.deltalake` | deltalake 0.31.1 |
| BigQuery | `pw.io.bigquery` | — |
| SQLite | `pw.io.sqlite` | rusqlite 0.35.0 |
| QuestDB | `pw.io.questdb` | — |
| Slack | `pw.io.slack` | — |
| PyFilesystem | `pw.io.pyfilesystem` | — |
| Null | `pw.io.null` | — |
| Milvus | `pw.io.milvus` | — |
| SharePoint | `pw.xpacks.connectors.sharepoint` | — |

#### Output/Sink Connectors

Most input connectors have corresponding write methods. Notable Rust-backed sinks:
- Kafka (rdkafka)
- PostgreSQL (postgres + pg_walstream for WAL streaming)
- Elasticsearch (bulk API)
- S3/Delta Lake/Iceberg

> [!NOTE]
> PostgreSQL WAL streaming (`pg_walstream` 0.6.1) enables CDC (Change Data Capture) from PostgreSQL without polling. This is the recommended input method for PostgreSQL sources with real-time requirements.

### Vector Database Integration

| Vector DB | Rust Crate | Python Module |
|-----------|-----------|---------------|
| Qdrant | qdrant-client 1.15.0 | via xpacks |
| pgvector | pgvector 0.4.1 | via postgres connector |
| Milvus | — | `pw.io.milvus` |
| uSearch (embedded) | usearch 2.15.3 | via stdlib indexing |

### Full-Text Search

- **Tantivy** 0.22.1 — Embedded full-text search engine used for indexing operations within Pathway pipelines.

### LLM/RAG Extension Pack

Located in `python/pathway/xpacks/llm/`:

| Module | Purpose |
|--------|---------|
| `embedders.py` | Text embedding integration |
| `splitters.py` | Document chunking strategies |
| `vector_store.py` | VectorStoreServer — serves vector search over HTTP |
| `parsers.py` | Document parsing (PDF, etc.) |

This xpack enables building RAG (Retrieval-Augmented Generation) pipelines natively within Pathway's streaming framework.

### Standard Library

| Package | Capabilities |
|---------|-------------|
| `pathway.stdlib.temporal` | Temporal joins, windowing (tumbling, sliding, session) |
| `pathway.stdlib.statistical` | Interpolation |
| `pathway.stdlib.graphs` | PageRank, Bellman-Ford, Louvain community detection |
| `pathway.stdlib.ml` | Machine learning utilities |
| `pathway.stdlib.indexing` | Index construction and querying |
| `pathway.stdlib.utils` | AsyncTransformer, pandas_transformer, bucketing |
| `pathway.stdlib.ordered` | Diff operations on ordered data |
| `pathway.stdlib.stateful` | Stateful stream operations |
| `pathway.stdlib.viz` | Visualization (plot, show) |

### Schema Registry Integration

Confluent Schema Registry support via `schema_registry_converter` 4.4.0:

```python
from pathway.internals._io_helpers import SchemaRegistrySettings

registry = SchemaRegistrySettings(
    url="https://schema-registry:8081",
    # Auth options:
    token="...",
    # OR
    username="...",
    password="...",
    # OR
    custom_headers={"Authorization": "Bearer ..."},
    proxy="http://proxy:8080"
)
```

### Testing Infrastructure

| Test Type | Location | Scope |
|-----------|----------|-------|
| Unit tests | `python/pathway/tests/` | Expressions, temporal, ML, CLI |
| Integration tests | `integration_tests/` | Real DB/service connections |
| Rust tests | `tests/integration/main.rs` | Engine-level integration |

**Test Utilities** (`pathway.tests.utils`):
- `assert_equal_tables` — Compare table contents with index
- `assert_equal_tables_wo_index` — Compare table contents ignoring index

Integration tests require running instances of:
- PostgreSQL, MongoDB, MySQL, MSSQL
- Kafka, NATS, RabbitMQ
- S3/MinIO, DynamoDB (LocalStack)
- Milvus, Elasticsearch

> [!IMPORTANT]
> Integration tests connect to real services. Ensure Docker containers or equivalent test infrastructure are running before executing the integration test suite. Fixtures in `conftest.py` manage connection parameters.

### Open Feature Requests

| Issue | Request |
|-------|---------|
| #220 | NeonDB support |
| #217 | Azure Blob Storage support |
| #216 | S3 write connector (`pw.io.s3.write`) |
| #215 | Elasticsearch input via polling |
| #213 | Spawn persistence compaction workers separately |
| #212 | Native SQLite output connector |
| #211 | Native MySQL input connector |
| #167 | External vector DB indexing as connectors |
| #162 | Disable/rename special fields in output connectors |
| #150 | Composite instance support |
| #142 | Execution graph checksums |
| #140 | ClickHouse output connector |

> [!TIP]
> For production deployments requiring features listed above, check the GitHub issue tracker for current status. Several of these (SQLite output, MySQL input) have partial implementations in the codebase that may be usable with custom wrappers.

---

## Appendix: Complete Rust Dependency Map

### Core Computation

| Crate | Version | Role |
|-------|---------|------|
| differential-dataflow | vendorized | Incremental computation |
| timely | vendorized | Parallel dataflow execution |
| ndarray | 0.15.6 | N-dimensional arrays (matrix operations) |

### Python Interop

| Crate | Version | Role |
|-------|---------|------|
| pyo3 | 0.25.0 | Python bindings (abi3-py310) |

### Async/Networking

| Crate | Version | Role |
|-------|---------|------|
| tokio | 1.45.1 | Async runtime (rt-multi-thread) |
| hyper | 0.14 | HTTP server |

### Data Sources/Sinks

| Crate | Version | Role |
|-------|---------|------|
| rdkafka | 0.39.0 | Kafka (ssl-vendored, cmake-build, zstd, curl) |
| postgres | 0.19.10 | PostgreSQL |
| pg_walstream | 0.6.1 | PostgreSQL WAL streaming (CDC) |
| pgvector | 0.4.1 | PostgreSQL vector extension |
| mongodb | 3.2.2 | MongoDB |
| tiberius | 0.12 | MSSQL |
| elasticsearch | 8.17.0-alpha.1 | Elasticsearch |
| rusqlite | 0.35.0 | SQLite (bundled) |
| rust-s3 | 0.34.0 | S3 |
| deltalake | 0.31.1 | Delta Lake (with datafusion, s3) |
| iceberg | 0.8.0 | Apache Iceberg |
| aws-sdk-dynamodb | — | DynamoDB |
| aws-sdk-kinesis | — | Kinesis |
| async-nats | 0.41.0 | NATS |
| rumqttc | 0.24.0 | MQTT |
| rabbitmq-stream-client | 0.11.0 | RabbitMQ |
| qdrant-client | 1.15.0 | Qdrant vector DB |

### Search/Indexing

| Crate | Version | Role |
|-------|---------|------|
| tantivy | 0.22.1 | Full-text search |
| usearch | 2.15.3 | Vector similarity search |

### Observability

| Crate | Version | Role |
|-------|---------|------|
| opentelemetry | 0.30.0 | Traces + metrics |
| prometheus-client | 0.23.1 | Metrics export |

### Data Formats

| Crate | Version | Role |
|-------|---------|------|
| schema_registry_converter | 4.4.0 | Confluent Schema Registry |

### Memory

| Crate | Version | Role |
|-------|---------|------|
| jemallocator | 0.5.4 | Memory allocator (default) |

---

## Appendix: Project Metadata

| Property | Value |
|----------|-------|
| Version | 0.30.1 (Rust crate) |
| License | BUSL-1.1 (Business Source License) |
| Repository size | 1,891 files, 3.76M characters |
| Primary languages | Python + Rust |
| Source commit | `f0afd65` |

---

_Generated May 2026 from pathwaycom/pathway at f0afd65. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._