# Polars Admin & Architecture Guide

This guide is for engineers, architects, and operators who need to understand Polars below the surface: how the Rust workspace is organized, how a query flows from a frontend expression to executed machine code, how to configure and deploy it, where the security surface is, and how to extend or contribute to it. For task-oriented usage, see the companion **User Guide**.

> [!NOTE]
> Polars is a **library and embedded query engine**, not a server or daemon. There is no service to deploy, no port to open, and no built-in authentication — Polars runs inside *your* process (a Python script, a Rust binary, a notebook, a Spark-like job). "Operations" in this guide therefore means embedding, resourcing, tuning, and securing Polars within a host application, not running it as standalone infrastructure.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Architecture Overview](#2-architecture-overview)
3. [The Crate Workspace](#3-the-crate-workspace)
4. [Query Lifecycle and the Optimizer](#4-query-lifecycle-and-the-optimizer)
5. [Execution Engines](#5-execution-engines)
6. [System Requirements](#6-system-requirements)
7. [Build and Feature-Flag Configuration](#7-build-and-feature-flag-configuration)
8. [Runtime Configuration Reference](#8-runtime-configuration-reference)
9. [Deployment Patterns](#9-deployment-patterns)
10. [Operations: Resourcing, Performance, Memory](#10-operations-resourcing-performance-memory)
11. [Security Analysis](#11-security-analysis)
12. [Extending Polars](#12-extending-polars)
13. [Interoperability and the Foreign Function Interface](#13-interoperability-and-the-foreign-function-interface)
14. [Code Quality and Conventions](#14-code-quality-and-conventions)
15. [Testing Strategy](#15-testing-strategy)
16. [Potential Improvements and Tech Debt](#16-potential-improvements-and-tech-debt)
17. [Troubleshooting (Operational)](#17-troubleshooting-operational)
18. [Quick Reference](#18-quick-reference)

---

## 1. Executive Summary

Polars is an analytical **query engine for DataFrames**, written in Rust and exposed through Python, Rust, Node.js, R, and SQL frontends. Its defining properties:

- **Columnar, Arrow-native.** Data lives in the Apache Arrow memory format, enabling SIMD vectorization, cache efficiency, and zero-copy interchange.
- **Lazy + eager.** A logical-plan optimizer rewrites lazy queries before execution; eager calls are one-shot lazy queries.
- **Multi-engine.** An in-memory engine, a streaming engine for larger-than-RAM data, and a GPU engine, all driven by the same plan.
- **Multi-threaded by default.** Independent expressions execute in parallel across all cores with no user-managed threading.
- **Zero required dependencies.** The core imports in ~70 ms; every integration is an opt-in feature flag.

Architecturally, Polars is a **layered Cargo workspace of 29 crates**. A thin umbrella crate (`polars`) re-exports a stack that runs from the Arrow memory layer at the bottom, through the core data structures, the logical plan and optimizer, the physical expression and execution engines, up to the I/O and SQL frontends and the PyO3 Python bindings.

| Attribute | Value |
| --- | --- |
| Primary language | Rust (with a thin Python layer) |
| Python package | `polars` — version `1.41.2`, requires Python ≥ 3.10 |
| Rust crate | `polars` (umbrella) |
| Memory format | Apache Arrow (columnar) |
| Workspace crates | 29 (under `crates/`) plus the `pyo3-polars` extension crates |
| License | MIT |
| Frontends | Python, Rust, Node.js, R, SQL |
| Managed offering | Polars Cloud (`cloud.pola.rs`) — separate, optional |

---

## 2. Architecture Overview

```
            ┌──────────────────────────────────────────────────────────┐
 Frontends  │  Python (PyO3)   Rust    Node.js    R       SQL           │
            └──────────────────────────────────────────────────────────┘
                                   │ build a logical plan / expressions
                                   ▼
            ┌──────────────────────────────────────────────────────────┐
 Planning   │  polars-plan        Logical plan (IR) + Query Optimizer   │
            │  polars-sql         SQL → logical plan                    │
            └──────────────────────────────────────────────────────────┘
                                   │ optimized plan → physical plan
                                   ▼
            ┌──────────────────────────────────────────────────────────┐
 Execution  │  polars-expr        Physical expression engine           │
            │  polars-mem-engine  In-memory executor                   │
            │  polars-stream      Streaming (larger-than-RAM) engine    │
            │  polars-ooc         Out-of-core spilling                  │
            └──────────────────────────────────────────────────────────┘
                                   │ operate on
                                   ▼
            ┌──────────────────────────────────────────────────────────┐
 Data       │  polars-core        DataFrame / Series / ChunkedArray     │
            │  polars-ops         Joins, set ops, higher-order ops      │
            │  polars-compute     SIMD compute kernels                  │
            │  polars-time        Temporal kernels & windows            │
            └──────────────────────────────────────────────────────────┘
                                   │ stored as
                                   ▼
            ┌──────────────────────────────────────────────────────────┐
 Memory/IO  │  polars-arrow       Arrow arrays, bitmaps, buffers        │
            │  polars-io  polars-parquet  polars-json   Readers/writers │
            │  polars-utils  polars-buffer  polars-row   Primitives     │
            └──────────────────────────────────────────────────────────┘
```

The flow is uniform regardless of frontend: a frontend constructs a **logical plan**, `polars-plan`'s optimizer rewrites it, a **physical plan** is produced, and one of the execution engines runs it against Arrow-backed columns from `polars-core`.

### Why this layering matters operationally

- **Feature flags map to layers.** Enabling `parquet` pulls in `polars-parquet`; enabling `sql` pulls in `polars-sql`; `streaming` pulls in `polars-stream`. You compile only the layers you use (see §7).
- **The optimizer is the leverage point.** Most performance wins come from the plan rewrites in `polars-plan`, not from micro-tuning — which is why the lazy API is recommended.
- **Memory behavior is set at the bottom.** Allocator, prefetch, and out-of-core spilling are governed by `polars-utils`, `polars-arrow`, and `polars-ooc`.

---

## 3. The Crate Workspace

The workspace is declared in the root `Cargo.toml` with `members = ["crates/*", …]` and `default-members = ["crates/*"]`. The 29 crates under `crates/`:

| Crate | Layer | Responsibility |
| --- | --- | --- |
| `polars` | Facade | Umbrella crate; re-exports a curated public API and `prelude`. |
| `polars-core` | Data | `DataFrame`, `Series`, `ChunkedArray`, the dtype system, schema. |
| `polars-arrow` | Memory | Arrow arrays, bitmaps, buffers; the columnar foundation. |
| `polars-buffer` | Memory | Low-level buffer management. |
| `polars-compute` | Data | SIMD compute kernels (arithmetic, comparison, casts, filters, min/max, gather, rolling, EWM). |
| `polars-ops` | Data | Higher-level operations: joins (hash, as-of, IE), set ops, frame/series ops. |
| `polars-time` | Data | Temporal kernels, time windows, rolling-window group-by. |
| `polars-row` | Data | Row-encoding (for sorting / joining). |
| `polars-schema` | Data | Schema representation. |
| `polars-dtype` | Data | Data-type definitions, including categorical internals. |
| `polars-plan` | Planning | Logical plan (IR), the **query optimizer**, expression AST. |
| `polars-sql` | Planning | SQL parser/translator → logical plan. |
| `polars-lazy` | Planning | The `LazyFrame` API and lazy frame operations. |
| `polars-expr` | Execution | Physical expression engine (groups, reductions, hot-groups, idx tables). |
| `polars-mem-engine` | Execution | In-memory query executor (executors, planner, scan predicates). |
| `polars-stream` | Execution | Streaming engine (nodes, sources, sinks, physical plan, joins). |
| `polars-ooc` | Execution | Out-of-core spilling support. |
| `polars-io` | I/O | Readers/writers: CSV, JSON/NDJSON, IPC, Avro, cloud, file cache, catalog (Unity). |
| `polars-parquet` | I/O | Apache Parquet reader/writer (Arrow + low-level parquet, encodings, statistics, bloom filters). |
| `polars-json` | I/O | JSON / NDJSON serialization. |
| `polars-utils` | Primitive | Shared utilities: CPUID, memory/mmap, regex cache, idx maps, async utils. |
| `polars-error` | Primitive | Error types and the `polars_bail!`/`polars_err!` macros. |
| `polars-async` | Primitive | Async runtime primitives and executor. |
| `polars-config` | Primitive | Configuration plumbing (e.g. OOC spill dir). |
| `polars-ffi` | Interop | Foreign Function Interface for passing Arrow data across the ABI boundary. |
| `polars-python` | Bindings | The PyO3 bindings (DataFrame/LazyFrame/Expr/Series wrappers, conversions, interop). |
| `polars-testing` | Tooling | Assertion helpers for tests. |
| `polars-dylib` | Packaging | Dynamic-library build target. |
| `polars-mem-engine` … | — | (listed above) |

Beyond `crates/`, the workspace also contains:

- **`pyo3-polars/`** — the public extension SDK: `pyo3-polars` (host crate for plugins), `pyo3-polars-derive` (the `#[polars_expr]` proc-macro), and example plugins (`derive_expression`, `extend_polars_python_dispatch`, `io_plugin`).
- **`py-polars/`** — the Python package (`py-polars/src/polars/`, src-layout) plus the runtime build targets `polars-runtime-32`, `polars-runtime-64` (big-index), and `polars-runtime-compat` (legacy-CPU).

> [!IMPORTANT]
> The Rust crate that implements the Python bindings is historically called **`py-polars`**, distinct from the wrapped Rust engine crate **`polars`**. Both the installed Python *package* and the importable *module* are named `polars` — so `pip install polars` then `import polars`.

---

## 4. Query Lifecycle and the Optimizer

Every query — eager or lazy, from any frontend — passes through the same pipeline:

1. **Plan construction.** Expressions and contexts build a **logical plan** (the intermediate representation, IR, in `polars-plan`). The SQL frontend (`polars-sql`) and the PyO3 layer (`polars-python`) both lower into this same IR.
2. **Optimization.** `polars-plan`'s optimizer applies a series of rewrite passes (see below).
3. **Physical planning.** The optimized logical plan is converted to a physical plan by `polars-mem-engine` (in-memory) or `polars-stream` (streaming).
4. **Execution.** Physical operators run, evaluating physical expressions via `polars-expr` over `polars-core` columns, parallelized across the thread pool.

### Optimizer passes

The optimizer modules live under `crates/polars-plan/src/plans/optimizer/`. Key passes:

| Pass | Effect |
| --- | --- |
| **Predicate pushdown** | Move `filter`s as close to the data source as possible, so file readers skip rows/row-groups early. |
| **Projection pushdown** | Read only the columns the query needs; drop the rest at the source. |
| **Common-subexpression elimination (CSE)** | Compute a repeated subexpression once and reuse it (the `cse` feature). |
| **Slice pushdown** | Push `head`/`slice` limits down so less data is materialized. |
| **Simplify expression / simplify ordering** | Constant-fold and canonicalize expressions; remove redundant sorts. |
| **Type coercion / type check** | Insert casts and validate the schema *before* execution, catching errors early. |

Inspect the effect on any query:

```python
lf.explain()                  # optimized plan
lf.explain(optimized=False)   # plan as written
lf.show_graph()               # visual physical graph (needs the `graph` extra)
```

> [!TIP]
> The gap between `explain(optimized=False)` and `explain()` is, quite literally, the value the lazy API adds. If a query is slow, read the optimized plan first — a filter that failed to push down is the usual culprit.

---

## 5. Execution Engines

Polars selects an engine when you collect a plan:

| Engine | Crate | When | How to invoke |
| --- | --- | --- | --- |
| **In-memory** | `polars-mem-engine` | Default; data fits in RAM. | `lf.collect()` |
| **Streaming** | `polars-stream` | Larger-than-RAM, or for raw throughput. | `lf.collect(engine="streaming")` |
| **GPU** | (via `cudf`/`gpu` extra) | NVIDIA GPU available. | `lf.collect(engine="gpu")` or `pl.GPUEngine(...)` |

### Streaming engine internals

`polars-stream` models the query as a graph of **nodes** connected by pipes. I/O source nodes (`io_sources/` for CSV, NDJSON, IPC, Parquet, plus a multi-scan pipeline) feed transform nodes; sink nodes (`io_sinks/` writers for Parquet, IPC, CSV, NDJSON) drain results to disk. Operations that cannot run in a streaming fashion fall back to the in-memory engine transparently for that part of the plan.

The streaming engine's buffer sizes are tunable via environment variables (see §8): `POLARS_DEFAULT_LINEARIZER_BUFFER_SIZE`, `POLARS_DEFAULT_DISTRIBUTOR_BUFFER_SIZE`, `POLARS_DEFAULT_ZIP_HEAD_BUFFER_SIZE`.

### Out-of-core spilling

`polars-ooc` lets the engine spill intermediate state to disk when memory is constrained. The spill directory is controlled by `POLARS_OOC_SPILL_DIR`.

> [!NOTE]
> "Sinks" (`sink_parquet`, `sink_ipc`, `sink_csv`, `sink_ndjson`) are how the streaming engine produces large output without materializing the full result — read → transform → write all run in bounded memory.

---

## 6. System Requirements

| Requirement | Detail |
| --- | --- |
| Python (Python frontend) | ≥ 3.10. |
| Rust toolchain | Only for source builds or plugin development; not for normal use. |
| CPU (default wheels) | Modern x86-64 with **AVX2** (roughly 2013+) or AArch64. |
| CPU (legacy) | Use the `rtcompat` build (compiled without AVX) on older CPUs or x86-64 Python under Rosetta. |
| Memory | Proportional to working set; the streaming engine relaxes this for large data. |
| Optional system deps | Per integration — e.g. an ODBC/ADBC driver for databases, a cloud SDK for object storage. |

Polars detects CPU capabilities at import (`polars-utils/src/cpuid.rs`, `_cpu_check.py`). On an unsupported CPU it raises a `RuntimeError` rather than executing illegal instructions; the fix is the `rtcompat` wheel.

---

## 7. Build and Feature-Flag Configuration

Polars' footprint is controlled by **Cargo feature flags** (Rust) and **pip extras** (Python). Compile or install only what you need.

### Rust feature flags (selected)

| Group | Features |
| --- | --- |
| Core APIs | `lazy`, `sql`, `streaming`, `random`, `ndarray` |
| Data types | `dtype-date`, `dtype-datetime`, `dtype-time`, `dtype-duration`, `dtype-categorical`, `dtype-struct`, `dtype-i8`/`i16`/`i128`, `dtype-u8`/`u16`/`u128` |
| Temporal | `temporal` (Chrono interop), `timezones` |
| I/O | `parquet`, `json`, `ipc`, `serde`, `serde-lazy`, `decompress` (gzip/zlib/zstd) |
| Strings | `strings`, `string_pad`, `string_to_integer`, `extract_jsonpath` |
| Frame ops | `dynamic_group_by`, `sort_multiple`, `rows` (enables `pivot`/`transpose`), `join_asof`, `cross_join`, `semi_anti_join`, `diagonal_concat`, `partition_by`, `dataframe_arithmetic` |
| Series/expr ops | `is_in`, `round_series`, `mode`, `cum_agg`, `rolling_window`, `interpolate`, `rank`, `ewma`, `log`, `diff`, `pct_change`, `list_eval`, `arg_where`, `search_sorted`, `trigonometry`, … |
| Performance | `nightly` (SIMD + specialization), `performant`, `cse`, `bigidx` |
| Formatting | `fmt` |

```toml
# Cargo.toml — a typical analytics build
[dependencies]
polars = { version = "x", features = [
    "lazy", "streaming", "parquet", "json", "temporal",
    "dtype-datetime", "dtype-categorical", "cse",
] }
```

> [!CAUTION]
> `performant` and `nightly` trade longer compile times for faster binaries. `bigidx` makes data structures less cache-efficient (slower) in exchange for 64-bit row indices — enable it only if you exceed 2³² rows.

### Python install targets

The Python package ships three precompiled runtime variants, selected by the extra you install:

| Install | Runtime | For |
| --- | --- | --- |
| `pip install polars` | `polars-runtime-64` (default, AVX2) | Modern CPUs. |
| `pip install polars[rt64]` | big-index 64-bit row index | > 4.3 billion rows. |
| `pip install polars[rtcompat]` | `polars-runtime-compat` (no AVX) | Legacy CPUs / Rosetta. |

The active package can be forced/diagnosed with `POLARS_FORCE_PKG` / `POLARS_PREFER_PKG`.

The full pip-extras table (pandas, numpy, pyarrow, calamine, adbc, fsspec, deltalake, iceberg, gpu, plot, style, …) is in the **User Guide → Installation**.

---

## 8. Runtime Configuration Reference

Two configuration surfaces exist: the **`pl.Config`** object (display/behavior, see User Guide §16) and **environment variables** (engine/runtime behavior, below). Environment variables are read at import or at first use of the relevant subsystem.

### Memory and CPU

| Variable | Effect | Source |
| --- | --- | --- |
| `POLARS_OVERRIDE_TOTAL_MEMORY` | Override detected total memory (bytes) for memory-aware decisions. | `polars-utils/src/sys.rs` |
| `POLARS_MEMORY_PREFETCH` | Memory prefetch strategy. | `polars-utils/src/mem.rs` |
| `POLARS_DISABLE_AVX512` | Disable AVX-512 code paths. | `polars-utils/src/cpuid.rs` |
| `POLARS_SKIP_CPU_CHECK` | Skip the import-time CPU capability check (diagnostic). | `_cpu_check.py` |
| `POLARS_THP` | Transparent huge pages hint. | `polars/__init__.py` |
| `_RJEM_MALLOC_CONF` | jemalloc allocator tuning. | `polars/__init__.py` |

### Threading and async

| Variable | Effect | Source |
| --- | --- | --- |
| `POLARS_MAX_THREADS` | Size of the compute thread pool (set before import). | core |
| `POLARS_ASYNC_THREAD_COUNT` | Async runtime thread count. | `polars-async/src/lib.rs` |
| `POLARS_MAX_BLOCKING_THREAD_COUNT` | Max blocking threads. | `polars-async/src/lib.rs` |
| `POLARS_THREAD_NAME` | Name for spawned threads. | `polars-utils/src/mmap.rs` |

### Streaming engine

| Variable | Effect | Source |
| --- | --- | --- |
| `POLARS_DEFAULT_LINEARIZER_BUFFER_SIZE` | Linearizer node buffer size. | `polars-stream/src/lib.rs` |
| `POLARS_DEFAULT_DISTRIBUTOR_BUFFER_SIZE` | Distributor node buffer size. | `polars-stream/src/lib.rs` |
| `POLARS_DEFAULT_ZIP_HEAD_BUFFER_SIZE` | Zip-head buffer size. | `polars-stream/src/lib.rs` |
| `POLARS_VISUALIZE_IR` | Dump the IR for visualization. | `polars-stream/src/skeleton.rs` |
| `POLARS_VISUALIZE_PHYSICAL_PLAN` | Dump the physical plan. | `polars-stream/src/skeleton.rs` |
| `POLARS_TRACK_METRICS` / `POLARS_LOG_METRICS` | Track / log engine metrics. | `polars-stream/src/skeleton.rs` |

### I/O and cloud

| Variable | Effect | Source |
| --- | --- | --- |
| `POLARS_OOC_SPILL_DIR` | Directory for out-of-core spilling. | `polars-config/src/lib.rs` |
| `POLARS_UPLOAD_CHUNK_SIZE` / `POLARS_PARTITIONED_UPLOAD_CHUNK_SIZE` | Cloud upload chunk sizes. | `polars-io/src/configs.rs` |
| `POLARS_UPLOAD_CONCURRENCY` / `POLARS_PARTITIONED_UPLOAD_CONCURRENCY` | Cloud upload concurrency. | `polars-io/src/configs.rs` |
| `POLARS_DOWNLOAD_CHUNK_SIZE` | Cloud download chunk size. | `polars-io/src/pl_async.rs` |
| `POLARS_CONCURRENCY_BUDGET` | Concurrency budget for async I/O. | `polars-io/src/pl_async.rs` |
| `POLARS_CLOUD_WRITER_COALESCE_RUN_LENGTH` / `..._COPY_BUFFER_SIZE` | Cloud-writer tuning. | `polars-io/src/configs.rs` |

### Diagnostics and limits

| Variable | Effect | Source |
| --- | --- | --- |
| `POLARS_VERBOSE` | Verbose engine logging. | core |
| `POLARS_PANIC_ON_ERR` | Turn recoverable errors into panics (debugging). | `polars-error/src/lib.rs` |
| `POLARS_BACKTRACE_IN_ERR` | Include a Rust backtrace in error messages. | `polars-error/src/lib.rs` |
| `POLARS_MAX_EXPR_DEPTH` | Maximum expression nesting depth. | `polars-expr/src/planner.rs` |
| `POLARS_REGEX_SIZE_LIMIT` | Compiled-regex size limit. | `polars-utils/src/regex_cache.rs` |
| `POLARS_TIMEOUT_MS` | Operation timeout (ms). | `polars-python/src/timeout.rs` |
| `POLARS_TABLE_WIDTH` | Printed-table width. | `polars-core/src/fmt.rs` |
| `POLARS_FMT_*` | Display formatting (mirrors `pl.Config`). | `polars-core` |

> [!NOTE]
> Where an environment variable and a `pl.Config` setting overlap (mostly display/formatting), `pl.Config` is the ergonomic API; the env var is useful for setting behavior before the process imports Polars, or in containers where you cannot edit code.

---

## 9. Deployment Patterns

Because Polars is an embedded library, "deployment" means packaging it inside a host application.

### Containerized batch / ETL job

```dockerfile
FROM python:3.12-slim
RUN pip install --no-cache-dir 'polars[parquet,fsspec]'   # only the extras you use
COPY job.py /app/job.py
ENV POLARS_MAX_THREADS=8 \
    POLARS_OOC_SPILL_DIR=/scratch
ENTRYPOINT ["python", "/app/job.py"]
```

> [!TIP]
> Pin `POLARS_MAX_THREADS` to the container's CPU quota. Polars sizes its thread pool from the host's visible cores, which in a cgroup-limited container can be far more than the quota — leading to oversubscription and context-switch thrashing.

### Larger-than-RAM job

Scan lazily, collect with the streaming engine, and stream output through a sink:

```python
(
    pl.scan_parquet("s3://bucket/events/*.parquet")
      .filter(pl.col("ts") >= cutoff)
      .group_by("tenant").agg(pl.len())
      .sink_parquet("s3://bucket/out/summary.parquet")
)
```

Set `POLARS_OOC_SPILL_DIR` to fast local scratch (NVMe) and give the container enough ephemeral disk for spills.

### Rust service embedding

```toml
polars = { version = "x", features = ["lazy", "streaming", "parquet"] }
```

Build with `--release`, and consider `performant`/`nightly` for hot paths. The `polars-dylib` crate provides a dynamic-library build if you need a shared object rather than static linking.

### GPU node

Install `polars[gpu]` on an NVIDIA node and call `collect(engine="gpu")`. Unsupported operators fall back to CPU, so a query never fails purely for lack of GPU support — but verify with `explain` that the heavy operators actually run on the GPU.

---

## 10. Operations: Resourcing, Performance, Memory

### Thread pool

The compute thread pool is fixed at first use. Size it with `POLARS_MAX_THREADS` (env, before import) and inspect it with `pl.thread_pool_size()`.

> [!WARNING]
> Mixing Polars' internal parallelism with `multiprocessing` can fork a process that already holds the thread pool, causing deadlocks. If you must use process-level parallelism, create Polars frames *after* forking, or use the `spawn` start method.

### Memory

- Prefer **Parquet** over CSV/Excel for large inputs — columnar + compressed + prunable.
- Use **lazy + streaming** to bound peak memory; use **sinks** for large outputs.
- Set `POLARS_OOC_SPILL_DIR` to fast disk so spilling does not become the bottleneck.
- `POLARS_OVERRIDE_TOTAL_MEMORY` helps in containers where the engine misreads available memory.

### Performance checklist

- [ ] Use `scan_*` (not `read_*`) for files, so the optimizer prunes columns/rows.
- [ ] Build the whole lazy chain and `collect()` **once** at the end.
- [ ] Read `lf.explain()` to confirm predicate/projection pushdown happened.
- [ ] Enable `cse` for queries with repeated subexpressions.
- [ ] Cast high-cardinality repeated strings to `Categorical`/`Enum` before joins/group-bys.
- [ ] Avoid `map_elements` (per-row Python); prefer native expressions or Rust plugins.
- [ ] Pin `POLARS_MAX_THREADS` to the real CPU allocation in containers.

### Observability

`POLARS_VERBOSE`, `POLARS_TRACK_METRICS`, and `POLARS_LOG_METRICS` surface engine activity; `POLARS_VISUALIZE_IR` / `POLARS_VISUALIZE_PHYSICAL_PLAN` dump plans for debugging.

---

## 11. Security Analysis

Polars is an in-process library with **no network listener, no authentication, and no privilege model of its own** — it runs with the privileges of the host process. The security surface is therefore about *what data and inputs you feed it* and *what extensions you load*.

| Area | Consideration | Mitigation |
| --- | --- | --- |
| **Untrusted files** | Parsing attacker-controlled CSV/Parquet/JSON/IPC can trigger large allocations or pathological work (e.g. the wide-sheet `read_excel` OOM, issue #27677). | Run parsing in a resource-limited sandbox (cgroups/ulimits); cap rows/columns at read time; prefer Parquet with validated schemas. |
| **Cloud credentials** | `storage_options` and credential providers (`CredentialProviderAWS/Azure/GCP`) handle secrets. | Inject via environment/instance roles, never hard-code; scope IAM to the buckets/prefixes used. |
| **Database connection strings** | `read_database_uri` takes a URI that may embed credentials. | Pass credentials out-of-band; avoid logging the URI; use least-privilege DB roles. |
| **SQL frontend** | `pl.SQLContext`/`pl.sql` execute SQL against in-memory frames — there is no external database to inject into, but building SQL strings from untrusted input is still unwise. | Parameterize / validate any user-supplied SQL fragments; prefer the expression API for untrusted input. |
| **User-defined functions** | `map_elements`/`map_batches` execute arbitrary Python; `cloudpickle` serialization of UDFs (the `cloudpickle` extra) deserializes code. | Never unpickle UDFs from untrusted sources — pickled callables are arbitrary code execution. |
| **Native plugins** | Expression and IO plugins are compiled native code loaded into the process; they run with full host privileges and are **not sandboxed**. | Load only plugins you build or audit; treat a third-party plugin like any native dependency. |
| **FFI boundary** | `polars-ffi`/`pyo3-polars` pass Arrow data across the ABI; misuse of the unsafe FFI surface can cause memory unsafety. | Use the provided safe wrappers; do not hand-roll the C ABI. |
| **Panic vs. error** | `POLARS_PANIC_ON_ERR` turns errors into panics; a panic across the FFI boundary is undefined behavior if not caught. | Leave `POLARS_PANIC_ON_ERR` off in production; rely on returned errors. |

> [!IMPORTANT]
> The single highest-impact hardening step for a service that ingests untrusted data files is to **bound resources** (memory cgroup, file-size limits, row/column caps) around the parse, because a hostile input's main lever is forcing a huge allocation — exactly the failure mode behind the reported `read_excel` and `MemoryError` issues.

---

## 12. Extending Polars

Polars exposes a public extension SDK in `pyo3-polars/` so you can add compiled functionality without forking the engine.

### Expression plugins

Write a custom expression in Rust, annotate it, compile it as a small extension module, and call it from Python at near-native speed:

```rust
use pyo3_polars::derive::polars_expr;
use polars::prelude::*;

#[polars_expr(output_type = Float64)]
fn my_kernel(inputs: &[Series]) -> PolarsResult<Series> {
    // operate on inputs[0] ...
    Ok(out)
}
```

The `#[polars_expr]` proc-macro lives in `pyo3-polars-derive`. Worked examples are in `pyo3-polars/example/derive_expression/`. This is the recommended path when a transformation cannot be expressed with native expressions but `map_elements` is too slow.

### IO plugins

Register a custom data source so lazy scanning works against it (example in `pyo3-polars/example/io_plugin/`). The source integrates with the streaming engine's source nodes.

### Python dispatch extension

`pyo3-polars/example/extend_polars_python_dispatch/` shows exposing extra Rust functions over `DataFrame`/`Series` to Python.

> [!NOTE]
> Plugins compile against a specific Polars ABI. Build them against the same Polars version your users run, and rebuild on upgrade — a plugin built for one engine version may not load against another.

---

## 13. Interoperability and the Foreign Function Interface

Polars is built on Arrow, so interchange with the Arrow ecosystem is cheap (often zero-copy):

- **PyArrow:** `df.to_arrow()` / `pl.from_arrow(...)`.
- **pandas / NumPy:** `to_pandas`/`from_pandas`, `to_numpy`/`from_numpy` (require the respective extras).
- **DataFrame Interchange Protocol:** `pl.from_dataframe(...)` consumes any frame implementing the protocol.
- **C ABI:** `polars-ffi` and `pyo3-polars` move Arrow arrays across the ABI boundary for cross-language plugins.

`CompatLevel` controls the Arrow compatibility level when exporting, for consumers that expect a particular Arrow version's layout.

---

## 14. Code Quality and Conventions

- **Workspace discipline.** One responsibility per crate, layered bottom-up; a crate depends only on lower layers. The umbrella `polars` crate is the only intended public entry point for Rust users (via `polars::prelude::*`).
- **Error handling.** Centralized in `polars-error`: the `polars_bail!` / `polars_err!` macros construct typed `PolarsError`s with a consistent message style. Feature-gated code paths emit `activate '<feature>' feature` errors when a needed Cargo feature is off — a deliberate, discoverable failure mode.
- **Feature-flagging.** Functionality is gated so that builds stay minimal; this is pervasive and is the main reason the dependency graph stays light.
- **SIMD & specialization.** Hot kernels live in `polars-compute`; the `nightly` feature unlocks portable SIMD and specialization.
- **Lints.** A workspace `clippy.toml` enforces lint configuration across crates.
- **Typed Python layer.** The Python package ships type stubs (`_plr.pyi`, `py.typed`) and validates inputs early, raising `ValueError`/`TypeError`/`DuplicateError` from `config.py`, `schema.py`, and `selectors.py` before work reaches the engine.

---

## 15. Testing Strategy

| Layer | Location | Notes |
| --- | --- | --- |
| Rust integration tests | `crates/polars/tests/it/` | End-to-end engine tests (core ops, arrow, IO, lazy, time). |
| Rust unit tests | Inline per crate | Standard `#[cfg(test)]` modules. |
| Python unit tests | `py-polars/tests/unit/` | The bulk of functional coverage — dataframe, lazyframe, series, expr, io, sql, operations, datatypes. |
| Python benchmarks | `py-polars/tests/benchmark/` | Including H2O.ai-style group-by/join benchmarks and PDS-H. |
| Parametric/property tests | `py-polars/tests/unit/testing/parametric/` | Hypothesis-style strategies over schemas/dtypes. |
| Doc tests | `py-polars/tests/docs/` | Verify documented examples run. |
| Assertion helpers | `polars-testing`, `polars.testing` | `assert_frame_equal`, `assert_series_equal`. |

> [!TIP]
> When contributing, the Python `tests/unit/` tree is organized to mirror the API (operations/namespaces/string, io/database, lazyframe, …) — add a test next to the feature area you touch.

---

## 16. Potential Improvements and Tech Debt

These observations are derived from the source tree and the project's own open issues; they are pointers for maintainers and advanced operators, not defects in your installation.

- **Wide-input allocation guards.** `read_excel` can allocate for the full sheet width when a stray cell sits in a far column (issue #27677), causing multi-GB allocations. A bounded/streamed Excel reader path would harden ingestion of untrusted spreadsheets.
- **Literal dtype consistency.** Scalar vs. list literal integer inference differs (`pl.lit(1)` → `Int32`, `pl.lit([1])` element → `Int64`; issues #13114, #27894). Tightening literal type inference would remove a class of surprising join/comparison mismatches.
- **Streaming sink robustness.** `sink_delta` panics on null `Struct` values (issue #27079) and some operations still panic rather than returning typed errors (e.g. `hist` on `String`, issue #27155). Converting remaining engine panics into recoverable `PolarsError`s improves embeddability, since a panic across the PyO3/FFI boundary is hazardous.
- **Parquet predicate pruning.** Row-group pruning via max-statistics is not applied for some float predicate shapes (issue #27860); extending statistics-based pruning would cut I/O on selective float filters.
- **Thread-pool/config mutability.** Including the thread-pool size in mutable `Config` was flagged as unsafe (issue #27817); the thread pool's fixed-at-startup nature should be reflected consistently in the config surface.
- **Streaming docs.** The streaming concept is intentionally under-documented upstream ("not included until we have something we are proud of"); operationally it is production-relevant, so teams relying on it should track the engine's release notes closely.

---

## 17. Troubleshooting (Operational)

### Symptom: import fails with a CPU/illegal-instruction error

The default wheel needs AVX2. Install `polars[rtcompat]`, or as a diagnostic set `POLARS_SKIP_CPU_CHECK=1` (the compatibility wheel is the real fix). Optionally `POLARS_DISABLE_AVX512=1` to rule out AVX-512 paths.

### Symptom: process uses far more threads than the CPU quota

Polars sized its pool from the host's visible cores. Set `POLARS_MAX_THREADS` to the container's quota and confirm with `pl.thread_pool_size()`.

### Symptom: OOM-killed on a large job

Switch from `read_*`/`collect()` to `scan_*` + `collect(engine="streaming")` or a `sink_*`. Set `POLARS_OOC_SPILL_DIR` to fast disk and `POLARS_OVERRIDE_TOTAL_MEMORY` if the engine misreads container memory.

### Symptom: cloud reads are slow or rate-limited

Tune `POLARS_DOWNLOAD_CHUNK_SIZE`, `POLARS_CONCURRENCY_BUDGET`, and the upload chunk/concurrency variables in `polars-io/src/configs.rs`. Confirm projection/predicate pushdown is reaching the object store with `lf.explain()`.

### Symptom: a feature raises `activate '<x>' feature`

A required Cargo feature was not compiled in (Rust) — rebuild with that feature. In Python this usually surfaces instead as a `ModuleNotFoundError` for an optional dependency; install the corresponding extra.

### Symptom: a panic with no detail

Set `POLARS_BACKTRACE_IN_ERR=1` to attach a Rust backtrace, reproduce minimally, and file upstream. Do **not** run with `POLARS_PANIC_ON_ERR=1` in production.

---

## 18. Quick Reference

### Crate layers (bottom → top)

`polars-arrow` / `polars-buffer` / `polars-utils` → `polars-core` (+`polars-compute`, `polars-ops`, `polars-time`) → `polars-plan` (+ `polars-sql`, `polars-lazy`) → `polars-expr` / `polars-mem-engine` / `polars-stream` / `polars-ooc` → `polars-io` (+ `polars-parquet`, `polars-json`) → `polars-python` / `polars-ffi` → `polars` (facade).

### Engine selection

| Goal | Call |
| --- | --- |
| In-memory | `lf.collect()` |
| Streaming | `lf.collect(engine="streaming")` |
| GPU | `lf.collect(engine="gpu")` |
| Multiple plans | `pl.collect_all([...])` |
| Stream to disk | `lf.sink_parquet(...)` / `sink_ipc` / `sink_csv` / `sink_ndjson` |
| Inspect plan | `lf.explain()`, `lf.explain(optimized=False)`, `lf.show_graph()` |

### Highest-value environment variables

| Variable | Use |
| --- | --- |
| `POLARS_MAX_THREADS` | Pin thread-pool size (containers). |
| `POLARS_OOC_SPILL_DIR` | Out-of-core spill location. |
| `POLARS_OVERRIDE_TOTAL_MEMORY` | Correct memory detection in cgroups. |
| `POLARS_VERBOSE` | Engine logging. |
| `POLARS_BACKTRACE_IN_ERR` | Backtraces for bug reports. |
| `POLARS_FORCE_PKG` / `POLARS_PREFER_PKG` | Select the runtime variant. |

### Build profiles

| Target | Command (in `py-polars`) |
| --- | --- |
| Dev (fast compile) | `make build` |
| Release | `make build-release` |
| Release, no debug syms | `make build-nodebug-release` |
| Fastest binary | `make build-dist-release` |
| Older CPU | prefix any with `LTS_CPU=1` |

### Key paths in the repository

| Path | Contents |
| --- | --- |
| `crates/` | The 29 engine crates. |
| `crates/polars-plan/src/plans/optimizer/` | Optimizer passes. |
| `crates/polars-stream/src/nodes/` | Streaming sources/sinks/operators. |
| `py-polars/src/polars/` | The Python package (src-layout). |
| `pyo3-polars/` | Plugin SDK + examples. |
| `docs/source/user-guide/` | Official user guide source. |

### Official resources

- User guide & API docs: <https://docs.pola.rs/>
- Rust API (docs.rs): <https://docs.rs/polars/latest/polars/>
- Contributing guide: <https://docs.pola.rs/development/contributing/>
- Managed/distributed: <https://cloud.pola.rs/>

---

_Generated June 2026 from pola-rs/polars at 1c11555. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._