Polars User Guide

Polars is an extremely fast query engine for DataFrames, written in Rust. It is designed to be fast, easy to use, and expressive. You write data transformations as expressions, and Polars compiles and runs them across all your CPU cores — eagerly for interactive work, or lazily so the query optimizer can rewrite the plan before a single row is read.

This guide is task-oriented. It takes you from installation to your first DataFrame, through every everyday operation (selecting, filtering, aggregating, joining), into the expression API that makes Polars powerful, and finally into the lazy and streaming engines that let you process datasets larger than memory. Concepts are explained as you need them; deep dives live near the end.

Note

Examples in this guide use the Python API (import polars as pl), the most widely used Polars frontend. Polars also has first-class Rust, Node.js, R, and SQL frontends; the Rust equivalent is shown where it differs meaningfully. The underlying engine is identical across all of them.

Table of Contents

1. Quick Start
2. Installation
3. Core Ideas: Expressions, Contexts, Eager vs. Lazy
4. Reading and Writing Data
5. Selecting and Creating Columns
6. Filtering Rows
7. Aggregating and Grouping
8. Sorting, Reshaping, and Combining
9. Joining DataFrames
10. The Expression API in Depth
11. Working with Specific Data Types
12. The Lazy API
13. Streaming: Larger-Than-RAM Data
14. Querying with SQL
15. Interoperability (pandas, NumPy, Arrow)
16. Controlling Display and Output
17. How It Works
18. Troubleshooting
19. Frequently Asked Questions
20. Reference

1. Quick Start

Install Polars and run your first query in under a minute.

Python

pip install polars

Rust

cargo add polars -F lazy

Create a DataFrame, transform it, and inspect the result:

import polars as pl
import datetime as dt

df = pl.DataFrame(
    {
        "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
        "birthdate": [
            dt.date(1997, 1, 10),
            dt.date(1985, 2, 15),
            dt.date(1983, 3, 22),
            dt.date(1981, 4, 30),
        ],
        "weight": [57.9, 72.5, 53.6, 83.1],  # (kg)
        "height": [1.56, 1.77, 1.65, 1.75],  # (m)
    }
)

print(df)

shape: (4, 4)
┌────────────────┬────────────┬────────┬────────┐
│ name           ┆ birthdate  ┆ weight ┆ height │
│ ---            ┆ ---        ┆ ---    ┆ ---    │
│ str            ┆ date       ┆ f64    ┆ f64    │
╞════════════════╪════════════╪════════╪════════╡
│ Alice Archer   ┆ 1997-01-10 ┆ 57.9   ┆ 1.56   │
│ Ben Brown      ┆ 1985-02-15 ┆ 72.5   ┆ 1.77   │
│ Chloe Cooper   ┆ 1983-03-22 ┆ 53.6   ┆ 1.65   │
│ Daniel Donovan ┆ 1981-04-30 ┆ 83.1   ┆ 1.75   │
└────────────────┴────────────┴────────┴────────┘

Compute new columns with an expression:

result = df.select(
    pl.col("name"),
    pl.col("birthdate").dt.year().alias("birth_year"),
    (pl.col("weight") / (pl.col("height") ** 2)).alias("bmi"),
)
print(result)

shape: (4, 3)
┌────────────────┬────────────┬───────────┐
│ name           ┆ birth_year ┆ bmi       │
│ ---            ┆ ---        ┆ ---       │
│ str            ┆ i32        ┆ f64       │
╞════════════════╪════════════╪═══════════╡
│ Alice Archer   ┆ 1997       ┆ 23.791913 │
│ Ben Brown      ┆ 1985       ┆ 23.141498 │
│ Chloe Cooper   ┆ 1983       ┆ 19.687787 │
│ Daniel Donovan ┆ 1981       ┆ 27.134694 │
└────────────────┴────────────┴───────────┘

Write to disk and read it back:

df.write_csv("people.csv")
df_csv = pl.read_csv("people.csv", try_parse_dates=True)

Tip

The single most important idea in Polars is the expression. pl.col("weight") / (pl.col("height") ** 2) is not a value — it is a recipe that Polars runs against a column, in parallel, inside a context like select. Master expressions and the rest of the API follows naturally. See Core Ideas.

2. Installation

Polars installs through your language's package manager. The core has zero required dependencies; optional features are opt-in to keep the footprint small.

Python

pip install polars

Rust

cargo add polars -F lazy

Or in Cargo.toml:

[dependencies]
polars = { version = "x", features = ["lazy"] }

Then import it:

Python

import polars as pl

Rust

use polars::prelude::*;

Python optional dependencies (extras)

Install extras with bracket syntax, e.g. pip install 'polars[numpy,fsspec]'.

Note

Throughout this guide, when a feature needs an extra we say so. If you are unsure, pip install 'polars[all]' installs everything; trim it down later for a leaner environment.

Big index builds (>4.3 billion rows)

By default a Polars DataFrame is limited to 2³² (~4.3 billion) rows. If you expect more, install the big-index build, which uses a 64-bit row index (2⁶⁴ ≈ 18 quintillion rows):

Python

pip install polars[rt64]

Rust

cargo add polars -F bigidx

Caution

Only use bigidx/rt64 if you actually hit the 2³² row boundary. The default build is faster and uses less memory because its data structures are more cache-efficient.

Legacy CPUs

The default Python wheels are compiled for modern CPUs (AVX2). On an older CPU (roughly pre-2011), or on an x86-64 Python running under Rosetta on Apple Silicon, install the compatibility build:

pip install polars[rtcompat]

Compiling from source (Python)

For a bleeding-edge build or maximum performance, compile against your own CPU:

1. Install the latest Rust compiler.
2. pip install maturin
3. cd py-polars and run one of:

Tip

Pass LTS_CPU=1 with any build command to target an older CPU that lacks AVX2.

Verifying your installation

import polars as pl
pl.show_versions()

This prints the Polars version, the index type (32- or 64-bit), and the status of every optional dependency — the first thing to check when debugging an environment problem.

3. Core Ideas: Expressions, Contexts, Eager vs. Lazy

Three ideas underpin everything in Polars. Internalize them once and the entire API becomes predictable.

Expressions

An expression is a description of a computation over one or more columns. It does nothing on its own:

pl.col("weight") / (pl.col("height") ** 2)

The line above does not produce a number. It produces an Expr object — an abstract recipe meaning "take weight, divide it by height squared." Polars only materializes an expression when you run it inside a context.

Contexts

A context is a DataFrame (or LazyFrame) method that takes expressions and applies them. The four contexts you will use constantly:

# select: only the computed columns come out
df.select(pl.col("name"), pl.col("weight") * 0.95)

# with_columns: original columns kept, new ones appended
df.with_columns(bmi=pl.col("weight") / pl.col("height") ** 2)

# filter: keep matching rows
df.filter(pl.col("height") > 1.7)

# group_by + agg: one row per group
df.group_by("name").agg(pl.col("weight").mean())

Important

Because expressions are independent recipes, Polars runs the expressions inside a context in parallel, across all CPU cores, automatically. You get multi-threaded execution without writing any threading code.

Eager vs. Lazy execution

Polars has two execution modes that share the same expression API:

• Eager — methods on a DataFrame run immediately, line by line. Best for interactive exploration and small data.
• Lazy — methods on a LazyFrame build up a query plan. Nothing runs until you call .collect(). This lets the query optimizer rewrite your query (pushing filters down to the file reader, dropping unused columns, eliminating common subexpressions) before any data is touched.

# Eager: executes now
df = pl.read_csv("people.csv")
result = df.filter(pl.col("height") > 1.7)

# Lazy: builds a plan, runs only at collect()
lf = pl.scan_csv("people.csv")
result = lf.filter(pl.col("height") > 1.7).collect()

Warning

A common surprise for newcomers: a lazy query produces no output until you call .collect(). If you build a LazyFrame chain and "nothing happens," you almost certainly forgot the final .collect(). The lazy API is covered fully in The Lazy API.

Tip

Rule of thumb: explore eagerly with read_*, but run production pipelines lazily with scan_* so the optimizer and streaming engine can help you.

4. Reading and Writing Data

Polars reads and writes the common columnar and text formats, plus cloud storage and databases. Every format has an eager reader (read_*, returns a DataFrame) and most also have a lazy scanner (scan_*, returns a LazyFrame and enables predicate/projection pushdown).

Format support matrix

CSV

# Read, asking Polars to detect and parse date columns
df = pl.read_csv("people.csv", try_parse_dates=True)

# Write
df.write_csv("out.csv")

# Lazy scan (preferred for large files)
lf = pl.scan_csv("people.csv")

Common read_csv parameters: separator, has_header, columns (subset), schema_overrides, skip_rows, n_rows, null_values, encoding, infer_schema_length.

Parquet

Parquet is the recommended format for analytics: columnar, compressed, and self-describing. Always prefer scan_parquet for large datasets so Polars can prune row groups and columns.

df = pl.read_parquet("data.parquet")
lf = pl.scan_parquet("data/*.parquet")        # globs work
df.write_parquet("data.parquet", compression="zstd")

You can read just the schema or metadata without loading rows:

schema = pl.read_parquet_schema("data.parquet")
meta = pl.read_parquet_metadata("data.parquet")

Reading many files at once

scan_* functions accept a glob or a list of paths and concatenate them into one frame:

lf = pl.scan_parquet("year=2024/**/*.parquet")   # hive-partitioned tree
df = lf.collect()

Cloud storage

With the fsspec extra (and the relevant cloud SDK), read_*/scan_* accept s3://, gs://, and az:// URIs. For credentials, pass a storage_options dict or use one of the built-in credential providers (CredentialProviderAWS, CredentialProviderAzure, CredentialProviderGCP).

lf = pl.scan_parquet(
    "s3://my-bucket/data/*.parquet",
    storage_options={"aws_region": "us-east-1"},
)

Databases

# Via a SQLAlchemy/ADBC connection or URI
df = pl.read_database_uri(
    query="SELECT * FROM users WHERE active",
    uri="postgresql://user:pass@host:5432/db",
)
df.write_database("users_copy", connection="postgresql://…")

Note

read_database takes an existing connection object; read_database_uri takes a connection string and chooses an engine (ConnectorX by default, ADBC if requested). ConnectorX is fastest for reads; SQLAlchemy and ADBC support writes.

Excel

# Requires: pip install 'polars[calamine]'  (or another engine)
df = pl.read_excel("report.xlsx", sheet_name="Q1")
df.write_excel("out.xlsx", worksheet="Summary")

5. Selecting and Creating Columns

select — pick and compute columns

df.select("name", "weight")                       # by name
df.select(pl.col("name"), pl.col("weight") * 2)   # with expressions
df.select(pl.all())                               # everything
df.select(pl.all().exclude("birthdate"))          # everything except

with_columns — add or replace columns

df.with_columns(
    birth_year=pl.col("birthdate").dt.year(),
    bmi=pl.col("weight") / (pl.col("height") ** 2),
)

Named keyword arguments become column names. Equivalently, use .alias():

df.with_columns(
    pl.col("birthdate").dt.year().alias("birth_year"),
)

Expression expansion — one expression, many columns

A single expression can target several columns at once and produce several output columns. This is expression expansion and is unique to Polars' design:

result = df.select(
    pl.col("name"),
    (pl.col("weight", "height") * 0.95).round(2).name.suffix("-5%"),
)

Here pl.col("weight", "height") * 0.95 expands into two computations, and .name.suffix("-5%") renames each result. Use selectors (next) for pattern-based expansion.

Selectors — choose columns by property

The polars.selectors module (imported as cs) selects columns by dtype, name pattern, or position:

import polars.selectors as cs

df.select(cs.numeric())               # all numeric columns
df.select(cs.string())                # all string columns
df.select(cs.starts_with("weight"))   # name pattern
df.select(cs.numeric() - cs.first())  # set algebra on selectors

Tip

Selectors compose with set operators: cs.numeric() & cs.matches("2024"), cs.all() - cs.string(). They are the cleanest way to apply one transformation to a whole family of columns.

6. Filtering Rows

filter keeps rows where a boolean expression is true:

df.filter(pl.col("birthdate").dt.year() < 1990)

Multiple conditions passed as separate arguments are combined with logical AND:

df.filter(
    pl.col("birthdate").is_between(dt.date(1982, 12, 31), dt.date(1996, 1, 1)),
    pl.col("height") > 1.7,
)

For OR / NOT, combine boolean expressions explicitly:

df.filter((pl.col("height") > 1.8) | (pl.col("weight") < 60))
df.filter(~pl.col("name").str.starts_with("A"))

Useful boolean expression methods: is_in, is_between, is_null, is_not_null, is_nan, is_finite, is_duplicated, is_unique.

df.filter(pl.col("name").is_in(["Ben Brown", "Alice Archer"]))

7. Aggregating and Grouping

Whole-frame aggregations

Aggregation expressions used in select collapse each column to one value:

df.select(
    pl.col("weight").mean().alias("avg_weight"),
    pl.col("height").max().alias("tallest"),
    pl.len().alias("n_rows"),
)

group_by(...).agg(...)

Split rows into groups, then compute per-group expressions:

result = df.group_by(
    (pl.col("birthdate").dt.year() // 10 * 10).alias("decade"),
    maintain_order=True,
).agg(
    pl.len().alias("sample_size"),
    pl.col("weight").mean().round(2).alias("avg_weight"),
    pl.col("height").max().alias("tallest"),
)

shape: (3, 4)
┌────────┬─────────────┬────────────┬─────────┐
│ decade ┆ sample_size ┆ avg_weight ┆ tallest │
│ ---    ┆ ---         ┆ ---        ┆ ---     │
│ i32    ┆ u32         ┆ f64        ┆ f64     │
╞════════╪═════════════╪════════════╪═════════╡
│ 1990   ┆ 1           ┆ 57.9       ┆ 1.56    │
│ 1980   ┆ 3           ┆ 69.73      ┆ 1.77    │
└────────┴─────────────┴────────────┴─────────┘

Note

maintain_order=True preserves the order in which groups first appear. It costs a little performance, so omit it when group order does not matter.

Inside agg, an expression that does not reduce to a scalar (for example pl.col("name")) produces a list per group, letting you gather group members:

df.group_by("decade").agg(pl.col("name"))   # name column becomes List[str]

Common aggregation expressions

sum, mean, median, min, max, std, var, quantile, count, n_unique, first, last, len, arg_min, arg_max. There are also horizontal (across-columns) variants: pl.sum_horizontal, pl.mean_horizontal, pl.max_horizontal, pl.min_horizontal, pl.any_horizontal, pl.all_horizontal.

Window functions: .over()

Aggregate within groups without collapsing rows, by appending .over() to an expression:

df.with_columns(
    avg_weight_by_decade=pl.col("weight")
        .mean()
        .over(pl.col("birthdate").dt.year() // 10),
)

This adds the group average back to every row — the equivalent of a SQL window function.

8. Sorting, Reshaping, and Combining

Sorting

df.sort("weight")                          # ascending
df.sort("weight", descending=True)
df.sort(["decade", "weight"], descending=[False, True])   # multi-key

Selecting rows

df.head(10)        # first 10 rows
df.tail(10)        # last 10 rows
df.sample(5)       # random 5 rows
df.slice(10, 5)    # 5 rows starting at offset 10
df.unique()        # drop duplicate rows

Reshaping

df.pivot(values="weight", index="decade", on="name", aggregate_function="mean")

Combining frames

9. Joining DataFrames

Combine two frames on one or more key columns:

df.join(other, on="name", how="left")
df.join(other, left_on="id", right_on="user_id", how="inner")

Join types

As-of joins

Join on the nearest key rather than an exact match — essential for time series (e.g. matching trades to the most recent quote):

trades.join_asof(quotes, on="timestamp", by="symbol", strategy="backward")

Tip

Both frames must be sorted on the on key for join_asof. strategy may be "backward" (default), "forward", or "nearest".

10. The Expression API in Depth

Expressions are built by chaining methods onto a starting point. The most common starting points are module-level functions:

Conditional logic with when/then/otherwise

df.with_columns(
    category=pl.when(pl.col("bmi") < 18.5).then(pl.lit("underweight"))
              .when(pl.col("bmi") < 25).then(pl.lit("normal"))
              .otherwise(pl.lit("overweight")),
)

Chaining transformations

Expressions read left to right as a pipeline:

pl.col("name").str.to_lowercase().str.replace(" ", "_").alias("slug")

Namespaces

Type-specific operations live under namespaces on an expression:

df.with_columns(
    first_name=pl.col("name").str.split(" ").list.first(),
    birth_month=pl.col("birthdate").dt.month(),
)

Folds — efficient horizontal reduction

To reduce across many columns efficiently, use pl.fold (or the *_horizontal helpers):

pl.fold(acc=pl.lit(0), function=lambda acc, x: acc + x, exprs=pl.col("*"))

User-defined functions

When no built-in expression fits, drop into Python — but prefer native expressions first, since UDFs run row-by-row and lose parallelism guarantees:

df.with_columns(
    pl.col("weight").map_elements(my_func, return_dtype=pl.Float64)  # per element
)
df.with_columns(
    pl.col("weight").map_batches(my_vectorized_func)                 # per Series
)

Warning

map_elements calls your Python function once per value and is slow. Reach for it only when a transformation truly cannot be expressed with native expressions. For compiled speed, write the UDF in Rust as a plugin (see How It Works).

11. Working with Specific Data Types

Polars has a rich, Arrow-backed type system. The headline types:

Casting

df.with_columns(pl.col("weight").cast(pl.Int32))

Missing data

Polars distinguishes null (missing) from NaN (float not-a-number):

df.with_columns(pl.col("weight").fill_null(0))
df.with_columns(pl.col("weight").fill_null(strategy="forward"))
df.filter(pl.col("weight").is_not_null())
df.drop_nulls()

Strings

df.select(
    pl.col("name").str.to_uppercase(),
    pl.col("name").str.contains("Brown"),
    pl.col("name").str.split(" ").list.len().alias("word_count"),
)

Temporal

df.select(
    pl.col("birthdate").dt.year(),
    pl.col("birthdate").dt.weekday(),
    pl.col("birthdate").dt.offset_by("1mo"),
    pl.col("birthdate").dt.truncate("1y"),
)

Note

Timezone handling needs the timezone extra on Windows (it is built in on Linux/macOS). The temporal Rust feature enables Chrono interop on the Rust side.

Categorical and Enum

Use Categorical for repeated string values to save memory and speed up grouping/joining. Use Enum when you know the full set of categories ahead of time:

df.with_columns(pl.col("name").cast(pl.Categorical))
ratings = pl.Enum(["low", "medium", "high"])

Tip

When working with multiple categorical columns that must share an encoding (for joins/comparisons), enable the global string cache: pl.enable_string_cache() or the with pl.StringCache(): context manager.

12. The Lazy API

The lazy API is where Polars delivers its biggest wins. Instead of executing each step immediately, you build a query plan; Polars optimizes the whole plan, then runs it on .collect().

Use the lazy API because it:

• lets the query optimizer rewrite your query (predicate pushdown, projection pushdown, common-subexpression elimination, slice pushdown);
• enables streaming of larger-than-memory datasets;
• catches schema errors before any data is processed.

Starting a lazy query

From a file (best — the optimizer can reduce what is read):

lf = pl.scan_csv("reddit.csv")
result = (
    lf.with_columns(pl.col("name").str.to_uppercase())
      .filter(pl.col("comment_karma") > 0)
      .collect()
)

From an existing DataFrame:

result = df.lazy().filter(pl.col("height") > 1.7).collect()

Important

A LazyFrame does nothing until .collect(). Build the whole chain, then collect once at the end. Calling .collect() mid-chain defeats the optimizer.

Inspecting the plan

lf.explain()                 # optimized plan as text
lf.explain(optimized=False)  # the plan as you wrote it
lf.show_graph()              # visual graph (requires the `graph` extra)

Comparing the optimized and unoptimized plans is the best way to see the optimizer pushing your filter down to the scan.

Running variants

Sinks — streaming output

For pipelines whose output is also large, sinks write results directly to disk in a streaming fashion, never holding the full result in memory:

pl.scan_csv("huge.csv").filter(pl.col("x") > 0).sink_parquet("filtered.parquet")

Sink targets include sink_parquet, sink_ipc, sink_csv, and sink_ndjson.

GPU acceleration

With pip install 'polars[gpu]', run an optimized lazy query on an NVIDIA GPU by passing the GPU engine:

result = lf.collect(engine="gpu")

You can tune behavior with pl.GPUEngine(...). Operations not supported on the GPU fall back to CPU automatically.

13. Streaming: Larger-Than-RAM Data

If your data does not fit in memory, the streaming engine processes the query in batches:

result = (
    pl.scan_parquet("huge/*.parquet")
      .filter(pl.col("amount") > 100)
      .group_by("region")
      .agg(pl.col("amount").sum())
      .collect(engine="streaming")
)

This can let you process a 250 GB dataset on a laptop. Besides easing memory pressure, the streaming engine is often faster than the in-memory engine.

Note

Some operations are inherently non-streaming or not yet implemented for streaming; Polars transparently falls back to the in-memory engine for those parts of the plan. You do not need to do anything — but if you are debugging memory use, inspect the physical plan with lf.show_graph() to see which nodes are memory-intensive.

Tip

Pair streaming with sinks (sink_parquet, etc.) when the output is also too large to hold in memory — the entire read → transform → write path then runs in bounded memory.

14. Querying with SQL

Polars includes a SQL frontend. Register frames as tables and run standard SQL against them — executed by the same optimized engine.

ctx = pl.SQLContext(people=df)
result = ctx.execute(
    "SELECT name, weight FROM people WHERE height > 1.7 ORDER BY weight DESC",
    eager=True,
)

Or the one-shot helper, which auto-registers DataFrames/LazyFrames in scope:

pl.sql("SELECT * FROM df WHERE height > 1.7", eager=True)

Convert a SQL scalar expression into a Polars expression:

df.select(pl.sql_expr("weight / (height * height) AS bmi"))

Note

The SQL frontend supports SELECT, WHERE, GROUP BY, ORDER BY, joins, CTEs (WITH), CREATE TABLE ... AS, and a large library of SQL functions and operators. SQL and the expression API are fully interoperable — mix whichever reads best for a given query.

15. Interoperability (pandas, NumPy, Arrow)

Polars converts to and from the rest of the Python data ecosystem. Conversions to Arrow are zero-copy where possible because Polars is built on the Arrow memory format.

From other formats into Polars

From Polars into other formats

df.to_pandas()      # requires pandas
df.to_numpy()       # requires numpy
df.to_arrow()       # PyArrow table (zero-copy where possible)
df.to_dicts()       # list of row dicts
df.to_dict()        # dict of columns

Tip

Migrating from pandas? The official user guide has a dedicated pandas-comparison section. The biggest mental shift is from in-place mutation and the index to immutable frames + expressions — there is no row index in Polars, and every operation returns a new frame.

16. Controlling Display and Output

The pl.Config object controls how DataFrames print. Set options globally, or scope them with a context manager.

# Global
pl.Config.set_tbl_rows(20)
pl.Config.set_tbl_cols(-1)            # show all columns
pl.Config.set_fmt_str_lengths(100)

# Scoped — reverts on exit
with pl.Config(tbl_rows=50, tbl_cols=20):
    print(df)

Frequently used Config settings:

pl.Config.save_to_file("polars.config")   # persist settings
pl.Config.load_from_file("polars.config")
pl.Config.restore_defaults()

Note

Display and behavior can also be controlled with POLARS_* environment variables (for example POLARS_TABLE_WIDTH). The full environment-variable reference lives in the Admin Guide.

17. How It Works

You do not need this section to use Polars, but it explains why the API is shaped the way it is.

One engine, many frontends

Polars is a Rust query engine. The Python, Rust, Node.js, R, and SQL frontends are thin layers that build the same internal logical plan, which the same optimizer and execution engines run. Performance and semantics are therefore consistent across languages.

Apache Arrow memory format

Data is stored column-by-column in the Apache Arrow columnar format. Columnar storage is cache-friendly and SIMD-friendly, enables zero-copy interchange with other Arrow tools, and is the reason Polars imports in ~70 ms with zero required dependencies.

From expression to result

1. Your expressions and contexts build a logical plan (an abstract description of the query).
2. The optimizer rewrites it — pushing filters and column projections down to the data source, eliminating common subexpressions, reordering operations.
3. A physical plan is produced and dispatched to the in-memory engine, the streaming engine, or the GPU engine.
4. Independent expressions run in parallel across CPU cores.

This is why lazy queries are faster: the optimizer sees the whole query before any data is read.

Eager is lazy underneath

Even eager DataFrame operations are built on the same machinery — an eager call is essentially a one-step lazy query that collects immediately. That is why the expression API is identical in both modes.

Extending Polars

• Expression plugins — write a custom expression in Rust, compile it, and call it from Python with near-native speed (the pyo3-polars crate; see the docs/source/user-guide/plugins/ section of the project).
• IO plugins — register a custom data source so scan_*-style lazy reading works against it.

18. Troubleshooting

Error: ModuleNotFoundError for an optional dependency

Cause: You used a feature whose backing library is an opt-in extra (e.g. read_excel without an Excel engine, or to_pandas without pandas). Solution: Install the relevant extra, e.g. pip install 'polars[excel]' or pip install 'polars[pandas]'. Prevention: Run pl.show_versions() to see which optional dependencies are present before relying on them.

Error: the configured CPU is not supported / illegal instruction at import

Cause: The default wheels are compiled for modern CPUs (AVX2). On an older CPU, or under Rosetta, the default build cannot run. Solution: Install the compatibility build: pip install polars[rtcompat]. Prevention: On legacy hardware always start from rtcompat. As a last-resort diagnostic you can set POLARS_SKIP_CPU_CHECK=1, but the compatibility wheel is the correct fix.

Error: MemoryError / memory allocation of … bytes failed

Cause: An operation tried to materialize more data than fits in RAM — for example reading a very wide Excel sheet, or a non-streaming aggregation over a huge dataset (see issue #27677 for a pathological read_excel case). Solution: Switch to the lazy API with scan_* and run collect(engine="streaming"); restrict columns/rows at read time (columns=, n_rows=); for output, use a sink_* instead of collect. Prevention: Prefer Parquet over Excel/CSV for large data, and always scan large files lazily.

My lazy query produced no output

Cause: A LazyFrame chain only describes a query; it does not run until collected. Solution: Add .collect() (or a sink_*) at the end of the chain. Prevention: Remember the rule: scan_* → transform → collect().

pl.lit(1) and pl.lit([1]) have different integer types

Cause: A scalar integer literal infers as Int32, while a list literal element infers as Int64 (issue #13114). Solution: Pin the dtype explicitly: pl.lit(1, dtype=pl.Int64). Prevention: When literal dtypes matter (e.g. before a join or comparison), always pass an explicit dtype.

ValueError from a Config setting

Cause: A Config.set_* method received an out-of-range or invalid argument (the config.py validators raise ValueError). Solution: Check the method's accepted values; e.g. set_tbl_cols(-1) means "all," but arbitrary negative values for other settings are rejected. Prevention: Use pl.Config.state() to inspect current settings and confirm a change took effect.

DuplicateError building a Schema / frame

Cause: Two columns share the same name. Solution: Rename one with .alias() or .name.suffix(...) before the conflicting operation. Prevention: When using expression expansion, add a .name.suffix/.name.prefix so expanded outputs stay unique.

A panic / PanicException from the engine

Cause: An internal invariant was violated — sometimes a genuine bug (e.g. sink_delta on null struct values, issue #27079; hist on a String column, issue #27155). Solution: Re-run with POLARS_BACKTRACE_IN_ERR=1 (and POLARS_PANIC_ON_ERR=1) to capture a full backtrace, then report it with a minimal reproduction. Prevention: Keep Polars up to date; many panics are fixed quickly upstream.

19. Frequently Asked Questions

Q: How is Polars different from pandas? Polars is a multi-threaded, Arrow-backed query engine with a lazy optimizer and no row index. Operations return new (immutable) frames rather than mutating in place, and transformations are written as composable expressions. It is typically much faster and uses less memory; its import time is ~70 ms versus pandas' ~520 ms.

Q: Do I have to use the lazy API? No — eager DataFrame operations work line by line and are great for exploration. But for performance and for larger-than-memory data, the lazy API (scan_* … collect) is strongly recommended.

Q: When does my expression actually run? Inside a context. pl.col("x") * 2 alone is just a recipe; it runs when placed in select, with_columns, filter, or agg. In the lazy API, the context call is itself deferred until .collect().

Q: Is there a row index like in pandas? No. Polars has no implicit index. If you need a positional index column, add one with pl.int_range/with_row_index.

Q: How do I process data that doesn't fit in memory? Scan it lazily and collect with the streaming engine: pl.scan_parquet(...).…​.collect(engine="streaming"). For large outputs, stream straight to disk with sink_parquet/sink_ipc/sink_csv.

Q: Can I run Polars on a GPU? Yes. Install polars[gpu] and call lf.collect(engine="gpu"). Unsupported operations fall back to the CPU automatically.

Q: How do I write a custom function? Prefer native expressions. If you must, use map_elements (per value, slow) or map_batches (per Series, faster). For compiled speed, write an expression plugin in Rust via pyo3-polars.

Q: How do I handle more than 4.3 billion rows? Install the big-index build: pip install polars[rt64] (Python) or enable the bigidx feature (Rust). Only do this if you actually exceed 2³² rows — the default build is faster.

Q: My old CPU crashes on import. What do I do? Install the compatibility build compiled without AVX: pip install polars[rtcompat].

Q: Can I use SQL? Yes. Register frames with pl.SQLContext (or use pl.sql(...)) and run standard SQL, executed by the same optimized engine. SQL and expressions interoperate freely.

Q: How do I see which optional features are installed? pl.show_versions() prints the version, the index type, and every optional dependency's status.

20. Reference

Top-level constructors and functions (selected)

Reading and writing (selected)

read_csv, read_parquet, read_json, read_ndjson, read_ipc, read_avro, read_excel, read_delta, read_database / read_database_uri, read_clipboard — and the lazy scanners scan_csv, scan_parquet, scan_ndjson, scan_ipc, scan_delta, scan_iceberg, scan_pyarrow_dataset. Writers: write_csv, write_parquet, write_json, write_ndjson, write_ipc, write_avro, write_excel, write_delta, write_database, plus streaming sinks sink_parquet, sink_ipc, sink_csv, sink_ndjson.

Expression namespaces

.str (strings) · .dt (temporal) · .list (lists) · .arr (fixed arrays) · .struct (structs) · .cat (categoricals) · .name (output naming).

Selected environment variables

A few useful ones for end users (the full list is in the Admin Guide):

Official resources

• User guide & API docs: https://docs.pola.rs/
• Python API reference: https://docs.pola.rs/api/python/stable/reference/index.html
• Rust API reference (docs.rs): https://docs.rs/polars/latest/polars/
• Community: StackOverflow tag python-polars, and the project Discord.