# pandas Admin Guide

This guide is for the people who build, package, extend, and operate pandas rather than merely call it: contributors, downstream maintainers, performance engineers, and architects integrating pandas into a larger system. It covers how the library is structured internally, how it is built from source (including its compiled Cython core), how to configure its runtime behavior, the security considerations of its I/O surface, and the developer operations that keep the project moving.

For task-oriented usage — selecting, cleaning, grouping, merging, time series — see the **pandas User Guide**. This guide deliberately goes *under* the public API.

> [!NOTE]
> pandas is a **library**, not an application. It has no server, daemon, or end-user command-line tool. "Operations" here means the build, test, packaging, and contribution workflow — the things a maintainer or packager runs.

---

## 1. Executive Summary

| Attribute | Value |
|---|---|
| Project | pandas — Python data analysis & manipulation library |
| Language | Python (public API) + Cython / C / C++ (performance core) |
| License | BSD 3-Clause |
| Governance | NumFOCUS-sponsored, community-maintained |
| Required runtime deps | `numpy` (`>=1.26,<3`), `python-dateutil` (`>=2.8.2`), `tzdata` (Windows/Emscripten) |
| Supported Python | 3.11 – 3.14, including experimental free-threaded 3.13 |
| Build system | Meson + `meson-python`, Cython `>=3.1,<4`, a C/C++ compiler |
| Source layout | `pandas/` package + `pandas/_libs/` compiled extensions |
| Distribution | PyPI wheels + conda-forge |

pandas' design centers on two labeled containers — `Series` (1-D) and `DataFrame` (2-D) — layered over NumPy arrays and a set of pandas-native **extension arrays**. A uniform method surface (selection, reduction, alignment, group-by, reshape, merge, I/O) is implemented once on a shared base and inherited by both containers. Hot paths (datetime math, parsing, hashing, windowing) drop into compiled Cython in `pandas/_libs/`.

---

## 2. Architecture

### 2.1 Layered view

```
        ┌─────────────────────────────────────────────────────────┐
        │  Public API:  pd.DataFrame, pd.Series, pd.read_*, pd.merge │
        │               pd.concat, pd.to_datetime, options, ...     │
        └───────────────┬─────────────────────────────────────────┘
                        │  (re-exported from pandas/__init__.py __all__)
        ┌───────────────▼─────────────────────────────────────────┐
        │  pandas/core:  frame.py · series.py · generic.py (NDFrame) │
        │   indexing · groupby · reshape · ops · strings · window    │
        └───────────────┬─────────────────────────────────────────┘
        ┌───────────────▼─────────────────────────────────────────┐
        │  Data containers:                                          │
        │   internals/  BlockManager  (column blocks)               │
        │   arrays/     ExtensionArray family (Categorical, Arrow,   │
        │               masked Int/Float/bool, datetime, interval…)  │
        │   dtypes/     dtype registry & casting rules               │
        └───────────────┬─────────────────────────────────────────┘
        ┌───────────────▼─────────────────────────────────────────┐
        │  Compiled core (pandas/_libs/, *.pyx Cython):             │
        │   tslibs/ (timestamps, offsets, parsing, tz, period)      │
        │   window/ (rolling/ewm aggregations) · hashtable · lib    │
        │   vendored C/C++: ujson, klib, numpy datetime, fast_float │
        └───────────────┬─────────────────────────────────────────┘
        ┌───────────────▼─────────────────────────────────────────┐
        │  Foundations:  NumPy ndarray  ·  (optional) Apache Arrow   │
        └─────────────────────────────────────────────────────────┘
```

### 2.2 The `NDFrame` contract

`Series` and `DataFrame` both inherit from the abstract `NDFrame` (`pandas/core/generic.py`). This is what makes the API *uniform*: shared behavior — indexing, alignment, missing-data handling, reductions, I/O method dispatch, the options-aware `__repr__` — is defined once on `NDFrame` and specialized only where the dimensionality genuinely differs. When you learn a method on `Series`, you have very nearly learned it on `DataFrame`.

### 2.3 The internals: `BlockManager`

A `DataFrame` is **not** a Python list of `Series`. Internally, columns of the same dtype are consolidated into 2-D **blocks**, and a `BlockManager` (`pandas/core/internals/managers.py`) maps the user-facing row/column axes onto those blocks. A single-column `Series` uses a `SingleBlockManager`. This design lets column-wise operations run as vectorized array operations across whole blocks rather than column by column.

> [!NOTE]
> The `BlockManager` is private. Downstream code should never construct or unpack it directly — pandas explicitly deprecated passing a `BlockManager` into the `DataFrame` constructor. Build frames from arrays, dicts, or other public constructors instead.

### 2.4 The extension-array system

Beyond NumPy's native dtypes, pandas defines an `ExtensionArray` interface (`pandas/core/arrays/base.py`) that lets a custom array type behave like a first-class column. pandas ships many implementations, and third parties can register their own:

| Extension array | dtype | Purpose |
|---|---|---|
| `Categorical` | `category` | Repeated labels stored once + integer codes |
| Masked `IntegerArray` / `FloatingArray` / `BooleanArray` | `Int64`, `Float64`, `boolean` | Nullable numeric/boolean via a separate validity mask |
| `StringArray` | `string` | Dedicated text storage |
| `DatetimeArray` / `TimedeltaArray` / `PeriodArray` | `datetime64[ns]`, `timedelta64[ns]`, `period` | Time-series natives |
| `IntervalArray` | `interval` | Ranges with open/closed endpoints |
| `ArrowExtensionArray` | `*[pyarrow]` | Apache Arrow-backed storage |
| `SparseArray` | `Sparse[...]` | Memory-efficient mostly-empty data |

The public extension points live under `pandas.api.extensions` (`register_extension_dtype`, `ExtensionArray`, `ExtensionDtype`), `pandas.api.types`, and `pandas.api.indexers`.

### 2.5 The compiled core (`pandas/_libs`)

Performance-critical code is written in **Cython** and compiled to native extension modules. The build (see §4) compiles two families plus vendored C/C++:

| Location | Extension modules |
|---|---|
| `pandas/_libs/tslibs/` | `timestamps`, `timedeltas`, `period`, `offsets`, `parsing`, `conversion`, `tzconversion`, `timezones`, `np_datetime`, `strptime`, `fields`, `ccalendar`, `dtypes`, `nattype`, `vectorized`, `base` |
| `pandas/_libs/window/` | `aggregations` (compiled as C++), `indexers` |
| Vendored C/C++ | `ujson` (JSON), `klib` (hash tables), numpy `datetime`, `fast_float` (fast string→float, Apache-2.0) |

> [!IMPORTANT]
> Because `pandas/_libs` is compiled, an editable/source install that has not built the extensions will raise `ImportError: C extension: ... not built`. The message tells you to run the editable build (see §4.2). `.pyx` source is *not* importable directly.

---

## 3. Package Map

| Path | Responsibility |
|---|---|
| `pandas/__init__.py` | Public API barrel — defines `__all__`, the canonical public surface |
| `pandas/core/frame.py` | `DataFrame` |
| `pandas/core/series.py` | `Series` |
| `pandas/core/generic.py` | `NDFrame` shared base |
| `pandas/core/indexing.py` | `.loc` / `.iloc` / `.at` / `.iat` machinery |
| `pandas/core/groupby/` | Split-apply-combine engine |
| `pandas/core/reshape/` | `merge`, `concat`, `pivot`, `melt`, `get_dummies`, `cut`/`qcut` |
| `pandas/core/arrays/` | Extension-array implementations |
| `pandas/core/internals/` | `BlockManager` and block types |
| `pandas/core/dtypes/` | dtype registry, casting, type inference |
| `pandas/core/window/` | rolling / expanding / ewm |
| `pandas/core/strings/` | the `.str` accessor |
| `pandas/core/computation/` | `eval` / `query` expression engine |
| `pandas/core/config_init.py` | Registers every `pd.set_option` setting |
| `pandas/_config/` | The options framework itself (`get_option`, `option_context`, …) |
| `pandas/io/` | All readers/writers (`parsers/`, `excel/`, `json/`, `sql.py`, `pytables.py`, `parquet.py`, `stata.py`, `formats/`) |
| `pandas/io/formats/` | Display formatting + the `Styler` (`style.py`, Jinja2 templates) |
| `pandas/plotting/` | Plot accessor + the matplotlib backend |
| `pandas/tseries/` | Offsets, frequencies, holiday calendars |
| `pandas/_libs/` | Compiled Cython/C/C++ core |
| `pandas/api/` | Stable public sub-namespaces (`types`, `extensions`, `indexers`, `typing`, `interchange`) |
| `pandas/errors/` | Public exception/warning classes |
| `pandas/_testing/` | Test helpers (`assert_frame_equal`, etc.) |
| `pandas/tests/` | The test suite (the bulk of the repo's files) |
| `asv_bench/` | Airspeed Velocity performance benchmarks |
| `doc/`, `web/` | Sphinx documentation and the project website |
| `scripts/` | Repo maintenance / validation scripts (not shipped to users) |

> [!NOTE]
> The vast majority of the repository's ~2,600 files are under `pandas/tests/`. The shipped, importable surface is much smaller — `pandas/` minus `tests/`, plus the compiled `_libs`.

---

## 4. Building From Source

A normal `pip install pandas` pulls a pre-built wheel and needs no compiler. Building from source — required for contributing or for unsupported platforms — compiles the Cython core.

### 4.1 Build toolchain

| Tool | Constraint | Role |
|---|---|---|
| Python | `3.11`–`3.14` | Runtime |
| Meson | `>=1.2.3,<2` | Build system |
| `meson-python` | `>=0.18` | PEP 517 build backend |
| Cython | `>=3.1.0,<4` | Compiles `.pyx` → C/C++ |
| C compiler | platform | Compiles generated C (and vendored C) |
| C++ compiler | platform | Compiles `window/aggregations` + `fast_float` |
| `versioneer` | `>=0.29` | Derives version from git tags |
| NumPy | `>=1.26,<3` | Build-time headers + runtime |

### 4.2 Editable development build

```bash
# Clone and create the dev environment (conda/mamba shown; see environment.yml)
git clone https://github.com/pandas-dev/pandas
cd pandas
conda env create -f environment.yml
conda activate pandas-dev

# Build the C/Cython extensions in editable mode
python -m pip install -ve . --no-build-isolation -Ceditable-verbose=true
```

> [!TIP]
> `--no-build-isolation` reuses the dependencies already in your activated environment instead of building in a fresh isolated one — this is what makes incremental Cython rebuilds fast. The project also ships a `pixi.toml` with ready-made environments (`py311`–`py314`, `minimum-versions`, `pyarrow-nightly`, `numpy-nightly`, `py313-freethreading`) and a `build-pandas` task.

### 4.3 Code generation

Some Cython sources are generated from `.in` templates via Cython **Tempita** before compilation (`generate_pxi.py`), and version metadata is written by `generate_version.py`. Meson wires these into the build graph automatically; you do not run them by hand.

---

## 5. Configuration & Runtime Behavior

pandas has two configuration surfaces: the **options registry** (set at runtime) and a small set of **environment variables** (read at import time, mostly for opting into future behavior).

### 5.1 The options registry

Options are namespaced dotted keys registered in `pandas/core/config_init.py` and manipulated through the `pandas._config` API:

```python
pd.set_option("display.max_rows", 100)
pd.get_option("display.precision")
pd.reset_option("display.max_rows")
pd.describe_option("display")          # list a whole namespace
with pd.option_context("mode.chained_assignment", "raise"):
    ...                                 # scoped, auto-reverts
```

Key namespaces and representative options:

| Namespace | Representative keys | Controls |
|---|---|---|
| `display.*` | `max_rows`, `max_columns`, `precision`, `max_seq_items`, `max_info_columns`, `chop_threshold`, `expand_frame_repr`, `notebook_repr_html`, `float_format`, `max_colwidth` | How objects render |
| `mode.*` | `chained_assignment`, `copy_on_write`, `sim_interactive` | Core behaviors / safety |
| `compute.*` | `use_bottleneck`, `use_numexpr`, `use_numba` | Acceleration backends |
| `io.*` | `io.excel.*.reader`/`.writer`, `io.hdf.*` | I/O backend selection |
| `plotting.*` | `plotting.backend` | Plot backend (default `matplotlib`) |
| `styler.*` | `render.encoding`, `format.decimal`, `html.mathjax`, `latex.hrules` | `Styler` output |
| `future.*` | `future.infer_string` | Opt into next-major behavior early |

### 5.2 Environment variables (read at import)

| Variable | Default | Effect |
|---|---|---|
| `PANDAS_COPY_ON_WRITE` | `0` | Enable Copy-on-Write semantics globally |
| `PANDAS_FUTURE_INFER_STRING` | `1` | Infer the dedicated `string` dtype for text instead of `object` |
| `PANDAS_FUTURE_DISTINGUISH_NAN_AND_NA` | `0` | Treat `NaN` and `NA` as distinct |
| `PANDAS_FUTURE_PYTHON_SCALARS` | `0` | Return Python scalars (not NumPy scalars) from indexing/reductions |

> [!IMPORTANT]
> The `PANDAS_FUTURE_*` and `mode.copy_on_write` switches preview behavior slated to become the default in a future major release. Test downstream code against them *before* the upgrade lands — they change types and copy semantics in ways that can surface latent bugs. The `future.infer_string` path is already on by default.

### 5.3 Copy-on-Write (CoW)

Copy-on-Write makes any value returned from an indexing operation behave as if it were a copy: mutating it never silently changes the parent object. It eliminates the long-standing `SettingWithCopyWarning` ambiguity and lets pandas defer/avoid copies internally for better performance. It is the default direction of travel; enable it explicitly with `pd.set_option("mode.copy_on_write", True)` or `PANDAS_COPY_ON_WRITE=1` to validate your code ahead of time.

---

## 6. Security Considerations

pandas is a data-processing library; its risk surface is **the data and files it ingests**, not network services. The following deserve explicit attention when pandas runs on untrusted input.

> [!CAUTION]
> **Never `read_pickle` data you do not control.** Pickle deserialization can execute arbitrary code by design. The same caution applies to any pickle-backed path, including legacy pickle-based caches. Treat `.pkl` files from external sources as executable, not as data.

| Vector | Risk | Mitigation |
|---|---|---|
| `read_pickle` / `to_pickle` | Arbitrary code execution on load | Only unpickle trusted files; prefer Parquet/Feather for interchange |
| `read_excel` / `read_xml` / `read_html` | XML/spreadsheet entity-expansion ("billion laughs"), external-entity (XXE) attacks via the underlying parser | Keep `lxml`/`openpyxl` current; avoid parsing untrusted XML with external entities enabled |
| `read_sql` / `to_sql` | SQL injection if queries are built by string formatting | Use parameterized queries (`params=`), never f-strings/`%` with user input |
| `read_csv` with `engine="python"` | The python engine evaluates regex separators | Prefer the `c` or `pyarrow` engine for untrusted input |
| `eval` / `DataFrame.query` | Expression evaluation; the (non-default) `engine`/`local_dict` paths can reach Python objects | Do not pass untrusted strings to `eval`/`query`; default `numexpr` engine limits the surface |
| Remote paths (`s3://`, `http://`) via `fsspec` | SSRF / credential exposure through `storage_options` | Validate URLs; scope cloud credentials narrowly |

> [!WARNING]
> `pd.read_csv`, `read_json`, and friends accept URLs and compressed archives. Reading an attacker-controlled URL can trigger outbound requests (SSRF) and decompression bombs. Size-limit and validate any path that originates from user input.

---

## 7. Operations: Test, Benchmark, Release

Because pandas ships no runtime service, day-to-day "operations" are the contributor and maintainer workflow.

### 7.1 Running the test suite

```bash
# Full suite, parallel across cores
python -m pytest pandas --numprocesses=auto

# A single module / test
python -m pytest pandas/tests/frame/test_constructors.py -q
```

Registered pytest markers (declared in `pandas/conftest.py`) let you scope runs:

| Marker | Meaning |
|---|---|
| `slow` | Slow test (deselect with `-m "not slow"`) |
| `network` | Requires network access |
| `db` | Requires a MySQL/Postgres database |
| `single_cpu` | Must run on a single CPU |
| `clipboard` | Exercises clipboard I/O |
| `arm_slow` | Slow specifically on arm64 |

The suite uses **Hypothesis** for property-based testing (a CI profile caps examples for speed) and ships rich comparison helpers in `pandas._testing` (`assert_frame_equal`, `assert_series_equal`).

### 7.2 Benchmarks

Performance is guarded by an **ASV** (Airspeed Velocity) benchmark suite under `asv_bench/`. Run it to catch regressions before they merge:

```bash
cd asv_bench
asv run
asv continuous main HEAD     # compare a branch against main
```

### 7.3 Code quality gates

A comprehensive `pre-commit` configuration enforces standards on every commit. Highlights:

| Hook | Enforces |
|---|---|
| `ruff` + `ruff-format` | Linting and formatting |
| `cython-lint` | Cython style |
| `clang-format` | C/C++ formatting in `_libs/src` |
| `mypy` + `pyright` (manual stage) | Static typing |
| `codespell` | Spelling |
| `meson-fmt`, `shellcheck`, `isort`, `pyupgrade` | Build files, shell, imports, syntax modernization |
| Custom `scripts/*` validators | Namespace consistency, error placement, private-import policing, whatsnew sorting |

```bash
pre-commit install        # run automatically on git commit
pre-commit run --all-files
```

### 7.4 Documentation & website

The reference docs are built with Sphinx from `doc/source/` (`python doc/make.py html`); the marketing/community site is generated by `web/pandas_web.py` from `web/pandas/`. Both have dedicated `pixi` tasks (`ci-build-documentation`, `ci-build-website`).

---

## 8. Extending pandas

pandas is designed to be built upon. The supported extension points:

| You want to add… | Use |
|---|---|
| A custom column type | Subclass `ExtensionArray` + `ExtensionDtype`, register with `pandas.api.extensions.register_extension_dtype` |
| A `.myaccessor` namespace on Series/DataFrame/Index | `@pd.api.extensions.register_dataframe_accessor("myns")` (and the series/index variants) |
| A custom rolling-window indexer | Subclass from `pandas.api.indexers` |
| An alternate plotting backend | Register a `pandas_plotting_backends` entry point; select via `pd.set_option("plotting.backend", "...")` |
| Type-checking against pandas internals | Import from `pandas.api.typing` |
| Zero-copy interchange with other libraries | The DataFrame Interchange Protocol (`pandas.api.interchange`) and the Arrow PyCapsule interface (`__arrow_c_array__` / `__arrow_c_stream__`) |

> [!TIP]
> Register accessors instead of monkey-patching `DataFrame`. A registered accessor (`df.myns.method()`) is the project-blessed way to attach domain-specific behavior without colliding with current or future pandas methods.

---

## 9. Known Issues & Improvement Areas

Drawn from the project's own open issue tracker and deprecation plans at this commit.

| Area | Item | Reference |
|---|---|---|
| Nullable dtypes | `DataFrame.add(fill_value=...)` promotes nullable integer dtypes to `Float64`, inconsistent with `Series` | issue #65805 |
| I/O — JSON | `DataFrame.to_dict(orient="index", into=...)` does not apply `into` to nested mappings | issue #65778 |
| I/O — Arrow | `pd.array` does not preserve missing values when converting NumPy masked arrays | issue #65804 |
| I/O — HDF5 | Historically wrote a hardcoded pandas version into stored files instead of the actual version | issue #62792 |
| API deprecation | **PDEP-8**: `inplace` keyword deprecation, tracked for removal | issue #63207 |
| API removal | A consolidated list of deprecations slated for removal in pandas **4.0** | issue #56596 |
| CI / hardening | Proposal to set up AddressSanitizer (ASAN) in CI for the C/Cython core | issue #52990 |

> [!NOTE]
> pandas manages breaking change through formal **PDEPs** (pandas Enhancement Proposals) and a documented deprecation cycle: a behavior is first warned about, then changed in the next major release. Track the `whatsnew` notes (`doc/source/whatsnew/`) and the "deprecations to be removed in 4.0" issue when planning an upgrade.

### Upgrade guidance for downstream maintainers

- **Test against the future flags now.** Enable `mode.copy_on_write` and the `PANDAS_FUTURE_*` variables in CI to surface code that relies on soon-to-change copy semantics, integer-promotion behavior, and scalar return types.
- **Stop relying on `inplace=`.** With PDEP-8 deprecating it, prefer reassignment (`df = df.method(...)`), which also composes cleanly with method chaining.
- **Pin to the public API.** Import only from documented namespaces (`pandas`, `pandas.api.*`). Reaching into `pandas.core.*` internals (e.g. the `BlockManager`) is unsupported and breaks between releases.

---

## 10. Quick Reference

**Build from source (editable):**
```bash
python -m pip install -ve . --no-build-isolation -Ceditable-verbose=true
```

**Run tests / skip slow:**
```bash
python -m pytest pandas --numprocesses=auto -m "not slow"
```

**Benchmark a branch:**
```bash
cd asv_bench && asv continuous main HEAD
```

**Lint everything:**
```bash
pre-commit run --all-files
```

**Inspect environment for a bug report:**
```python
pd.show_versions()
```

**Configuration cheat sheet:**

| Want | Do |
|---|---|
| Change display | `pd.set_option("display.max_rows", N)` |
| Scope a setting | `with pd.option_context(key, val): ...` |
| Enable Copy-on-Write | `pd.set_option("mode.copy_on_write", True)` |
| Preview future behavior | `PANDAS_FUTURE_*=1` env vars |
| Switch plot backend | `pd.set_option("plotting.backend", "...")` |

**Public extension namespaces:** `pandas.api.types`, `pandas.api.extensions`, `pandas.api.indexers`, `pandas.api.typing`, `pandas.api.interchange`.

---

_Generated June 2026 from pandas-dev/pandas at b5c64e2. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._