# CPython — Administrator & Developer Guide

CPython is the reference implementation of the Python programming language — the canonical interpreter that defines what "Python" means. This guide covers everything an administrator, packager, platform integrator, or core developer needs: building from source, configuring the interpreter and build system, deploying across six supported platforms, operating at scale, performance tuning, security hardening, and deep internals of the compiler pipeline, object model, memory allocator, and JIT. See User Guide for everyday usage.

---

## 1. Architecture Overview

### 1.1 High-Level Architecture

CPython is a bytecode-compiling interpreter with an optional JIT backend. Source code flows through a well-defined pipeline:

```text
Source Text
    │
    ▼
Tokenizer (Parser/tokenizer/, Parser/lexer/)
    │
    ▼
PEG Parser (Parser/) → Concrete Syntax Tree
    │
    ▼
AST Compiler (Python/ast.c, Python/ast_opt.c)
    │
    ▼
Bytecode Compiler (Python/compile.c, Python/codegen.c, Python/flowgraph.c)
    │
    ▼
Code Objects (Objects/codeobject.c)
    │
    ▼
Evaluation Loop (Python/ceval.c, Python/generated_cases.c.h)
    │                          │
    ▼                          ▼
Standard Interpreter      JIT Compiler (Tools/jit/ → native code)
    │
    ▼
Object Model (Objects/*.c) ←→ Memory Allocator (Objects/obmalloc.c, Objects/mimalloc/)
    │
    ▼
C Extension Modules (Modules/*.c) ←→ Standard Library (Lib/*.py)
```

The interpreter core is written in C (approximately 14M characters of C source across `Python/`, `Objects/`, `Modules/`, `Parser/`, `Include/`, and `Programs/`). The standard library adds another 10M+ characters of Python code in `Lib/`.

> [!NOTE]
> CPython's PEG parser (introduced in Python 3.9, sole parser since 3.10) replaced the legacy LL(1) parser. The grammar definition lives in `Grammar/python.gram` and is processed by `Tools/peg_generator/` to produce `Parser/parser.c`. The tokenizer was refactored into separate `Parser/tokenizer/` and `Parser/lexer/` subdirectories.

### 1.2 Core Subsystems

| Subsystem | Directory | Files | Size | Purpose |
|-----------|-----------|-------|------|---------|
| Interpreter Core | `Python/` | 113 | 4.4M chars | Compiler, eval loop, import system, GC, JIT runtime |
| Object Model | `Objects/` | 51 | 3.8M chars | All built-in types (int, str, list, dict, set, etc.) |
| C Extensions | `Modules/` | 115 | 6.4M chars | _io, _ssl, _sqlite3, _ctypes, _zstd, _decimal, _sre, expat, cjkcodecs |
| Argument Clinic | `Modules/clinic/` | 79 | 2.2M chars | Auto-generated argument parsing code |
| Parser | `Parser/` | 24 | 367K chars | PEG parser, tokenizer, lexer |
| Entry Points | `Programs/` | 8 | 90K chars | `main()` for python, _testembed, etc. |
| Public C API | `Include/` | 77 | 361K chars | Stable ABI headers |
| Internal C API | `Include/internal/` | 144 | 1.6M chars | CPython-internal headers |
| CPython-specific API | `Include/cpython/` | 62 | 304K chars | CPython extension headers |
| Standard Library | `Lib/` | 500+ | 10M+ chars | Pure Python modules |
| Memory Allocator | `Objects/mimalloc/` | 18 | 393K chars | Microsoft mimalloc integration |
| String Library | `Objects/stringlib/` | 20 | 158K chars | Shared string algorithms |
| Development Tools | `Tools/` | 100+ | 1M+ chars | Clinic, cases generator, JIT build, PEG generator |

### 1.3 Platform Support Matrix

CPython officially supports six platform families:

| Platform | Directory | Build System | Status |
|----------|-----------|-------------|--------|
| Linux / POSIX | Root (`configure.ac`, `Makefile.pre.in`) | autoconf + make | Primary tier |
| macOS | `Mac/` | autoconf + make + framework builds | Primary tier |
| Windows | `PC/`, `PCbuild/` | Visual Studio (MSBuild) | Primary tier |
| Android | `Platforms/Android/` | Cross-compilation + Gradle | Supported |
| iOS / Apple | `Platforms/Apple/` | Cross-compilation + Xcode | Supported |
| Emscripten (WASM) | `Platforms/emscripten/` | Emscripten SDK + make | Experimental |
| WASI (WASM) | `Platforms/WASI/` | WASI SDK + make | Experimental |

### 1.4 Build System Architecture

CPython uses two distinct build systems:

**POSIX (Linux, macOS, Android, iOS, WASM):**
```text
configure.ac → autoconf → configure → Makefile.pre.in → Makefile → make
                                         │
                                    pyconfig.h (generated)
```

**Windows:**
```text
PCbuild/*.vcxproj → MSBuild → python.exe, pythonXY.dll
    │
    ├── get_external.py (downloads dependencies: OpenSSL, libffi, sqlite, etc.)
    └── PCbuild/build.bat (wrapper)
```

### 1.5 Memory Management Architecture

CPython uses a layered memory architecture:

```text
Layer 3:  Object-specific allocators (e.g., PyUnicode, PyLong small-int cache)
Layer 2:  Python object allocator (obmalloc.c — arena-based, or mimalloc)
Layer 1:  Python raw allocator (wraps malloc/free with hooks)
Layer 0:  System allocator (libc malloc, or platform equivalent)
```

The mimalloc allocator (`Objects/mimalloc/`) is an integrated copy of Microsoft's mimalloc, providing significant performance improvements for multi-threaded workloads, especially under the free-threading (no-GIL) build.

### 1.6 Free-Threading Architecture

CPython now includes experimental free-threading support (building without the GIL):

```text
Traditional CPython:          Free-Threaded CPython:
┌──────────────┐              ┌──────────────┐
│  Thread 1    │              │  Thread 1    │
│  (holds GIL) │              │  (no GIL)    │
├──────────────┤              ├──────────────┤
│  Thread 2    │              │  Thread 2    │
│  (waiting)   │              │  (running)   │
├──────────────┤              ├──────────────┤
│  Thread 3    │              │  Thread 3    │
│  (waiting)   │              │  (running)   │
└──────────────┘              └──────────────┘
         │                             │
    Sequential                   True parallel
    execution                    execution
```

Key components:
- `PYTHON_GIL=0` environment variable to disable GIL at runtime
- `--disable-gil` configure flag for free-threaded builds
- QSBR (Quiescent State-Based Reclamation) for safe memory reclamation (`Include/internal/pycore_qsbr.h`)
- Biased reference counting for reduced contention
- Per-object locks for mutable containers
- mimalloc as the default allocator in free-threaded builds

> [!NOTE]
> Free-threading is experimental. The `PYTHON_GIL` environment variable controls GIL behavior at runtime: `PYTHON_GIL=1` forces the GIL on (default for standard builds), `PYTHON_GIL=0` disables it (only effective in free-threaded builds). See issue #113956 regarding `_Py_SetImmortal` on allocating threads and issue #148953 regarding potential deadlocks in `_Py_qsbr_reserve`.

### 1.7 JIT Compiler Architecture

The experimental JIT compiler uses a copy-and-patch approach:

```text
Tools/jit/build.py
    │
    ▼
LLVM/Clang compiles C stubs → object files
    │
    ▼
Extracts machine code templates (one per micro-op)
    │
    ▼
Generates jit_stencils.h (embedded in CPython)
    │
    ▼
At runtime: Python/jit.c patches templates together
    │
    ▼
Executable native code for hot loops
```

The JIT operates at the micro-op (uop) level, below the bytecode tier. The `Tools/cases_generator/` produces the tier-1 (bytecode) and tier-2 (uop) interpreter implementations from a single definition in `Python/bytecodes.c`.

---

## 2. System Requirements

### 2.1 Build Requirements by Platform

#### 2.1.1 Linux / POSIX

| Requirement | Minimum | Recommended |
|-------------|---------|-------------|
| C compiler | GCC 8+ or Clang 9+ | GCC 13+ or Clang 18+ |
| C++ compiler | g++ (for JIT, optional) | g++ 13+ |
| Make | GNU Make 3.81+ | GNU Make 4.3+ |
| autoconf | 2.71+ (only if modifying configure.ac) | 2.72 |
| pkg-config | Any | Latest |
| LLVM | 18+ (JIT only) | 19+ |

Required system libraries for full feature builds:

```bash
# Debian/Ubuntu
sudo apt-get install build-essential gdb lcov pkg-config \
    libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
    libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
    lzma lzma-dev tk-dev uuid-dev zlib1g-dev libzstd-dev

# Fedora/RHEL
sudo dnf install gcc make pkgconfig \
    bzip2-devel libffi-devel gdbm-devel ncurses-devel openssl-devel \
    readline-devel sqlite-devel tk-devel uuid-devel xz-devel zlib-devel \
    libzstd-devel
```

Optional libraries that enable specific modules:

| Library | Module Enabled | Package (Debian) |
|---------|---------------|------------------|
| libssl | `_ssl`, `_hashlib` | `libssl-dev` |
| libffi | `_ctypes` | `libffi-dev` |
| libsqlite3 | `_sqlite3` | `libsqlite3-dev` |
| libreadline | `readline` | `libreadline-dev` |
| libbz2 | `_bz2` | `libbz2-dev` |
| liblzma | `_lzma` | `liblzma-dev` |
| libzstd | `_zstd` | `libzstd-dev` |
| zlib | `zlib` | `zlib1g-dev` |
| libgdbm | `_gdbm` | `libgdbm-dev` |
| tk | `_tkinter` | `tk-dev` |
| libuuid | `_uuid` | `uuid-dev` |
| libncurses | `_curses` | `libncurses-dev` |
| libmpdec | `_decimal` (external) | `libmpdec-dev` |
| libexpat | `pyexpat` (bundled fallback) | `libexpat1-dev` |

#### 2.1.2 macOS

| Requirement | Detail |
|-------------|--------|
| Xcode | Command Line Tools (minimum), full Xcode for framework builds |
| Homebrew | Recommended for dependencies |
| macOS | 10.13+ (High Sierra) for x86_64, 11.0+ for arm64 |

```bash
# Install dependencies via Homebrew
brew install openssl@3 xz gdbm tcl-tk libffi zstd
```

#### 2.1.3 Windows

| Requirement | Detail |
|-------------|--------|
| Visual Studio | 2019 or later (Community edition sufficient) |
| Workload | "Desktop development with C++" |
| Individual components | Windows SDK (latest), C++ CMake tools |
| Python | 3.12+ (for build scripts themselves) |

External dependencies are downloaded automatically by `PCbuild/get_external.py`:

```bash
# From PCbuild/ directory
python get_external.py --all
```

#### 2.1.4 Android

| Requirement | Detail |
|-------------|--------|
| Android NDK | r26+ |
| Android SDK | API level 21+ (minimum target) |
| Host Python | Matching version for cross-compilation |
| Gradle | For running test suite |

#### 2.1.5 iOS / Apple

| Requirement | Detail |
|-------------|--------|
| Xcode | 15+ with iOS SDK |
| Host Python | Matching version for cross-compilation |
| Target | iOS 13+ or equivalent tvOS/watchOS |

#### 2.1.6 Emscripten (WebAssembly)

| Requirement | Detail |
|-------------|--------|
| Emscripten SDK | 3.1.x+ |
| Node.js | For testing (`PYTHON_NODE_VERSION` env var) |
| Host Python | Matching version for cross-compilation |

#### 2.1.7 WASI (WebAssembly)

| Requirement | Detail |
|-------------|--------|
| WASI SDK | Latest (`WASI_SDK_PATH` env var) |
| wasmtime or wasmer | WASI runtime for execution |
| Host Python | Matching version |

### 2.2 Documentation Build Requirements

```bash
# In Doc/ directory
make venv    # Creates virtual environment with dependencies
make html    # Builds HTML documentation
```

Dependencies (from `Doc/requirements.txt`):
- `sphinx < 9.0.0`
- `blurb`
- `sphinxext-opengraph`
- `sphinx-notfound-page`
- `python-docs-theme`

### 2.3 Hardware Recommendations

| Scenario | CPU | RAM | Disk |
|----------|-----|-----|------|
| Development build (debug) | 4+ cores | 4 GB | 3 GB |
| Release build (optimized, PGO+LTO) | 8+ cores | 8 GB | 5 GB |
| Full test suite | 4+ cores | 4 GB | 2 GB additional |
| JIT-enabled build | 8+ cores | 8 GB | 5 GB (LLVM required) |
| Documentation build | 2+ cores | 2 GB | 1 GB |

> [!IMPORTANT]
> Profile-Guided Optimization (PGO) and Link-Time Optimization (LTO) builds require significantly more time and memory than standard builds. A PGO+LTO build on a 4-core machine can take 30+ minutes. The `--enable-optimizations` flag enables both PGO and LTO.

---

## 3. Configuration Reference

### 3.1 Build Configuration (`configure` Flags)

The `configure` script (`configure.ac` → autoconf → `configure`) accepts hundreds of options. The most significant:

#### 3.1.1 Core Build Options

| Flag | Default | Description |
|------|---------|-------------|
| `--prefix=DIR` | `/usr/local` | Installation prefix |
| `--enable-shared` | disabled | Build `libpython` as shared library |
| `--enable-static` | enabled | Build static `libpython` |
| `--enable-framework[=DIR]` | disabled | macOS framework build |
| `--enable-universalsdk[=SDK]` | disabled | macOS universal binary |
| `--with-universal-archs=ARCHS` | — | Architectures for universal build |

#### 3.1.2 Optimization Options

| Flag | Default | Description |
|------|---------|-------------|
| `--enable-optimizations` | disabled | Enable PGO + LTO |
| `--enable-lto` | disabled | Link-Time Optimization (thin or full) |
| `--with-lto=TYPE` | `full` | LTO type: `full`, `thin`, `no` |
| `--enable-bolt` | disabled | BOLT post-link optimization |
| `--enable-pystats` | disabled | Internal statistics collection |

#### 3.1.3 Debug Options

| Flag | Default | Description |
|------|---------|-------------|
| `--with-pydebug` | disabled | Debug build (`Py_DEBUG`), assertions, reference leak checking |
| `--with-trace-refs` | disabled | Trace all reference counting operations |
| `--with-assertions` | disabled (release) | Enable C assertions without full debug build |
| `--with-address-sanitizer` | disabled | ASAN instrumentation |
| `--with-memory-sanitizer` | disabled | MSAN instrumentation |
| `--with-undefined-behavior-sanitizer` | disabled | UBSAN instrumentation |
| `--with-thread-sanitizer` | disabled | TSAN instrumentation |
| `--with-valgrind` | disabled | Valgrind-friendly build |

#### 3.1.4 Feature Options

| Flag | Default | Description |
|------|---------|-------------|
| `--disable-gil` | enabled (GIL on) | Build free-threaded CPython |
| `--enable-experimental-jit` | disabled | Enable the copy-and-patch JIT compiler |
| `--with-computed-gotos` | auto | Use computed gotos in eval loop (GCC/Clang) |
| `--with-pymalloc` | enabled | Use Python's memory allocator |
| `--with-mimalloc` | auto | Use mimalloc allocator |
| `--with-freelists` | enabled | Enable object freelists |
| `--with-dtrace` | disabled | DTrace/SystemTap probes |

#### 3.1.5 Dependency Override Options

| Flag | Default | Description |
|------|---------|-------------|
| `--with-openssl=DIR` | auto | OpenSSL installation directory |
| `--with-openssl-rpath=DIR` | auto | OpenSSL runtime library path |
| `--with-ssl-default-suites=STRING` | auto | Default SSL cipher suites |
| `--with-builtin-hashlib-hashes=NAMES` | auto | Hash algorithms to build in |
| `--with-system-expat` | bundled | Use system expat instead of bundled |
| `--with-system-libmpdec` | bundled | Use system libmpdec instead of bundled |
| `--with-system-libffi` | auto | Use system libffi |
| `--with-tcltk-includes=FLAGS` | auto | Tcl/Tk include flags |
| `--with-tcltk-libs=FLAGS` | auto | Tcl/Tk linker flags |
| `--with-dbmliborder=LIBS` | auto | Order of dbm backends to try |

> [!IMPORTANT]
> When cross-compiling, you must supply `--host=TRIPLE` and `--build=TRIPLE` along with a host Python (`--with-build-python=PATH`) that matches the target version exactly. Cross-compilation is required for Android, iOS, Emscripten, and WASI targets.

#### 3.1.6 Installation Layout Options

| Flag | Default | Description |
|------|---------|-------------|
| `--with-suffix=SUFFIX` | platform-dependent | Executable suffix |
| `--with-platlibdir=DIR` | `lib` | Platform library directory name |
| `--with-pkg-config=MODE` | auto | pkg-config usage: `yes`, `no`, `check` |
| `--with-ensurepip=MODE` | `upgrade` | ensurepip action: `install`, `upgrade`, `no` |
| `--with-readline=TYPE` | auto | Readline library: `readline`, `editline`, `no` |

### 3.2 Compile-Time Definitions (`pyconfig.h`)

The `configure` script generates `pyconfig.h` from `pyconfig.h.in`. Key defines:

| Define | Effect |
|--------|--------|
| `Py_DEBUG` | Debug build: assertions, extra checks, `-d` suffix |
| `Py_TRACE_REFS` | Track all object references (massive overhead) |
| `Py_GIL_DISABLED` | Free-threaded build |
| `Py_STATS` | Collect internal statistics |
| `HAVE_COMPUTED_GOTOS` | Use computed-goto dispatch in eval loop |
| `WITH_PYMALLOC` | Enable pymalloc arena allocator |
| `PY_HAVE_THREAD_NATIVE_ID` | Platform supports native thread IDs |
| `HAVE_FORK` | Platform has fork() |
| `HAVE_DYNAMIC_LOADING` | Platform supports dlopen()/LoadLibrary() |
| `MS_WINDOWS` | Windows platform |

### 3.3 Runtime Environment Variables

#### 3.3.1 Interpreter Behavior

| Variable | Values | Description |
|----------|--------|-------------|
| `PYTHONSTARTUP` | File path | Python script executed on interactive startup |
| `PYTHONPATH` | Path list | Additional module search paths |
| `PYTHONHOME` | Path | Override standard library location |
| `PYTHONUSERBASE` | Path | Base directory for user site-packages |
| `PYTHONDONTWRITEBYTECODE` | Any | Don't write `.pyc` files |
| `PYTHONHASHSEED` | `random` or integer | Hash randomization seed |
| `PYTHONIOENCODING` | `encoding[:errors]` | Stdin/stdout/stderr encoding |
| `PYTHONUTF8` | `0` or `1` | UTF-8 mode |
| `PYTHONWARNINGS` | Warning filters | Equivalent to `-W` flags |
| `PYTHONMALLOC` | Allocator name | Override memory allocator |
| `PYTHONTRACEMALLOC` | `N` (frames) | Enable tracemalloc with N frames |
| `PYTHONPROFILEIMPORTTIME` | `0` or `1` | Profile import times |
| `PYTHONNODEBUGRANGES` | Any | Disable code location ranges |
| `PYTHONINTMAXSTRDIGITS` | Integer | Max digits for int↔str conversion |
| `PYTHON_BASIC_REPL` | Any | Use basic REPL instead of `_pyrepl` |
| `PYTHON_GIL` | `0` or `1` | Control GIL in free-threaded builds |
| `PYTHON_BASIC_COMPLETER` | Any | Use basic tab completer |
| `PYTHONDOCS` | URL | Documentation URL override |
| `PYTHONTZPATH` | Path list | Time zone search paths |
| `PYTHONASYNCIODEBUG` | Any | Enable asyncio debug mode |
| `SOURCE_DATE_EPOCH` | Unix timestamp | Reproducible builds timestamp |
| `POSIXLY_CORRECT` | Any | POSIX-strict behavior in certain modules |

#### 3.3.2 SSL/Security

| Variable | Description |
|----------|-------------|
| `SSLKEYLOGFILE` | Path to write TLS key log (for Wireshark debugging) |

> [!WARNING]
> Setting `SSLKEYLOGFILE` in production enables extraction of TLS session keys, allowing decryption of all TLS traffic. This should only be used in development/debugging environments. Ensure the log file has restrictive permissions if used at all.

#### 3.3.3 Build and Test Environment

| Variable | Description |
|----------|-------------|
| `PYTHON_NODE_VERSION` | Node.js version for Emscripten browser tests |
| `CROSS_BUILD_DIR` | Cross-compilation build directory |
| `EMSDK_CACHE` | Emscripten SDK cache directory |
| `WASI_SDK_PATH` | WASI SDK installation path |
| `EXTERNALS_DIR` | Windows external dependencies directory |
| `CACHE_DIR` | Build cache directory |
| `CPYTHON_TEST_CPP_STD` | C++ standard for testing (e.g., `c++20`) |
| `CPYTHON_TEST_EXTRA_CFLAGS` | Extra compiler flags for test builds |
| `CPYTHON_TEST_LIMITED` | Test limited C API only |
| `CPYTHON_TEST_STD` | C standard for testing |
| `CPYTHON_TEST_ABI3T` | Test stable ABI (abi3t) |
| `TEST_INTERNAL_C_API` | Enable internal C API tests |
| `_PYTHON_HOSTRUNNER` | Host runner for cross-compiled test execution |
| `_PYTHON_SUBPROCESS_USE_POSIX_SPAWN` | Force posix_spawn in subprocess |
| `_PYTHON_HOST_PLATFORM` | Override platform for cross-compilation |
| `_PYTHON_SYSCONFIGDATA_NAME` | Override sysconfig data module name |
| `_PYTHON_SYSCONFIGDATA_PATH` | Override sysconfig data path |
| `_PYTHON_PROJECT_BASE` | Override project base directory |

#### 3.3.4 REPL and IDE

| Variable | Description |
|----------|-------------|
| `IDLESTARTUP` | IDLE startup script |
| `PYREPL_TRACE` | Enable _pyrepl debug tracing |
| `MANPAGER` | Manual page viewer |
| `PAGER` | Default pager for help() |
| `LINES` | Terminal lines (affects pager) |
| `TERM` | Terminal type |
| `TERM_PROGRAM` | Terminal program identifier |
| `TERMINFO` | Terminfo database path |
| `TERMINFO_DIRS` | Terminfo search directories |

#### 3.3.5 Platform-Specific

| Variable | Platform | Description |
|----------|----------|-------------|
| `TCL_LIBRARY` | All | Tcl library path for tkinter |
| `DISPLAY` | Linux/X11 | X11 display server |
| `WAYLAND_DISPLAY` | Linux/Wayland | Wayland display |
| `XDG_CURRENT_DESKTOP` | Linux | Desktop environment identifier |
| `HOME` | POSIX | User home directory |
| `ComSpec` | Windows | Command interpreter path |
| `SystemRoot` | Windows | Windows system root |
| `APPDATA` | Windows | Application data directory |
| `PROGRAMFILES` | Windows | Program Files directory |
| `PROGRAMFILES(x86)` | Windows | Program Files (x86) directory |
| `USERNAME` | Windows | Current username |
| `HOMEDRIVE` | Windows | Home drive letter |
| `PROCESSOR_ARCHITECTURE` | Windows | CPU architecture |
| `PROCESSOR_ARCHITEW6432` | Windows | Native CPU architecture (WoW64) |
| `PROCESSOR_IDENTIFIER` | Windows | CPU identifier string |
| `PreferredToolArchitecture` | Windows | VS preferred tool architecture |
| `BUILDINSTALLER_BUILDPYTHON_MAKE_EXTRAS` | macOS | Extra make flags for macOS installer |

### 3.4 Configuration Files

| File | Purpose |
|------|---------|
| `.coveragerc` | Coverage.py configuration for test coverage |
| `.editorconfig` | Editor settings (indent style, line endings) |
| `.gitattributes` | Git line-ending and diff configuration |
| `.pre-commit-config.yaml` | Pre-commit hooks configuration |
| `.readthedocs.yml` | Read the Docs build configuration |
| `.ruff.toml` | Ruff linter configuration |
| `configure.ac` | Autoconf source (generates `configure`) |
| `Makefile.pre.in` | Make template (generates `Makefile`) |
| `pyconfig.h.in` | Config header template (generates `pyconfig.h`) |

### 3.5 `Makefile.pre.in` Key Targets

| Target | Description |
|--------|-------------|
| `all` | Build Python interpreter + modules |
| `build_all` | Build everything including shared modules |
| `install` | Install Python to `--prefix` |
| `altinstall` | Install without creating `python3` symlink |
| `test` | Run test suite |
| `testall` | Run tests with multiple optimization levels |
| `clinic` | Run Argument Clinic on all files |
| `regen-all` | Regenerate all generated files |
| `regen-opcode` | Regenerate opcode tables |
| `regen-token` | Regenerate token definitions |
| `regen-ast` | Regenerate AST code from ASDL |
| `regen-pegen` | Regenerate PEG parser |
| `regen-cases` | Regenerate bytecode/uop cases |
| `regen-jit` | Regenerate JIT stencils |
| `clean` | Remove object files |
| `clobber` | Remove everything regeneratable |
| `distclean` | Remove everything including `configure` output |
| `profile-opt` | Build with PGO |
| `bolt-opt` | Apply BOLT optimization |
| `tags` / `TAGS` | Generate ctags/etags |
| `smelly` | Check for unintended exported symbols |
| `sharedmods` | Build shared extension modules |

> [!TIP]
> For development, use `make -j$(nproc)` for parallel builds. The `regen-all` target is needed after modifying grammar files, ASDL definitions, opcode definitions, or Argument Clinic inputs. Running `make clinic` alone regenerates only Argument Clinic output.

---

## 4. Deployment & Scaling

### 4.1 Building from Source

#### 4.1.1 Standard POSIX Build

```bash
# Clone
git clone https://github.com/python/cpython.git
cd cpython

# Configure (debug build for development)
./configure --with-pydebug

# Build
make -j$(nproc)

# Verify
./python -c "import sys; print(sys.version)"
```

Expected output:
```text
Python 3.15.0a0 (heads/main:49918f5, May 7 2026, 12:00:00) [GCC 13.2.0] on linux
```

#### 4.1.2 Optimized Release Build

```bash
./configure --enable-optimizations --with-lto --prefix=/opt/python3.15

# PGO training + LTO compilation (slow, 15-40 minutes)
make -j$(nproc)

# Install
sudo make install
```

> [!TIP]
> The `--enable-optimizations` flag runs the test suite as PGO training data, then rebuilds with profile feedback plus LTO. For faster iteration during development, skip this and use `--with-pydebug` instead.

#### 4.1.3 Free-Threaded Build

```bash
./configure --disable-gil --with-pydebug
make -j$(nproc)

# Verify GIL is disabled
./python -c "import sys; print(sys._is_gil_enabled())"
```

Expected output:
```text
False
```

#### 4.1.4 JIT-Enabled Build

```bash
# Requires LLVM 18+
./configure --enable-experimental-jit --with-pydebug
make -j$(nproc)

# Verify JIT is available
./python -c "import sys; print(sys._jit)"
```

#### 4.1.5 Windows Build

```bash
# From repository root in Developer Command Prompt
PCbuild\build.bat -e -p x64

# Or for debug build:
PCbuild\build.bat -e -d -p x64
```

The `-e` flag runs `get_external.py` to download dependencies (OpenSSL, Tcl/Tk, SQLite, etc.). Build output goes to `PCbuild/amd64/` (release) or `PCbuild/amd64_d/` (debug).

#### 4.1.6 macOS Framework Build

```bash
./configure --enable-framework=/Library/Frameworks \
            --with-openssl=$(brew --prefix openssl@3)
make -j$(nproc)
sudo make frameworkinstall
```

#### 4.1.7 Android Cross-Compilation

```bash
# Using the platform build tool
python Platforms/Android/__main__.py \
    --cross-build-dir=build/android \
    arm64-v8a  # or armeabi-v7a, x86, x86_64
```

Key flags:

| Flag | Description |
|------|-------------|
| `--connected` | Run tests on connected device via ADB |
| `--managed` | Run tests on managed emulator |
| `--site-packages DIR` | Additional site-packages to include |
| `--cwd DIR` | Working directory on device |
| `--cross-build-dir DIR` | Build output directory |
| `-v` | Verbose output |
| `--cache-dir DIR` | Cache directory for downloads |
| `--clean` | Clean before building |
| `--fast-ci` | Fast CI test subset |
| `--slow-ci` | Slow CI (full) test run |

#### 4.1.8 iOS / Apple Cross-Compilation

```bash
python Platforms/Apple/__main__.py \
    --cross-build-dir=build/ios \
    iOS arm64
```

Key flags:

| Flag | Description |
|------|-------------|
| `platform` | Target platform (iOS, tvOS, etc.) |
| `host` | Target architecture |
| `--simulator` | Build for simulator instead of device |
| `--cross-build-dir DIR` | Build output directory |
| `--clean` | Clean before building |
| `--cache-dir DIR` | Cache directory |
| `--fast-ci` / `--slow-ci` | CI test modes |

#### 4.1.9 Emscripten (WebAssembly) Build

```bash
# Ensure EMSDK is activated
python Platforms/emscripten/__main__.py \
    --cross-build-dir=build/emscripten \
    wasm32-emscripten
```

Key flags:

| Flag | Description |
|------|-------------|
| `--cross-build-dir DIR` | Build output directory |
| `target` | Build target |
| `--test` | Run test suite after build |
| `--pythoninfo` | Display build info |
| `--clean` | Clean before building |
| `--quiet` | Suppress output |
| `--emsdk-cache DIR` | Emscripten SDK cache directory |
| `--host-runner CMD` | Host runner for executing WASM |

#### 4.1.10 WASI Build

```bash
python Platforms/WASI/__main__.py \
    --wasi-sdk=/opt/wasi-sdk \
    --host-runner=wasmtime
```

Key flags:

| Flag | Description |
|------|-------------|
| `--quiet` | Suppress output |
| `--logdir DIR` | Log output directory |
| `--clean` | Clean before building |
| `--wasi-sdk DIR` | WASI SDK path |
| `--host-runner CMD` | WASI runtime (wasmtime, wasmer) |
| `--host-triple TRIPLE` | Host triple override |

### 4.2 Installation

#### 4.2.1 Standard Installation

```bash
# Install to --prefix (default /usr/local)
sudo make install

# Creates:
#   /usr/local/bin/python3.15
#   /usr/local/bin/python3
#   /usr/local/lib/python3.15/
#   /usr/local/lib/libpython3.15.a (or .so if --enable-shared)
#   /usr/local/include/python3.15/
```

#### 4.2.2 Alternative Installation (Side-by-Side)

```bash
# Does NOT create 'python3' symlink — safe alongside system Python
sudo make altinstall

# Creates only versioned binaries:
#   /usr/local/bin/python3.15
#   /usr/local/bin/pip3.15
```

> [!WARNING]
> Never run `make install` on a system where `python3` is the system Python (e.g., modern Ubuntu, Fedora). Use `make altinstall` or install to a separate `--prefix`. Overwriting the system Python breaks package managers and system tools.

#### 4.2.3 Windows Installation Layout

The `PC/layout/main.py` tool handles Windows packaging with extensive options:

```bash
python PC/layout/main.py --help
```

Key layout flags:

| Flag | Description |
|------|-------------|
| `--source DIR` | Source build directory |
| `--build DIR` | Build configuration (Release, Debug) |
| `--arch ARCH` | Target architecture |
| `--include-dev` | Include development headers and libs |
| `--include-tools` | Include Tools/ scripts |
| `--include-tcltk` | Include Tcl/Tk |
| `--include-tests` | Include test suite |
| `--include-pip` | Include pip |
| `--zip-lib` | Create python3XX.zip for stdlib |
| `--flat-dlls` | Place DLLs in root directory |
| `--preset-embed` | Embeddable distribution preset |
| `--preset-appx` | Microsoft Store preset |
| `--preset-nuget` | NuGet package preset |
| `--preset-iot` | IoT (minimal) preset |

### 4.3 Embedding CPython

CPython can be embedded in other applications via the C API:

```c
#include <Python.h>

int main(int argc, char *argv[]) {
    PyStatus status;
    PyConfig config;

    PyConfig_InitPythonConfig(&config);
    config.isolated = 1;
    config.site_import = 0;

    status = Py_InitializeFromConfig(&config);
    if (PyStatus_Exception(status)) {
        PyConfig_Clear(&config);
        Py_ExitStatusException(status);
    }

    PyRun_SimpleString("print('Hello from embedded Python!')");

    Py_Finalize();
    return 0;
}
```

Compile with:
```bash
gcc -o embed embed.c $(python3.15-config --cflags --ldflags --embed)
```

The `Programs/_testembed.c` file is the comprehensive test and example for all embedding scenarios.

### 4.4 Shared Library Deployment

When built with `--enable-shared`:

```bash
# Set library path
export LD_LIBRARY_PATH=/opt/python3.15/lib:$LD_LIBRARY_PATH

# Or configure ldconfig
echo "/opt/python3.15/lib" | sudo tee /etc/ld.so.conf.d/python3.15.conf
sudo ldconfig
```

### 4.5 Scaling Considerations

CPython's GIL (Global Interpreter Lock) historically limits CPU-bound parallelism to one thread. Strategies for scaling:

| Strategy | Mechanism | Best For |
|----------|-----------|----------|
| `multiprocessing` | Separate processes | CPU-bound parallelism |
| `concurrent.futures.ProcessPoolExecutor` | Process pool | CPU-bound batch work |
| `asyncio` | Event loop + coroutines | I/O-bound concurrency |
| Free-threading (`--disable-gil`) | True thread parallelism | CPU-bound (experimental) |
| Subinterpreters (`concurrent.interpreters`) | Isolated interpreters | CPU-bound isolation |
| C extensions releasing GIL | `Py_BEGIN_ALLOW_THREADS` | CPU-bound C work |

> [!NOTE]
> The `concurrent.interpreters` module (subinterpreters) provides a middle ground: each interpreter has its own GIL, enabling true parallelism without the complexity of multiprocessing. Combined with `Lib/concurrent/` futures, this provides a practical scaling path.

---

## 5. Operations

### 5.1 Building the Documentation

```bash
cd Doc/

# First time: create venv with sphinx + dependencies
make venv

# Build HTML docs
make html

# Output in Doc/build/html/
```

Other documentation targets:

| Target | Output |
|--------|--------|
| `make html` | HTML pages |
| `make latex` | LaTeX source |
| `make text` | Plain text |
| `make epub` | EPUB ebook |
| `make clean` | Remove build output |
| `make check` | Check for markup issues |
| `make serve` | Local HTTP server for preview |

### 5.2 Running the Test Suite

#### 5.2.1 Basic Test Run

```bash
# Run the full test suite
make test

# Or directly:
./python -m test

# Run specific test module
./python -m test test_list

# Run with verbose output
./python -m test -v test_dict
```

#### 5.2.2 Test Runner Flags (libregrtest)

The test runner (`Lib/test/libregrtest/cmdline.py`) has 50+ flags. Key ones:

| Flag | Description |
|------|-------------|
| `-v` / `--verbose` | Verbose output |
| `-w` / `--rerun` | Re-run failed tests |
| `-j N` / `--multiprocess N` | Run N tests in parallel |
| `-R N:M` | Hunt for reference leaks (N warmups, M repetitions) |
| `-u RESOURCE` | Enable resource-intensive tests (e.g., `network`, `largefile`) |
| `-x TEST` | Exclude a specific test |
| `--timeout N` | Per-test timeout in seconds |
| `--fail-env-changed` | Fail if test changes environment |
| `--coverage` | Run with code coverage |
| `-T` / `--coverage-dir DIR` | Coverage output directory |
| `--pgo` | Run PGO training subset |
| `--single` | Run one test at a time (for bisecting) |
| `--slow` | Print slowest tests at end |
| `--randomize` | Randomize test order |
| `--match PAT` | Run only tests matching pattern |
| `--ignore PAT` | Ignore tests matching pattern |
| `-F` / `--forever` | Run tests repeatedly until failure |

> [!TIP]
> For development, `./python -m test -j0 -v test_name` runs a single test module with all available cores in verbose mode. Use `-R 3:5` to hunt for reference leaks when modifying C code: it runs the test 3 times to warm up, then 5 times to detect leaks.

#### 5.2.3 Resource-Intensive Tests

Some tests are skipped by default. Enable them with `-u`:

| Resource | Tests Enabled |
|----------|--------------|
| `network` | Tests requiring internet |
| `largefile` | Tests creating large files |
| `audio` | Audio device tests |
| `gui` | GUI tests (requires display) |
| `decimal` | Extended decimal tests |
| `cpu` | CPU-intensive tests |
| `subprocess` | subprocess tests |
| `urlfetch` | Tests downloading from URLs |
| `all` | All of the above |

```bash
# Run everything
./python -m test -u all -j$(nproc)
```

#### 5.2.4 Cross-Platform Testing

For cross-compiled builds, use the `_PYTHON_HOSTRUNNER` environment variable:

```bash
# Emscripten: run via Node.js
export _PYTHON_HOSTRUNNER="node"

# WASI: run via wasmtime
export _PYTHON_HOSTRUNNER="wasmtime run --dir=. --"

# Android: run via adb
export _PYTHON_HOSTRUNNER="adb shell"
```

### 5.3 Argument Clinic

Argument Clinic processes C extension source files to generate argument parsing boilerplate:

```bash
# Run Clinic on all files
make clinic

# Or directly:
python Tools/clinic/libclinic/cli.py --make --srcdir .
```

CLI flags:

| Flag | Description |
|------|-------------|
| `-f` / `--force` | Force regeneration of all files |
| `-o FILE` / `--output FILE` | Write output to specific file |
| `-v` / `--verbose` | Verbose output |
| `--converters` | List available argument converters |
| `--make` | Operate in make mode (check timestamps) |
| `--srcdir DIR` | Source directory root |
| `--exclude PATTERN` | Exclude files matching pattern |
| `--limited` | Use limited C API only |

> [!NOTE]
> Argument Clinic generates the `Modules/clinic/` directory (79 files, 2.2M chars). These files should be committed to the repository — they are regenerated by `make clinic` but checked in for bootstrapping (building Python without Python).

### 5.4 Code Generation Tools

#### 5.4.1 Cases Generator

The `Tools/cases_generator/` builds the bytecode interpreter from `Python/bytecodes.c`:

| Generator Script | Output | Description |
|-----------------|--------|-------------|
| `tier1_generator.py` | `Python/generated_cases.c.h` | Tier-1 bytecode interpreter |
| `tier2_generator.py` | `Python/executor_cases.c.h` | Tier-2 uop interpreter |
| `opcode_id_generator.py` | `Include/opcode_ids.h` | Opcode ID definitions |
| `opcode_metadata_generator.py` | `Include/internal/pycore_opcode_metadata.h` | Opcode metadata |
| `uop_id_generator.py` | `Include/internal/pycore_uop_ids.h` | Micro-op ID definitions |
| `uop_metadata_generator.py` | `Include/internal/pycore_uop_metadata.h` | Micro-op metadata |
| `optimizer_generator.py` | Optimizer cases | Optimization rules |
| `py_metadata_generator.py` | `Lib/_opcode_metadata.py` | Python-side opcode metadata |
| `target_generator.py` | Dispatch targets | Computed-goto dispatch table |
| `record_function_generator.py` | Recording functions | JIT recording functions |

All generators accept `-o/--output` and `input` file arguments. Tier-1 and tier-2 generators also accept `-l/--emit-line-directives`.

```bash
# Regenerate all cases
make regen-cases

# Or individually:
python Tools/cases_generator/tier1_generator.py \
    -o Python/generated_cases.c.h \
    Python/bytecodes.c
```

#### 5.4.2 PEG Parser Generator

```bash
# Regenerate the parser from Grammar/python.gram
make regen-pegen

# Or directly:
python -m Tools.peg_generator Grammar/python.gram \
    -o Parser/parser.c
```

PEG generator flags:

| Flag | Description |
|------|-------------|
| `-q` / `--quiet` | Suppress output |
| `-v` / `--verbose` | Verbose output |
| `--compile-extension` | Compile as extension module |
| `--optimized` | Generate optimized parser |
| `--skip-actions` | Skip semantic actions |

#### 5.4.3 JIT Build Tool

```bash
# Build JIT stencils
python Tools/jit/build.py x86_64-unknown-linux-gnu \
    -o Python/ \
    -v
```

JIT build flags:

| Flag | Description |
|------|-------------|
| `target` | Target triple (e.g., `x86_64-unknown-linux-gnu`) |
| `-d` / `--debug` | Debug mode |
| `-f` / `--force` | Force rebuild |
| `-o DIR` / `--output-dir DIR` | Output directory |
| `-p DIR` / `--pyconfig-dir DIR` | pyconfig.h directory |
| `-v` / `--verbose` | Verbose output |
| `--cflags FLAGS` | Extra compiler flags |
| `--llvm-version VER` | LLVM version to use |
| `--llvm-tools-install-dir DIR` | LLVM tools installation directory |

### 5.5 External Dependency Management (Windows)

```bash
# Download all external dependencies
python PCbuild/get_external.py --all

# Download specific dependency
python PCbuild/get_external.py openssl
```

The `EXTERNALS_DIR` environment variable overrides the default download location. `CACHE_DIR` controls where downloaded archives are cached.

### 5.6 Regeneration Targets Summary

| Target | What It Regenerates |
|--------|-------------------|
| `make regen-all` | Everything below |
| `make regen-opcode` | Opcode tables from `Lib/opcode.py` |
| `make regen-opcode-targets` | Computed-goto dispatch targets |
| `make regen-token` | Token definitions from `Grammar/Tokens` |
| `make regen-ast` | AST code from `Parser/Python.asdl` |
| `make regen-pegen` | PEG parser from `Grammar/python.gram` |
| `make regen-cases` | Bytecode/uop interpreter from `Python/bytecodes.c` |
| `make regen-jit` | JIT stencils (requires LLVM) |
| `make clinic` | Argument Clinic output |

### 5.7 Continuous Integration

CPython uses Buildbot for CI (`Tools/buildbot/`). The `.github/` directory contains GitHub Actions workflows. Key CI dimensions:

- Multiple platforms (Linux, macOS, Windows, WASI, Emscripten)
- Multiple architectures (x86_64, arm64, s390x, ppc64le)
- Debug and release builds
- Free-threaded builds
- Address/memory/undefined-behavior/thread sanitizer builds
- Reference leak hunts
- PGO builds

### 5.8 Pre-commit Hooks

The `.pre-commit-config.yaml` configures automated checks:

```bash
# Install pre-commit hooks
pip install pre-commit
pre-commit install

# Run manually
pre-commit run --all-files
```

### 5.9 Linting and Code Style

CPython uses Ruff (`.ruff.toml`) for Python code linting:

```bash
# Check
ruff check .

# Fix
ruff check --fix .
```

C code follows the CPython style guide (PEP 7): 4-space indentation, specific brace placement, 79-char line limit.

---

## 6. Performance

### 6.1 Build-Time Optimizations

#### 6.1.1 Profile-Guided Optimization (PGO)

PGO uses runtime profiling data to optimize hot code paths:

```bash
# Combined PGO + LTO
./configure --enable-optimizations
make -j$(nproc)
```

The process:
1. Build instrumented binary
2. Run training workload (test suite subset via `--pgo` flag)
3. Rebuild with profile data
4. Apply LTO

Typical speedup: 5-20% on real workloads.

#### 6.1.2 Link-Time Optimization (LTO)

```bash
# Thin LTO (faster compile, slightly less optimization)
./configure --with-lto=thin

# Full LTO (slower compile, maximum optimization)
./configure --with-lto=full
```

#### 6.1.3 BOLT Optimization

BOLT is a post-link optimizer that rearranges code layout based on profile data:

```bash
./configure --enable-bolt
make -j$(nproc)
make bolt-opt
```

Requires LLVM's `llvm-bolt` tool. Provides 2-5% additional speedup on top of PGO+LTO.

#### 6.1.4 Computed Gotos

Computed gotos use GCC/Clang's `&&label` extension for faster bytecode dispatch:

```bash
./configure --with-computed-gotos  # Usually auto-detected
```

Provides 15-20% speedup in the bytecode interpreter loop compared to switch-based dispatch.

### 6.2 Runtime Performance Tuning

#### 6.2.1 Memory Allocators

| Allocator | Flag | Best For |
|-----------|------|----------|
| pymalloc | `PYTHONMALLOC=pymalloc` | Default, general-purpose |
| mimalloc | `PYTHONMALLOC=mimalloc` | Multi-threaded, free-threading |
| malloc | `PYTHONMALLOC=malloc` | Debugging with system tools |
| pymalloc_debug | `PYTHONMALLOC=pymalloc_debug` | Debug build memory checking |

```bash
# Use mimalloc (if built with mimalloc support)
PYTHONMALLOC=mimalloc python script.py

# Debug allocator (catches buffer overflows, use-after-free)
PYTHONMALLOC=debug python script.py
```

#### 6.2.2 JIT Compiler

The experimental JIT compiler targets the micro-op (uop) tier:

```bash
# Build with JIT
./configure --enable-experimental-jit --enable-optimizations
make -j$(nproc)
```

The JIT uses LLVM to compile C stubs into machine code templates at build time, then patches them together at runtime for hot code paths. The copy-and-patch approach avoids the overhead of a full JIT compilation pipeline at runtime.

#### 6.2.3 Free-Threading Performance

Free-threaded builds trade single-threaded performance (~5-10% regression due to atomic operations) for true multi-threaded parallelism:

```bash
# Build free-threaded
./configure --disable-gil --enable-optimizations
make -j$(nproc)

# Control at runtime
PYTHON_GIL=0 python threaded_workload.py
```

> [!IMPORTANT]
> Free-threaded CPython uses biased reference counting: the owning thread uses non-atomic increments/decrements, while other threads use atomic operations. This minimizes contention for objects that stay on one thread. Extension modules must be updated to be thread-safe — check `Py_mod_gil` slot.

### 6.3 Profiling and Benchmarking

#### 6.3.1 New Profiling Module

CPython now includes `Lib/profiling/` with both sampling and tracing profilers:

```bash
# Sampling profiler
python -m profiling.sampling script.py

# Tracing profiler
python -m profiling.tracing script.py
```

The sampling profiler (`Lib/profiling/sampling/`, 19 files, 244K chars) periodically samples the call stack, providing low-overhead profiling suitable for production. The tracing profiler (`Lib/profiling/tracing/`, 3 files) hooks into the interpreter's tracing mechanism for precise call/return timing.

#### 6.3.2 Import Profiling

```bash
# Profile import times
PYTHONPROFILEIMPORTTIME=1 python -c "import ssl"
```

Expected output:
```text
import time: self [us] | cumulative | imported package
import time:       342 |        342 |   _io
import time:       123 |        123 |   marshal
...
```

#### 6.3.3 Tracemalloc

```bash
# Track memory allocations (10-frame traceback)
PYTHONTRACEMALLOC=10 python script.py
```

#### 6.3.4 Built-in Benchmarking Tools

| Tool | Location | Purpose |
|------|----------|---------|
| Import benchmark | `Tools/importbench/importbench.py` | Import system performance |
| Lock benchmark | `Tools/lockbench/lockbench.py` | Lock contention measurement |
| Free-threading scaling | `Tools/ftscalingbench/ftscalingbench.py` | Multi-threaded scaling analysis |
| Pickle benchmark | `Tools/picklebench/memory_dos_impact.py` | Pickle memory/performance/DoS |
| External inspection | `Tools/inspection/benchmark_external_inspection.py` | Remote debugging overhead |

#### 6.3.5 DTrace / SystemTap Probes

When built with `--with-dtrace`:

```bash
# List probes
sudo dtrace -l -n 'python*:::'

# Trace function calls
sudo dtrace -n 'python$target:::function-entry { printf("%s %s", copyinstr(arg0), copyinstr(arg1)); }' -c "./python script.py"
```

Available probes:
- `function-entry` / `function-return` — Python function calls
- `line` — Line execution
- `gc-start` / `gc-done` — Garbage collection
- `import-find-load-start` / `import-find-load-done` — Import events

### 6.4 Statistics Collection

When built with `--enable-pystats`:

```bash
./python -X pystats script.py
# Creates /tmp/py_stats/ with detailed internal statistics
```

Statistics cover: opcode frequencies, specialization success/failure rates, call sites, object allocations, and more.

### 6.5 Valgrind Integration

```bash
./configure --with-valgrind
make -j$(nproc)

# Run with suppression file
valgrind --suppressions=Misc/valgrind-python.supp ./python script.py
```

The `Misc/valgrind-python.supp` file suppresses known false positives from Python's memory allocator. For most accurate results, also use `PYTHONMALLOC=malloc` to bypass pymalloc.

---

## 7. Security

### 7.1 Security Architecture

CPython's security model operates at multiple levels:

```text
Application Level:  hashlib, ssl, secrets, hmac
    │
Interpreter Level:  hash randomization, safe eval, import restrictions
    │
Memory Level:       buffer overflow protections, object type checking
    │
Build Level:        ASAN/MSAN/UBSAN, stack protections, ASLR
    │
Network Level:      ssl module (OpenSSL), http.server TLS, certificate verification
```

### 7.2 Hash Randomization

Since Python 3.3, string hashing is randomized per-process to prevent hash-collision denial-of-service attacks:

```bash
# Default: random seed each process
python -c "print(hash('test'))"  # Different each run

# Reproducible hashing (testing only)
PYTHONHASHSEED=42 python -c "print(hash('test'))"

# Disable randomization (INSECURE)
PYTHONHASHSEED=0 python -c "print(hash('test'))"
```

> [!WARNING]
> Setting `PYTHONHASHSEED=0` disables hash randomization, making the process vulnerable to hash-collision DoS attacks. Never do this in production. Use a fixed seed (`PYTHONHASHSEED=N` where N > 0) only for reproducible testing.

### 7.3 SSL/TLS Configuration

CPython's `_ssl` module wraps OpenSSL. Key security considerations:

```python
import ssl

# Create secure default context
ctx = ssl.create_default_context()

# Properties that are ON by default:
#   - Certificate verification (CERT_REQUIRED)
#   - Hostname checking
#   - Modern protocol versions only (TLSv1.2+)
#   - Secure cipher suites
```

Build-time SSL configuration:

```bash
# Specify OpenSSL location
./configure --with-openssl=/opt/openssl-3.0 \
            --with-openssl-rpath=auto

# Override default cipher suites
./configure --with-ssl-default-suites="TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY25519_SHA256"

# Select hash algorithms to compile in
./configure --with-builtin-hashlib-hashes=md5,sha1,sha256,sha512,sha3,blake2
```

The `Tools/ssl/multissltests.py` tool tests against multiple OpenSSL versions simultaneously.

### 7.4 Integer-String Conversion Limit

To prevent denial-of-service via extremely large integer-to-string conversions:

```bash
# Default limit: 4300 digits
python -c "int('1' * 5000)"
# ValueError: Exceeds the limit (4300 digits) for integer string conversion

# Override via environment variable
PYTHONINTMAXSTRDIGITS=10000 python -c "print(len(str(10**9999)))"

# Disable limit (INSECURE for untrusted input)
PYTHONINTMAXSTRDIGITS=0 python script.py
```

### 7.5 Sandboxing and Isolation

#### 7.5.1 Isolated Mode

```bash
# Isolated mode: no user site-packages, no PYTHON* env vars
python -I script.py
```

Equivalent to `-s -E -P`:
- `-s`: Don't add user site-packages
- `-E`: Ignore all PYTHON* environment variables
- `-P`: Don't prepend script directory to `sys.path`

#### 7.5.2 Restricted Imports

```python
import importlib

# Disable import of specific modules
# (custom import hook approach)
```

#### 7.5.3 Subinterpreters for Isolation

```python
import concurrent.interpreters

# Each interpreter has its own GIL and module state
interp = concurrent.interpreters.create()
interp.exec("import os; print(os.getpid())")
```

### 7.6 Secure Build Flags

```bash
# Full security-hardened build
CFLAGS="-fstack-protector-strong -D_FORTIFY_SOURCE=2 -fPIE" \
LDFLAGS="-Wl,-z,relro,-z,now -pie" \
./configure --enable-optimizations --with-openssl=/usr/local/ssl

# Sanitizer builds for security testing
./configure --with-address-sanitizer    # Buffer overflows
./configure --with-memory-sanitizer     # Uninitialized reads
./configure --with-undefined-behavior-sanitizer  # UB
./configure --with-thread-sanitizer     # Data races
```

### 7.7 Known Security Issues

From the issue tracker:

| Issue | Description | Severity |
|-------|-------------|----------|
| #148954 | `xmlrpc.client` lacks sanitization in method name — potential injection | Medium |
| #148950 | Endless polling on EOF socket — potential DoS via resource exhaustion | Medium |
| #131684 | `pkgutil.walk_packages(None)` imports test modules — code execution risk | Low |

> [!CAUTION]
> The `xmlrpc.client` module does not sanitize method names (issue #148954). If your application constructs XML-RPC method names from user input, validate and sanitize the input before passing it to `xmlrpc.client.ServerProxy`. Consider using JSON-RPC or REST APIs instead for new applications.

### 7.8 HTTP Server TLS

CPython's `http.server` now includes HTTPS support:

```python
from http.server import HTTPSServer, SimpleHTTPRequestHandler
import ssl

context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain('cert.pem', 'key.pem')

server = HTTPSServer(('', 443), SimpleHTTPRequestHandler, context=context)
server.serve_forever()
```

The `ThreadingHTTPSServer` variant handles each request in a new thread.

> [!WARNING]
> The `http.server` module (including `HTTPSServer`) is not designed for production use. It lacks hardening against slowloris attacks, request smuggling, and other HTTP-level attacks. Use a production WSGI/ASGI server (Gunicorn, uvicorn, etc.) for public-facing deployments.

### 7.9 Pickle Security

The `pickle` module is inherently insecure for untrusted data:

```python
import pickle

# NEVER unpickle untrusted data
# pickle.loads(untrusted_bytes)  # Arbitrary code execution!

# The Tools/picklebench/memory_dos_impact.py benchmarks
# pickle memory and DoS characteristics
```

### 7.10 Source Date Epoch

For reproducible builds, set `SOURCE_DATE_EPOCH`:

```bash
# Reproducible .pyc timestamps
SOURCE_DATE_EPOCH=$(date +%s) make install
```

This ensures `.pyc` files and other build artifacts have deterministic timestamps.

---

## 8. Developer Internals

### 8.1 Object Model

Every Python object starts with a `PyObject` header:

```c
// Include/object.h
typedef struct _object {
    Py_ssize_t ob_refcnt;       // Reference count (or biased refcnt in free-threaded)
    PyTypeObject *ob_type;       // Type pointer
} PyObject;

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size;          // Number of items (for variable-size objects)
} PyVarObject;
```

Key type implementations in `Objects/`:

| File | Type | Size | Key Features |
|------|------|------|-------------|
| `longobject.c` | `int` | 437K | Arbitrary precision, small-int cache (-5 to 256) |
| `unicodeobject.c` | `str` | 625K | Compact (Latin-1/UCS-2/UCS-4), interning |
| `listobject.c` | `list` | 108K | Dynamic array, over-allocation |
| `dictobject.c` | `dict` | 280K | Compact dict (split keys), ordered insertion |
| `setobject.c` | `set` | 107K | Hash table with open addressing |
| `tupleobject.c` | `tuple` | 50K | Immutable, free list |
| `bytesobject.c` | `bytes` | 128K | Immutable byte sequence |
| `floatobject.c` | `float` | 65K | C double, free list |
| `complexobject.c` | `complex` | 25K | Pair of C doubles |
| `typeobject.c` | `type` | 400K | Metaclass, MRO, descriptor protocol |
| `funcobject.c` | `function` | 62K | Python functions, closures |
| `frameobject.c` | `frame` | 46K | Execution frame objects |
| `codeobject.c` | `code` | 122K | Compiled bytecode |
| `genobject.c` | `generator` | 62K | Generators, coroutines, async generators |
| `iterobject.c` | `iterator` | 10K | Iterator protocol |
| `memoryobject.c` | `memoryview` | 115K | Buffer protocol view |
| `moduleobject.c` | `module` | 24K | Module objects |
| `namespaceobject.c` | `SimpleNamespace` | 7K | Simple attribute namespace |
| `rangeobject.c` | `range` | 27K | Lazy integer ranges |
| `sliceobject.c` | `slice` | 17K | Slice objects |
| `boolobject.c` | `bool` | 8K | Boolean (subclass of int) |

### 8.2 Bytecode Compiler Pipeline

The compilation pipeline transforms source to executable code objects:

```text
Source string
    │
    ▼
tokenize (Parser/tokenizer/, Parser/lexer/)
    │  Produces token stream
    ▼
parse (Parser/parser.c, generated from Grammar/python.gram)
    │  Produces CST → AST via Parser/action_helpers.c
    ▼
ast_opt (Python/ast_opt.c)
    │  Constant folding, optimization
    ▼
symtable (Python/symtable.c)
    │  Variable scope analysis (local/global/free/cell)
    ▼
compile (Python/compile.c)
    │  AST → intermediate representation
    ▼
codegen (Python/codegen.c)
    │  IR → bytecode instructions
    ▼
flowgraph (Python/flowgraph.c)
    │  Basic block optimization, jump threading
    ▼
assemble
    │  Emit final bytecode, create code object
    ▼
PyCodeObject (Objects/codeobject.c)
```

### 8.3 Evaluation Loop

The main interpreter loop lives in `Python/ceval.c`, with the actual opcode dispatch in `Python/generated_cases.c.h` (generated from `Python/bytecodes.c`).

```text
Tier 1 (Bytecode):
  Python/ceval.c          — Main eval loop, frame management
  Python/generated_cases.c.h — Opcode implementations (generated)
  Python/ceval_macros.h   — Helper macros

Tier 2 (Micro-ops):
  Python/executor_cases.c.h — Uop implementations (generated)
  Python/optimizer.c       — Trace optimization
  Python/optimizer_analysis.c — Abstract interpretation
  Python/optimizer_bytecodes.c — Optimizer uop definitions
  Python/optimizer_symbols.c — Symbolic analysis

JIT:
  Python/jit.c             — JIT runtime (copy-and-patch)
  Tools/jit/build.py       — JIT stencil compiler
```

The evaluation loop uses a two-tier strategy:
1. **Tier 1**: Standard bytecode dispatch (switch or computed gotos)
2. **Tier 2**: Hot loops are "specialized" into micro-op traces, then optionally JIT-compiled

### 8.4 Specialization (Adaptive Interpreter)

CPython uses inline caching and instruction specialization:

```text
LOAD_ATTR (generic)
    │  After N executions of same pattern
    ▼
LOAD_ATTR_INSTANCE_VALUE (specialized for instance attribute)
    │  If type changes
    ▼
LOAD_ATTR (deoptimized back to generic)
```

Specialized opcodes exist for:
- `LOAD_ATTR` → `LOAD_ATTR_INSTANCE_VALUE`, `LOAD_ATTR_MODULE`, `LOAD_ATTR_CLASS`, etc.
- `LOAD_GLOBAL` → `LOAD_GLOBAL_BUILTIN`, `LOAD_GLOBAL_MODULE`
- `BINARY_OP` → `BINARY_OP_ADD_INT`, `BINARY_OP_ADD_FLOAT`, `BINARY_OP_ADD_UNICODE`
- `CALL` → `CALL_PY_EXACT_ARGS`, `CALL_BUILTIN_CLASS`, `CALL_BUILTIN_FAST`, etc.
- `COMPARE_OP` → `COMPARE_OP_INT`, `COMPARE_OP_FLOAT`, `COMPARE_OP_STR`

### 8.5 Garbage Collection

CPython uses reference counting as the primary memory management strategy, supplemented by a cyclic garbage collector:

```text
Primary: Reference Counting
  - Immediate cleanup when refcount reaches 0
  - Deterministic finalization (predictable __del__ timing)
  - Free-threaded builds use biased reference counting

Secondary: Cyclic GC (Python/gc.c)
  - Detects reference cycles (unreachable but non-zero refcount)
  - Three generations (0, 1, 2)
  - Generation 0: checked frequently (every ~700 allocations)
  - Generation 1: checked less often
  - Generation 2: checked rarely (long-lived objects)
```

GC control:

```python
import gc

gc.disable()                    # Disable cyclic GC
gc.collect()                    # Force collection
gc.set_threshold(700, 10, 10)   # Set generation thresholds
gc.get_stats()                  # Collection statistics
gc.freeze()                     # Freeze all tracked objects
```

### 8.6 Import System

The import system (`Python/import.c`, `Lib/importlib/`) follows this resolution order:

```text
import foo
    │
    ▼
1. sys.modules cache?  → Return cached module
    │ (miss)
    ▼
2. sys.meta_path finders (in order):
    ├── BuiltinImporter (built-in C modules)
    ├── FrozenImporter (frozen modules)
    └── PathFinder (sys.path-based)
            │
            ▼
        3. sys.path entries → sys.path_hooks → Finders:
            ├── zipimport.zipimporter (.zip files)
            └── FileFinder (.py, .pyc, .so/.pyd)
                    │
                    ▼
                4. Loader.exec_module()
                    │
                    ▼
                5. Module in sys.modules
```

CPython now supports lazy imports:
```python
# Deferred import (evaluated on first attribute access)
lazy import _colorize
```

### 8.7 Memory Allocator Internals

#### 8.7.1 pymalloc (Objects/obmalloc.c)

```text
Arena (256 KB, mmap'd)
    │
    ├── Pool (4 KB page, one size class)
    │   ├── Block (8 bytes)
    │   ├── Block (8 bytes)
    │   └── ...
    │
    ├── Pool (4 KB, different size class)
    │   ├── Block (16 bytes)
    │   └── ...
    │
    └── ...

Size classes: 8, 16, 24, 32, ..., 512 bytes (64 classes)
Requests > 512 bytes → system malloc
```

#### 8.7.2 mimalloc (Objects/mimalloc/)

Microsoft's mimalloc is integrated for improved multi-threaded performance:

```text
Heap (per-thread)
    │
    ├── Segment (configurable, typically 4 MB)
    │   ├── Page (size-class specific)
    │   │   ├── Block
    │   │   └── ...
    │   └── ...
    │
    └── Large allocations → OS directly
```

mimalloc is the default allocator in free-threaded builds. Key advantages:
- Per-thread heaps reduce contention
- Bounded worst-case fragmentation
- Compatible with `fork()` (unlike some alternatives)

### 8.8 C API Layers

CPython exposes three API tiers:

| Layer | Headers | Stability | Use |
|-------|---------|-----------|-----|
| Stable ABI (abi3) | `Include/*.h` (subset) | Binary-compatible across versions | Wheels tagged `abi3` |
| Public API | `Include/*.h` | Source-compatible, may change | Standard extensions |
| CPython-specific | `Include/cpython/*.h` | May change between versions | CPython-only extensions |
| Internal | `Include/internal/*.h` | No stability guarantees | CPython core only |

Key API families:

```text
Include/
├── Python.h          — Master include (pulls in everything public)
├── object.h          — PyObject, Py_INCREF/DECREF, type system
├── abstract.h        — Abstract object protocol (PySequence_*, PyMapping_*)
├── pystate.h         — Thread/interpreter state
├── pylifecycle.h     — Py_Initialize, Py_Finalize
├── import.h          — Import API
├── compile.h         — Compilation API
├── eval.h            — Evaluation API
├── pyerrors.h        — Exception handling
├── pymem.h           — Memory allocation (PyMem_*, PyObject_*)
├── pycapsule.h       — Capsule objects (opaque C pointer passing)
├── moduleobject.h    — Module definition API
├── methodobject.h    — C function/method objects
└── ...

Include/internal/     (144 files, 1.6M chars)
├── pycore_ceval.h       — Eval loop internals
├── pycore_gc.h          — GC internals
├── pycore_interp.h      — Interpreter state
├── pycore_object.h      — Object internals
├── pycore_pystate.h     — Thread state internals
├── pycore_qsbr.h        — QSBR (free-threading)
├── pycore_dict.h        — Dict internals
├── pycore_long.h        — Int internals
├── pycore_uop_ids.h     — Micro-op IDs
├── pycore_opcode_metadata.h — Opcode metadata
└── ...
```

### 8.9 C Extension Module Structure

Modern C extension modules use multi-phase initialization:

```c
// Module definition
static struct PyModuleDef mymodule = {
    PyModuleDef_HEAD_INIT,
    .m_name = "mymodule",
    .m_doc = "My extension module",
    .m_size = -1,  // or sizeof(module_state) for per-interpreter state
    .m_methods = mymodule_methods,
    .m_slots = mymodule_slots,  // Multi-phase init
};

// Slots for multi-phase init
static PyModuleDef_Slot mymodule_slots[] = {
    {Py_mod_exec, mymodule_exec},
    {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
    {Py_mod_gil, Py_MOD_GIL_NOT_USED},  // Free-threading support
    {0, NULL}
};

PyMODINIT_FUNC
PyInit_mymodule(void)
{
    return PyModuleDef_Init(&mymodule);
}
```

The `Py_mod_gil` slot is critical for free-threading compatibility:
- `Py_MOD_GIL_USED`: Module requires GIL (default)
- `Py_MOD_GIL_NOT_USED`: Module is thread-safe

### 8.10 Standard Library Module Architecture

#### 8.10.1 asyncio (`Lib/asyncio/`, 35 files, 530K chars)

```text
asyncio/
├── base_events.py      — BaseEventLoop (core event loop)
├── events.py           — Event loop policy, abstract base
├── futures.py          — Future objects
├── tasks.py            — Task objects (coroutine wrappers)
├── streams.py          — High-level stream API
├── protocols.py        — Protocol abstract base classes
├── transports.py       — Transport abstract base classes
├── selector_events.py  — select/poll/epoll-based event loop
├── unix_events.py      — Unix-specific (signals, child processes)
├── windows_events.py   — IOCP-based event loop
├── proactor_events.py  — Proactor-based event loop (Windows)
├── sslproto.py         — SSL/TLS protocol implementation
├── subprocess.py       — Subprocess management
├── queues.py           — Async queues
├── locks.py            — Async synchronization primitives
├── runners.py          — asyncio.run() and Runner
├── taskgroups.py       — TaskGroup (structured concurrency)
├── timeouts.py         — Timeout context manager
├── staggered.py        — Staggered connection racing
├── trsock.py           — Transport socket wrapper
├── mixins.py           — Event loop mixins
├── exceptions.py       — asyncio-specific exceptions
├── coroutines.py       — Coroutine utilities
├── constants.py        — Module constants
└── __main__.py         — python -m asyncio (async REPL)
```

#### 8.10.2 multiprocessing (`Lib/multiprocessing/`, 21 files, 306K chars)

```text
multiprocessing/
├── context.py          — Process start method contexts (fork, spawn, forkserver)
├── process.py          — Process class
├── pool.py             — Worker pool
├── queues.py           — Process-safe queues
├── managers.py         — Shared state managers
├── shared_memory.py    — SharedMemory (zero-copy IPC)
├── connection.py       — Pipe/socket connections
├── synchronize.py      — Locks, semaphores, events, conditions
├── sharedctypes.py     — Shared ctypes objects
├── spawn.py            — Spawn helpers
├── forkserver.py       — Fork server implementation
├── heap.py             — Shared memory heap
├── popen_*.py          — Platform-specific process creation
├── reduction.py        — Object serialization for IPC
├── resource_tracker.py — Shared resource cleanup
└── util.py             — Utility functions
```

#### 8.10.3 importlib (`Lib/importlib/`, 9 files, 141K chars)

```text
importlib/
├── __init__.py         — Public API (import_module, reload, etc.)
├── abc.py              — Abstract base classes (Finder, Loader)
├── machinery.py        — Import machinery access
├── metadata/           — Package metadata (PEP 566)
├── readers.py          — Resource readers
├── resources/          — Package resources (PEP 302)
├── _abc.py             — ABC implementation details
├── _bootstrap.py       — Core import bootstrapping (frozen)
└── _bootstrap_external.py — File-based importing (frozen)
```

#### 8.10.4 http (`Lib/http/`, 5 files, 209K chars)

```text
http/
├── __init__.py     — HTTP status codes
├── server.py       — HTTP/HTTPS servers (HTTPServer, HTTPSServer,
│                     ThreadingHTTPServer, ThreadingHTTPSServer,
│                     BaseHTTPRequestHandler, SimpleHTTPRequestHandler,
│                     CGIHTTPRequestHandler)
├── client.py       — HTTP client (HTTPConnection, HTTPSConnection)
├── cookiejar.py    — Cookie handling
└── cookies.py      — Cookie parsing (SimpleCookie)
```

#### 8.10.5 email (`Lib/email/`, 20 files, 376K chars)

```text
email/
├── message.py          — Message/EmailMessage classes
├── policy.py           — Email policies (compat32, email5)
├── contentmanager.py   — Content type management
├── headerregistry.py   — Structured header parsing
├── generator.py        — Message serialization
├── parser.py           — Message parsing (FeedParser, BytesFeedParser)
├── mime/               — MIME type classes
├── charset.py          — Character set handling
├── encoders.py         — Content transfer encoding
├── errors.py           — Email-specific exceptions
├── utils.py            — Utility functions (parseaddr, formatdate, etc.)
├── header.py           — Legacy header handling
├── base64mime.py       — Base64 MIME encoding
├── quoprimime.py       — Quoted-printable MIME encoding
└── iterators.py        — Message iteration
```

#### 8.10.6 Key C Extension Modules (`Modules/`)

| Module | File(s) | Size | Purpose |
|--------|---------|------|---------|
| `_io` | `_io/` | Multiple | I/O implementation (BufferedReader/Writer, TextIOWrapper, FileIO) |
| `_ssl` | `_ssl.c` + `_ssl/` | Large | OpenSSL bindings |
| `_sqlite3` | `_sqlite3/` | Multiple | SQLite3 bindings |
| `_ctypes` | `_ctypes/` | Multiple | Foreign function interface |
| `_decimal` | `_decimal/` | Large | Decimal arithmetic (libmpdec) |
| `_zstd` | `_zstd/` | Multiple | Zstandard compression |
| `_sre` | `_sre/sre.c` | 85K | Regular expression engine |
| `_json` | `_json.c` | 23K | JSON encoder/decoder acceleration |
| `_pickle` | `_pickle.c` | 176K | Pickle serialization |
| `_struct` | `_struct.c` | 55K | Binary data packing/unpacking |
| `_socket` | `socketmodule.c` | 260K | Socket interface |
| `_asyncio` | `_asynciomodule.c` | 105K | asyncio acceleration |
| `_multiprocessing` | `_multiprocessing/` | Multiple | multiprocessing acceleration |
| `_hashlib` | `_hashopenssl.c` | 55K | OpenSSL hash functions |
| `_csv` | `_csv.c` | 38K | CSV parser |
| `_datetime` | `_datetimemodule.c` | 205K | Date/time implementation |
| `_elementtree` | `_elementtree.c` | 109K | XML ElementTree acceleration |
| `pyexpat` | `expat/` | Bundled | XML expat parser |
| `_remote_debugging` | `_remote_debugging/` | Multiple | Remote debugging/inspection |
| `cjkcodecs` | `cjkcodecs/` | Multiple | CJK character encoding |
| `_zoneinfo` | `_zoneinfo.c` | 31K | Timezone support |
| `_statistics` | `_statisticsmodule.c` | 7K | Statistics acceleration |
| `_bisect` | `_bisectmodule.c` | 12K | Bisect algorithm |
| `_heapq` | `_heapqmodule.c` | 17K | Heap queue acceleration |
| `_collections` | `_collectionsmodule.c` | 75K | Deque, Counter, OrderedDict acceleration |
| `_functools` | `_functoolsmodule.c` | 46K | functools acceleration |
| `_operator` | `_operator.c` | 30K | operator module acceleration |

### 8.11 Parser Architecture

The PEG parser replaced the LL(1) parser in Python 3.9:

```text
Grammar/python.gram (PEG grammar definition)
    │
    ▼
Tools/peg_generator/ (pegen)
    │  Generates C parser from grammar
    ▼
Parser/parser.c (generated, ~24K lines)
    │
    ├── Parser/tokenizer/tokenize.c — Byte-level tokenization
    ├── Parser/lexer/lexer.c — Token classification
    ├── Parser/string_parser.c — f-string/string literal parsing
    ├── Parser/action_helpers.c — Semantic actions (AST construction)
    └── Parser/pegen.c — Parser runtime support
```

### 8.12 Compilation to Bytecode

The compiler (`Python/compile.c`, `Python/codegen.c`, `Python/flowgraph.c`) performs:

1. **Symbol table analysis** (`Python/symtable.c`): Determines variable scoping (local, global, free, cell)
2. **AST optimization** (`Python/ast_opt.c`): Constant folding, dead code elimination
3. **Code generation** (`Python/codegen.c`): AST → basic blocks with instructions
4. **Flow graph optimization** (`Python/flowgraph.c`): Jump threading, unreachable code removal, peephole optimizations
5. **Assembly** (in `Python/compile.c`): Final bytecode emission, line number table, exception table

### 8.13 String Implementation

Strings use a flexible internal representation (PEP 393):

```text
PyUnicodeObject
├── ASCII (all chars < 128): 1 byte per char, compact layout
├── Latin-1 (all chars < 256): 1 byte per char, compact layout
├── BMP (all chars < 65536): 2 bytes per char (UCS-2)
└── Full Unicode: 4 bytes per char (UCS-4)
```

The `Objects/stringlib/` directory (20 files, 158K chars) contains shared algorithms used by both `str` and `bytes`:
- Boyer-Moore-Horspool search
- Adaptive merge sort
- Split/join/replace algorithms
- Numeric formatting

### 8.14 Dict Implementation

Python dicts use a compact, ordered layout:

```text
Compact dict structure:
┌─────────────┐
│  Hash table │  Sparse array of indices (1/2/4/8 bytes each)
│  (indices)  │  Maps hash(key) % size → index into entries
├─────────────┤
│  Entries    │  Dense array of (hash, key, value) tuples
│  (compact)  │  Insertion-ordered
└─────────────┘
```

Split-table optimization: When multiple instances of the same class share the same attribute names, they share a single keys table, saving significant memory.

### 8.15 GDB Debugging Helpers

`Tools/gdb/` provides GDB Python extensions for debugging CPython:

```bash
# Load in GDB
source Tools/gdb/libpython.py

# Commands available:
py-bt          # Python traceback
py-list        # Python source listing
py-print       # Print Python variable
py-up/py-down  # Navigate Python frames
py-locals      # Print local variables
```

### 8.16 IDLE Architecture (`Lib/idlelib/`)

IDLE is CPython's integrated development environment (60 files, 771K chars):

```text
idlelib/
├── pyshell.py       — Main shell window
├── editor.py        — Editor window
├── run.py           — Execution engine (subprocess)
├── rpc.py           — RPC between IDLE GUI and execution subprocess
├── debugger.py      — Integrated debugger
├── colorizer.py     — Syntax highlighting
├── autocomplete.py  — Tab completion
├── calltip.py       — Function signature tooltips
├── config.py        — Configuration system
├── configdialog.py  — Settings dialog
├── codecontext.py   — Code context (scope display)
├── outwin.py        — Output window
├── tree.py          — Tree widget
├── browser.py       — Module browser
├── pathbrowser.py   — Path browser
├── help.py          — Help system
├── query.py         — Dialog boxes
├── replace.py       — Find/replace
├── grep.py          — Search in files
├── undo.py          — Undo/redo
└── ...
```

### 8.17 Interactive REPL (`Lib/_pyrepl/`)

The new `_pyrepl` (29 files, 293K chars) replaces the basic readline-based REPL:

```text
_pyrepl/
├── console.py       — Console abstraction
├── reader.py        — Input reader with multi-line editing
├── readline.py      — readline compatibility layer
├── commands.py      — Editor commands
├── completing_reader.py — Tab completion
├── keymap.py        — Key binding maps
├── unix_console.py  — Unix terminal implementation
├── windows_console.py — Windows terminal implementation
├── trace.py         — Debug tracing (PYREPL_TRACE)
└── ...
```

Set `PYTHON_BASIC_REPL=1` to fall back to the classic readline REPL.

### 8.18 Encoding System

`Lib/encodings/` contains 123 encoding modules covering:

- ASCII, UTF-8, UTF-16, UTF-32 (with BOM variants)
- Latin/ISO-8859 family (1-16)
- Windows code pages (cp1250-cp1258, cp437, cp850, etc.)
- CJK encodings (GB2312, GBK, GB18030, Big5, EUC-JP, Shift_JIS, EUC-KR, ISO-2022-*)
- EBCDIC variants
- Special encodings (rot_13, base64_codec, hex_codec, unicode_escape)
- Platform-specific (palmos, hp_roman8, mac_roman, etc.)

The `Modules/cjkcodecs/` provides C-accelerated CJK codec implementations.

### 8.19 Compression Architecture

CPython supports multiple compression formats:

| Module | C Module | Library | Format |
|--------|----------|---------|--------|
| `zlib` | `zlibmodule.c` | zlib | DEFLATE/gzip |
| `bz2` | `_bz2module.c` | libbz2 | bzip2 |
| `lzma` | `_lzmamodule.c` | liblzma | LZMA/XZ |
| `_zstd` | `_zstd/` | libzstd | Zstandard |
| `Lib/compression/` | — | — | Unified compression API (new) |

The new `Lib/compression/` module (4 files) provides a unified interface across compression formats.

### 8.20 Remote Debugging (`Modules/_remote_debugging/`)

A new module for external process inspection:

```text
_remote_debugging/
├── _remote_debugging.c     — C extension for attaching to processes
└── ...
```

The `Tools/inspection/benchmark_external_inspection.py` benchmarks the overhead of external debugging attachment.

---

## 9. Integration Points

### 9.1 Extending CPython with C Extensions

The extension module ABI layers:

| ABI | Header Guard | Binary Compatibility |
|-----|-------------|---------------------|
| Stable ABI (abi3) | `Py_LIMITED_API` | Forward-compatible across 3.x |
| Full API | None | Compatible within 3.x.y |
| Internal API | `Py_BUILD_CORE` | No guarantees |

Building an extension module:

```bash
# Using setuptools/distutils
python setup.py build_ext --inplace

# Using pkg-config
gcc -shared -o mymodule.so mymodule.c \
    $(python3.15-config --cflags --ldflags) \
    -fPIC
```

### 9.2 Argument Clinic Integration

When writing C extensions for CPython's standard library, Argument Clinic automates argument parsing:

```c
/*[clinic input]
module _mymodule

_mymodule.my_function

    name: str
        The name parameter.
    count: int = 1
        Number of repetitions.
    /

My function's docstring.
[clinic start generated code]*/
```

Running `make clinic` or `python Tools/clinic/libclinic/cli.py -f Modules/_mymodule.c` generates the argument parsing code in `Modules/clinic/_mymodule.c.h`.

### 9.3 sysconfig Integration

`Lib/sysconfig/` (2 files, 37K chars) provides build configuration to downstream tools:

```python
import sysconfig

# Get install paths
sysconfig.get_paths()

# Get build variables
sysconfig.get_config_vars()

# Get specific variable
sysconfig.get_config_var('CFLAGS')
```

```bash
# Command-line access
python -m sysconfig
```

### 9.4 ensurepip

`Lib/ensurepip/` (3 files, 9K chars) bootstraps pip into a Python installation:

```bash
# Bootstrap pip
python -m ensurepip

# Upgrade pip
python -m ensurepip --upgrade

# Install to specific location
python -m ensurepip --root /path/to/root
```

Configure at build time:
```bash
./configure --with-ensurepip=upgrade  # Default
./configure --with-ensurepip=install  # Don't upgrade existing
./configure --with-ensurepip=no       # Skip pip entirely
```

### 9.5 Freeze Tool

`Tools/freeze/` (12 files) creates standalone executables by embedding Python modules into C code:

```bash
# Freeze a script into a standalone executable
python Tools/freeze/freeze.py script.py
make -f Makefile
```

### 9.6 Internationalization

`Tools/i18n/` provides tools for extracting and compiling gettext translations:

```bash
# Extract translatable strings
python Tools/i18n/pygettext.py -o messages.pot script.py

# Compile .po files to .mo
python Tools/i18n/msgfmt.py messages.po
```

### 9.7 Unicode Data Generation

`Tools/unicode/` generates Unicode data tables used by the interpreter:

```bash
# Regenerate Unicode database from UCD
python Tools/unicode/makeunicodedata.py
```

### 9.8 C Analyzer

`Tools/c-analyzer/` analyzes C global variables in CPython's source, used to track and minimize mutable global state (important for subinterpreters and free-threading):

```bash
# Analyze C globals
python -m Tools.c-analyzer.c_analyzer
```

### 9.9 MSI/Installer Tools

- `Tools/msi/`: Windows MSI installer generation
- `Mac/BuildScript/`: macOS installer (`.pkg`) generation
- `PC/layout/`: Windows distribution layout tool

### 9.10 WebAssembly Integration

For embedding CPython in web applications or serverless environments:

```bash
# Emscripten build (browser/Node.js)
python Platforms/emscripten/__main__.py \
    --cross-build-dir=build/wasm \
    wasm32-emscripten

# WASI build (standalone WASM runtime)
python Platforms/WASI/__main__.py \
    --wasi-sdk=/opt/wasi-sdk
```

Browser test dependencies:
- `@playwright/test` — Browser automation
- `http-server` — Local HTTP server for serving WASM

### 9.11 wsgiref (`Lib/wsgiref/`, 7 files, 57K chars)

Reference implementation of PEP 3333 (WSGI):

```python
from wsgiref.simple_server import make_server

def app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return [b'Hello, World!']

httpd = make_server('', 8000, app)
httpd.serve_forever()
```

### 9.12 tkinter (`Lib/tkinter/`, 13 files, 308K chars)

Python binding for Tcl/Tk GUI toolkit. Requires `TCL_LIBRARY` environment variable if Tcl is installed in a non-standard location. Build with `--with-tcltk-includes` and `--with-tcltk-libs` if auto-detection fails.

### 9.13 SQLite Integration

`Lib/sqlite3/` (5 files, 19K chars) wraps the C `_sqlite3` module:

```bash
# SQLite CLI
python -m sqlite3 database.db
```

### 9.14 JSON CLI

```bash
# JSON tool (pretty-print, validate)
python -m json.tool input.json

# JSON Lines mode
echo '{"a":1}' | python -m json.tool --json-lines
```

> [!NOTE]
> Issue #132631 documents a known bug where the JSON CLI's `--json-lines` mode can raise "I/O operation on closed file" errors under certain piping conditions.

### 9.15 zoneinfo (`Lib/zoneinfo/`, 4 files, 37K chars)

Timezone database integration:

```python
from zoneinfo import ZoneInfo
from datetime import datetime

dt = datetime(2026, 5, 7, tzinfo=ZoneInfo("America/New_York"))
```

The `PYTHONTZPATH` environment variable controls the search path for timezone data files.

### 9.16 tomllib (`Lib/tomllib/`, 4 files, 29K chars)

TOML parser (read-only, added in Python 3.11):

```python
import tomllib

with open("pyproject.toml", "rb") as f:
    config = tomllib.load(f)
```

### 9.17 dbm (`Lib/dbm/`, 5 files, 23K chars)

Simple database interface with multiple backends:

```python
import dbm

# Backend selection order controlled by --with-dbmliborder
with dbm.open('mydb', 'c') as db:
    db['key'] = 'value'
```

Configure backend order at build time:
```bash
./configure --with-dbmliborder=gdbm:ndbm:bdb
```

---

## Appendix A: Troubleshooting

### A.1 Build Failures

#### Missing `libffi`
```text
*** WARNING: building of the _ctypes extension module failed
```

**Fix**: Install `libffi-dev` (Debian) or `libffi-devel` (Fedora).

#### Missing OpenSSL
```text
Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer
```

**Fix**: Install `libssl-dev` or build OpenSSL from source and use `--with-openssl=DIR`.

#### Missing zlib
```text
*** WARNING: building of the zlib extension module failed
```

**Fix**: Install `zlib1g-dev` (Debian) or `zlib-devel` (Fedora).

#### Missing Tcl/Tk
```text
*** WARNING: building of the _tkinter extension module failed
```

**Fix**: Install `tk-dev` (Debian) or `tk-devel` (Fedora). Use `--with-tcltk-includes` and `--with-tcltk-libs` if installed in non-standard locations.

#### Missing libzstd
```text
*** WARNING: building of the _zstd extension module failed
```

**Fix**: Install `libzstd-dev` (Debian) or `libzstd-devel` (Fedora). Zstandard is a newer dependency.

> [!TIP]
> Run `make` and check the summary at the end. It lists all modules that failed to build and why. The interpreter itself will still build — only the affected extension modules are skipped.

#### `configure` failures after modifying `configure.ac`
```text
configure: error: cannot find install-sh, install.sh, or shtool
```

**Fix**: Regenerate configure:
```bash
autoreconf -ivf
```

Requires `autoconf` 2.71+.

#### PGO build failure (insufficient test coverage)
```text
error: no profile data available for file 'xxx'
```

**Fix**: Ensure the PGO training run completed successfully. Check for test failures during the profiling phase.

#### Cross-compilation `--with-build-python` version mismatch
```text
Fatal Python error: init_importlib: Failed to import the site module
```

**Fix**: The build Python must be the exact same version as the target. Build a native Python first, then use it for cross-compilation.

### A.2 Runtime Issues

#### Segfault in C extension
```bash
# Build with debug and address sanitizer
./configure --with-pydebug --with-address-sanitizer
make -j$(nproc)

# Run with GDB helpers
gdb -ex "source Tools/gdb/libpython.py" -ex run --args ./python script.py

# In GDB:
(gdb) py-bt     # Python-level traceback
(gdb) py-locals  # Python local variables
```

#### Reference leak detection
```bash
# Hunt for reference leaks
./python -m test -R 3:5 test_mymodule
```

Output:
```text
test_mymodule leaked [1, 1, 1] references, sum=3
test_mymodule leaked [56, 56, 56] memory blocks, sum=168
```

#### Hash randomization issues
```bash
# Reproduce with fixed seed
PYTHONHASHSEED=0 python script.py

# Check current seed
python -c "import sys; print(sys.hash_info)"
```

#### Import errors after installation
```text
ModuleNotFoundError: No module named '_ssl'
```

**Fix**: The C extension failed to build (missing dependency). Check `make` output for warnings. Rebuild after installing the missing library.

#### Free-threading crashes
```text
Fatal Python error: _Py_qsbr_reserve: ...
```

**Fix**: Known issue (#148953). Free-threading is experimental — test thoroughly before deploying. Consider running with `PYTHON_GIL=1` to isolate whether the crash is GIL-related.

#### `dataclass(slots=True)` crash
```text
TypeError: type.__new__() takes exactly 3 arguments (0 given)
```

**Fix**: Known issue (#148947) where `dataclass(slots=True)` crashes with an empty `__class__` cell. Workaround: avoid `slots=True` if the class has complex closure interactions.

#### `dataclass` with existing `__init__`
```text
TypeError: Cannot overwrite attribute __init__
```

**Fix**: Known issue (#148941). Use `@dataclass(init=False)` if you provide your own `__init__`.

#### `pkgutil.walk_packages(None)` importing test modules
```text
Unexpected imports triggered by pkgutil.walk_packages(None)
```

**Fix**: Known issue (#131684). Pass explicit paths instead of `None`:
```python
import pkgutil, sys
pkgutil.walk_packages(sys.path)  # Still problematic — filter test paths
```

> [!CAUTION]
> `pkgutil.walk_packages(None)` will import test modules, which may have side effects including executing test code. Always specify explicit paths.

#### Integer string conversion limit
```text
ValueError: Exceeds the limit (4300 digits) for integer string conversion
```

**Fix**: Increase the limit if needed:
```bash
PYTHONINTMAXSTRDIGITS=0 python script.py  # Disable limit
```
Or in code:
```python
import sys
sys.set_int_max_str_digits(10000)
```

### A.3 Windows-Specific Issues

#### External dependency download failures
```text
Error downloading openssl-...
```

**Fix**: Set `EXTERNALS_DIR` to a writable location, or manually download dependencies:
```bash
python PCbuild/get_external.py --all
```

#### Visual Studio version mismatch
```text
error MSB8020: The build tools for v143 cannot be found
```

**Fix**: Ensure the correct Visual Studio workload ("Desktop development with C++") is installed with the matching platform toolset.

#### Debug/Release DLL mismatch
```text
ImportError: DLL load failed while importing _ssl
```

**Fix**: Ensure debug Python (`python_d.exe`) loads debug extensions (`_ssl_d.pyd`) and release Python loads release extensions. Don't mix.

### A.4 macOS-Specific Issues

#### Framework build + Homebrew OpenSSL
```text
Could not build the ssl module! ... openssl not found
```

**Fix**:
```bash
./configure --enable-framework \
    --with-openssl=$(brew --prefix openssl@3)
```

#### Universal binary build failures
```text
lipo: can't figure out the architecture type of ...
```

**Fix**: Ensure all dependencies (OpenSSL, Tcl/Tk, etc.) are built as universal binaries matching your `--with-universal-archs` setting.

### A.5 Test Suite Issues

#### Flaky tests
```bash
# Re-run failed tests
./python -m test -w test_asyncio

# Run with randomized order to find ordering dependencies
./python -m test --randomize

# Run forever until failure
./python -m test -F test_suspect
```

#### Network tests skipped
```text
test_urllib skipped -- Use of the 'network' resource not enabled
```

**Fix**:
```bash
./python -m test -u network test_urllib
```

#### GUI tests skipped
```text
test_tkinter skipped -- No module named '_tkinter'
```

**Fix**: Install Tcl/Tk development headers and rebuild. On headless systems, use `xvfb-run`:
```bash
xvfb-run ./python -m test -u gui test_tkinter
```

> [!NOTE]
> Many test modules require specific system resources. Use `./python -m test -u all` to enable all resource-intensive tests, but expect some platform-specific failures on constrained environments.

#### Test timeout
```text
Timeout (900.0 seconds) for test_xxx
```

**Fix**: Increase timeout:
```bash
./python -m test --timeout 1800 test_xxx
```

#### Reference leak false positives
```text
test_xxx leaked [2, 0, 0] references
```

**Fix**: Some reference count changes are expected (e.g., warming up caches). Use more warmup iterations:
```bash
./python -m test -R 10:5 test_xxx
```

### A.6 Documentation Build Issues

#### Sphinx version mismatch
```text
sphinx.errors.VersionRequirementError: ...
```

**Fix**: Recreate the venv:
```bash
cd Doc
make clean
rm -rf venv
make venv
```

#### Missing `blurb` or theme
```text
ModuleNotFoundError: No module named 'blurb'
```

**Fix**: The venv should install all dependencies. If not:
```bash
Doc/venv/bin/pip install -r Doc/requirements.txt
```

### A.7 Emscripten / WASI Issues

#### Emscripten SDK not found
```text
emcc: command not found
```

**Fix**: Activate the Emscripten SDK:
```bash
source /path/to/emsdk/emsdk_env.sh
```
Or set `EMSDK_CACHE`:
```bash
export EMSDK_CACHE=/path/to/emsdk/upstream/emscripten/cache
```

#### WASI SDK path
```text
wasi-sdk not found
```

**Fix**: Set the path:
```bash
export WASI_SDK_PATH=/opt/wasi-sdk
```

### A.8 Performance Issues

#### Slow startup
```bash
# Profile import times
PYTHONPROFILEIMPORTTIME=1 python -c "pass"
```

**Fix**: Use `python -S` to skip site-packages import, or `python -I` for fully isolated startup.

#### Memory growth
```bash
# Track allocations
PYTHONTRACEMALLOC=25 python script.py
```

```python
import tracemalloc
tracemalloc.start(25)
# ... run workload ...
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics('lineno')[:10]:
    print(stat)
```

> [!TIP]
> For production memory profiling, the new `profiling.sampling` module provides lower overhead than `tracemalloc`. Use `tracemalloc` for precise allocation tracking during development, and `profiling.sampling` for production sampling.

---

## Appendix B: Quick Reference Tables

### B.1 Make Targets

| Target | Description |
|--------|-------------|
| `make` | Build interpreter + modules |
| `make -j$(nproc)` | Parallel build |
| `make test` | Run test suite |
| `make install` | Install (creates `python3` symlink) |
| `make altinstall` | Install (versioned binary only) |
| `make clean` | Remove build artifacts |
| `make clobber` | Remove all generated files |
| `make distclean` | Full clean including configure output |
| `make clinic` | Run Argument Clinic |
| `make regen-all` | Regenerate all generated code |
| `make profile-opt` | PGO-optimized build |
| `make bolt-opt` | BOLT post-link optimization |
| `make tags` | Generate ctags |
| `make smelly` | Check exported symbols |
| `make sharedmods` | Build shared extension modules |
| `make frameworkinstall` | macOS framework install |

### B.2 python3.15-config Flags

```bash
python3.15-config --cflags      # Compiler flags
python3.15-config --ldflags     # Linker flags
python3.15-config --includes    # Include paths
python3.15-config --libs        # Libraries
python3.15-config --prefix      # Installation prefix
python3.15-config --embed       # Embedding flags
python3.15-config --configdir   # Config directory
```

### B.3 Special Builds

| Build Type | Configure Flags |
|-----------|----------------|
| Debug | `--with-pydebug` |
| Release (optimized) | `--enable-optimizations` |
| Free-threaded | `--disable-gil` |
| JIT-enabled | `--enable-experimental-jit` |
| Shared library | `--enable-shared` |
| Static library | `--enable-static` |
| Framework (macOS) | `--enable-framework` |
| ASAN | `--with-address-sanitizer` |
| MSAN | `--with-memory-sanitizer` |
| UBSAN | `--with-undefined-behavior-sanitizer` |
| TSAN | `--with-thread-sanitizer` |
| Valgrind | `--with-valgrind` |
| DTrace | `--with-dtrace` |
| Stats | `--enable-pystats` |
| Trace refs | `--with-trace-refs` |

### B.4 Key Source Files by Function

| Function | Primary Files |
|----------|--------------|
| Main entry point | `Programs/main.c`, `Modules/main.c` |
| Tokenizer | `Parser/tokenizer/tokenize.c`, `Parser/lexer/lexer.c` |
| Parser | `Parser/parser.c` (generated), `Grammar/python.gram` |
| AST | `Parser/Python.asdl`, `Python/ast.c`, `Python/ast_opt.c` |
| Compiler | `Python/compile.c`, `Python/codegen.c`, `Python/flowgraph.c` |
| Symbol table | `Python/symtable.c` |
| Eval loop | `Python/ceval.c`, `Python/generated_cases.c.h` |
| Bytecode defs | `Python/bytecodes.c` |
| Optimizer | `Python/optimizer.c`, `Python/optimizer_analysis.c` |
| JIT | `Python/jit.c`, `Tools/jit/build.py` |
| GC | `Python/gc.c` |
| Import | `Python/import.c`, `Lib/importlib/` |
| Memory | `Objects/obmalloc.c`, `Objects/mimalloc/` |
| Object header | `Include/object.h`, `Include/internal/pycore_object.h` |
| Type system | `Objects/typeobject.c` |
| Dict | `Objects/dictobject.c` |
| Str | `Objects/unicodeobject.c` |
| Int | `Objects/longobject.c` |
| List | `Objects/listobject.c` |
| Thread state | `Include/pystate.h`, `Include/internal/pycore_pystate.h` |
| Free-threading | `Include/internal/pycore_qsbr.h`, `Python/ceval_gil.c` |
| C API (stable) | `Include/*.h` |
| C API (internal) | `Include/internal/*.h` |
| C API (cpython) | `Include/cpython/*.h` |

### B.5 File Counts by Top-Level Directory

| Directory | Files | Chars | Priority |
|-----------|-------|-------|----------|
| `Python/` | 113 | 4.4M | HIGH |
| `Objects/` | 51 | 3.8M | HIGH |
| `Modules/` | 115 | 6.4M | HIGH |
| `Modules/clinic/` | 79 | 2.2M | HIGH |
| `Include/` | 77 | 361K | HIGH |
| `Include/internal/` | 144 | 1.6M | HIGH |
| `Include/cpython/` | 62 | 304K | HIGH |
| `Lib/` | 500+ | 10M+ | HIGH |
| `Parser/` | 24 | 367K | HIGH |
| `Programs/` | 8 | 90K | HIGH |
| `Tools/` | 100+ | 1M+ | HIGH |
| `PC/` | 24 | 407K | HIGH |
| `Mac/` | 15+ | 100K+ | HIGH |
| `Platforms/` | 14 | 133K | HIGH |
| `Doc/` | Many | Large | LOW |
| `Misc/` | 4 | 164K | LOW |

---

_Generated May 2026 from python/cpython at 49918f5. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._