# Qlib Admin & Developer Guide

This guide is for the engineers, architects, and researchers who deploy Qlib, configure it for production research, extend its components, and reason about its internals. It assumes familiarity with the day-to-day usage covered in the User Guide and focuses on architecture, the full configuration surface, security posture, operations, and the code-level structure needed to extend or replicate the platform.

> [!NOTE]
> **Package identity:** PyPI distribution `pyqlib`; import name `qlib`. Source is a `src`-less flat layout rooted at `qlib/`. The platform is pure Python with two compiled Cython extensions (`qlib.data._libs.rolling`, `qlib.data._libs.expanding`) built from `setup.py`.

---

## 1. Executive Summary

Qlib is an AI-oriented quantitative investment platform that covers the full research chain: data processing, feature engineering, model training, signal generation, portfolio construction, back-testing, and analysis. It is architected as a stack of **loosely-coupled components**, each usable stand-alone, wired together either declaratively (a single workflow YAML executed by `qrun`) or imperatively (Python "workflow by code").

| Attribute | Value |
|-----------|-------|
| Language | Python 3.8–3.12 (+ Cython extensions) |
| Distribution | `pyqlib` on PyPI; Docker image `pyqlib/qlib_image_stable:stable` |
| Experiment backend | MLflow (wrapped), default URI `file:./mlruns` |
| Optional services | Redis (caches), MongoDB (task management) |
| Data format | Custom compact binary, optimized for array assembly |
| Deployment modes | Offline (local data) / Online (shared data server, Qlib-Server) |
| License | MIT (Microsoft Open Source) |

**Operational stance:** Qlib is a *library and research framework*, not a long-running service. There is no API server, no authentication layer, and no network listener in the core. Its "production" footprint is the data directory, the MLflow tracking store, and optionally Redis/MongoDB. Security and operations therefore center on **data integrity, dependency management, and reproducibility** rather than on hardening a service surface.

---

## 2. Architecture

### 2.1 Component layers

```text
┌─────────────────────────────────────────────────────────────┐
│ qlib.init() → QlibConfig (global C) → register providers/ops │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
┌───────────────┐   Data Layer  (qlib/data)
│ Providers     │   LocalProvider → Calendar/Instrument/Feature/PIT/Expression/Dataset providers
│ Expression    │   $field raw access + operator graph (Ref, Mean, ...) compiled via _libs
│ Caches        │   ExpressionCache, DatasetCache (Disk* variants depend on Redis)
└───────────────┘
        │
        ▼
┌───────────────┐   Dataset Layer  (qlib/data/dataset)
│ DataHandler   │   Alpha158 / Alpha360 (qlib/contrib/data/handler)
│ DatasetH      │   train/valid/test segmentation, DataHandlerLP processing
└───────────────┘
        │
        ▼
┌───────────────┐   Model Layer  (qlib/model, qlib/contrib/model)
│ Model.fit     │   LGBModel + ~20 pytorch/tensorflow SOTA models
│ Model.predict │   Trainer / TrainerR / TrainerRM (MongoDB TaskManager)
└───────────────┘
        │
        ▼
┌───────────────┐   Workflow Layer  (qlib/workflow)
│ R (recorder)  │   QlibRecorder → MLflowExpManager → MLflowExperiment → MLflowRecorder
│ RecordTemp    │   SignalRecord → SigAnaRecord → PortAnaRecord
└───────────────┘
        │
        ▼
┌───────────────┐   Strategy + Execution  (qlib/strategy, qlib/backtest)
│ BaseStrategy  │   TopkDropoutStrategy, RL strategies
│ Executor      │   SimulatorExecutor, NestedExecutor, Exchange, Position, Account
└───────────────┘
        │
        ▼
┌───────────────┐   Analysis  (qlib/contrib/eva, contrib/evaluate, contrib/report)
│ IC / risk     │   calc_ic, risk_analysis, indicator_analysis
└───────────────┘
```

Every box is independently importable. The glue is **config-driven instantiation**: throughout Qlib, components are described by `{class, module_path, kwargs}` dicts and built at runtime by `init_instance_by_config` (`qlib/utils`). This is why a workflow YAML can swap any model, dataset, or strategy without code changes.

### 2.2 Initialization and the global config object

`qlib.init()` (`qlib/__init__.py`) is the single entry point that activates the framework:

1. Resets the global config `C` (a `QlibConfig`) from a mode template (`client` or `server`).
2. Applies region defaults (`cn`/`us`/`tw`) and any keyword overrides.
3. Resolves and (optionally) mounts the data path — including NFS auto-mount logic for shared deployments (`_mount_nfs_uri`).
4. Calls `C.register()`, which registers all expression operators (`register_all_ops`), data wrappers (`register_all_wrappers`), and constructs the `QlibRecorder` (`R`) backed by the configured experiment manager.

```python
import qlib
from qlib.constant import REG_CN
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)
```

Alternative initializers:

| Function | Use case |
|----------|----------|
| `qlib.init(default_conf="client", **kwargs)` | Standard explicit init. |
| `qlib.init_from_yaml_conf(conf_path, **kwargs)` | Init from a YAML config file. |
| `qlib.auto_init(**kwargs)` | Walk up the directory tree to find a project `config.yaml` and init from it; supports `conf_type: origin` (standalone) and `conf_type: ref` (reference a shared config + project overrides). Used by `python -m qlib.contrib.rolling`. |

> [!NOTE]
> The config object `C` is a singleton (`qlib/config.py`, `C = QlibConfig(_default_config)`). `qlib.init()` is **not** safe to call twice while a recorder is active — the `RecorderWrapper` raises `RecorderInitializationError` to prevent silently relocating experiment storage.

### 2.3 Data path management

`QlibConfig.DataPathManager` resolves `provider_uri` (and `mount_path`) into concrete data URIs, distinguishing **local** paths from **NFS** paths by regex (`get_uri_type`). `provider_uri` may be a string or a per-frequency dict:

```yaml
qlib_init:
    provider_uri:
        day:  "~/.qlib/qlib_data/cn_data"
        1min: "~/.qlib/qlib_data/cn_data_1min"
```

For NFS deployments, Qlib can auto-mount (`auto_mount: True`) on Windows (`mount -o anon`) or Linux/Unix (`mount.nfs`, requires `nfs-common` and sudo). This is the mechanism behind shared, multi-client data servers.

---

## 3. System Requirements

| Requirement | Detail |
|-------------|--------|
| Python | 3.8–3.12 (model-specific exceptions, e.g. TFT needs 3.6–3.7) |
| Build toolchain | C/C++ compiler + Cython for the `_libs` extensions; `cmake` listed in docs requirements |
| Core libraries | `numpy`, `scipy`, `pandas`, `scikit-learn` |
| Experiment store | `mlflow` (always), `filelock` for concurrency |
| Config | `ruamel.yaml`, `pydantic-settings`, `jinja2` |
| Optional: Redis | For `DiskExpressionCache` / `DiskDatasetCache` (server mode) |
| Optional: MongoDB | For `TrainerRM` / `TaskManager` parallel task orchestration |
| GPU | Required by some neural models (TFT requires CUDA 10.0) |

> [!IMPORTANT]
> Each benchmark model under `examples/benchmarks/` pins its own dependency versions in a local `requirements.txt` (often specific `torch`/`tensorflow`/`lightgbm` versions). Treat these as authoritative per-model environments — `run_all_model.py` enforces this by building an isolated virtualenv per model.

---

## 4. Configuration Mastery

There are **three distinct configuration surfaces** in Qlib. Conflating them is the most common operational mistake.

1. **Runtime config** — the `qlib.init()` keyword arguments, layered over the `_default_config` dict in `qlib/config.py`.
2. **Environment settings** — the `QSettings` (pydantic-settings) object, prefix `QLIB_`.
3. **Workflow config** — the per-experiment YAML consumed by `qrun` (model/dataset/strategy/backtest).

### 4.1 Runtime config (`_default_config`)

These are the keys you can pass to `qlib.init()` or set in `qlib_init:`. Reproduced from `qlib/config.py`:

| Key | Default | Purpose |
|-----|---------|---------|
| `calendar_provider` | `LocalCalendarProvider` | Trading-calendar backend. |
| `instrument_provider` | `LocalInstrumentProvider` | Instrument-universe backend. |
| `feature_provider` | `LocalFeatureProvider` | Raw-feature backend. |
| `pit_provider` | `LocalPITProvider` | Point-in-time fundamentals backend. |
| `expression_provider` | `LocalExpressionProvider` | Expression-evaluation backend. |
| `dataset_provider` | `LocalDatasetProvider` | Dataset assembly backend. |
| `provider` | `LocalProvider` | Top-level data provider. |
| `provider_uri` | `""` | Data directory (string or per-freq dict). |
| `expression_cache` | `None` | Expression cache class (e.g. `DiskExpressionCache`). |
| `dataset_cache` | `None` (client) / `DiskDatasetCache` (server) | Dataset cache class. |
| `local_cache_path` | `~/.cache/qlib_simple_cache` | Simple on-disk cache dir. |
| `kernels` | `NUM_USABLE_CPU` (`cpu_count - 2`) | Parallelism; may be an `int` or a `freq → int` callable. |
| `maxtasksperchild` | `None` | Tasks per worker process (use `1` for high-freq). |
| `joblib_backend` | `multiprocessing` | Parallel backend. |
| `default_disk_cache` | `1` | `0` skip / `1` use. |
| `mem_cache_size_limit` | `500` | In-memory cache size. |
| `mem_cache_limit_type` | `length` | Cache eviction metric. |
| `mem_cache_expire` | `3600` | Memory cache TTL (seconds). |
| `dataset_cache_dir_name` | `dataset_cache` | Cache subdir. |
| `features_cache_dir_name` | `features_cache` | Cache subdir. |
| `redis_host` | `127.0.0.1` | Redis host (cache locking). |
| `redis_port` | `6379` | Redis port. |
| `redis_task_db` | `1` | Redis DB index. |
| `redis_password` | `None` | Redis auth. |
| `logging_level` | `logging.INFO` | Global log level. |
| `logging_config` | (dict) | Full `logging.dictConfig` (formatters, filters, handlers). |
| `exp_manager` | `MLflowExpManager` | Experiment-manager class + `uri`/`default_exp_name`. |
| `mongo` | `{task_url: mongodb://localhost:27017/, task_db_name: default_task_db}` | MongoDB for TaskManager. |
| `pit_record_type` / `pit_record_nan` | (dicts) | Binary record layout for PIT data. |
| `min_data_shift` | `0` | Minute-data market-time shift for high-freq backtest. |

Mode templates (`MODE_CONF`) differ in cache defaults:

- **`client`** (default): caches disabled by default, `region: cn`, `auto_mount: False`, `timeout: 100`, `custom_ops: []`.
- **`server`**: `DiskExpressionCache` + `DiskDatasetCache` enabled (these depend on Redis).

> [!WARNING]
> `DiskExpressionCache` and `DiskDatasetCache` require a reachable Redis (`DEPENDENCY_REDIS_CACHE`). At init, if Redis is unreachable, Qlib **silently disables** these caches and logs a warning — it will not fail. Monitor for this warning in production; a disabled cache can silently 10–25× your data-loading time.

### 4.2 Region defaults

`set_region()` applies market-specific trading parameters:

| Region | `trade_unit` | `limit_threshold` | `deal_price` |
|--------|:------------:|:-----------------:|:------------:|
| `REG_CN` (`cn`) | 100 | 0.095 | close |
| `REG_US` (`us`) | 1 | `None` | close |
| `REG_TW` (`tw`) | 1000 | 0.1 | close |

### 4.3 Environment settings (`QLIB_` prefix)

`QSettings` (`qlib/config.py`) reads from the environment with prefix `QLIB_` and nested delimiter `_`:

| Env var | Maps to | Default |
|---------|---------|---------|
| `QLIB_PROVIDER_URI` | `provider_uri` | `~/.qlib/qlib_data/cn_data` |
| `QLIB_MLFLOW_URI` | `exp_manager.kwargs.uri` | `file:<cwd>/mlruns` |
| `QLIB_MLFLOW_DEFAULT_EXP_NAME` | default experiment name | `Experiment` |

> [!NOTE]
> Settings precedence is: **explicit `qlib.init()` kwargs** > **`QLIB_*` environment** > **built-in defaults**. Use environment variables for deployment-level config (data location, MLflow store) and `qlib.init()` kwargs for per-run overrides.

### 4.4 Workflow config schema

The `qrun` YAML is the primary *research* configuration. Its sections (`qlib_init`, `market`, `benchmark`, `data_handler_config`, `port_analysis_config`, `task.model`, `task.dataset`, `task.record`) are documented in the User Guide. Two operational features worth highlighting:

- **`BASE_CONFIG_PATH`** — inherit from a base config and override only changed fields (`update_config` merge in `qlib/cli/run.py`). Enables a base-template + experiment-delta pattern across a model sweep.
- **Jinja2 templating** — configs are rendered as Jinja2 before parsing; undeclared `{{ VAR }}` variables are filled from matching environment variables (`render_template`). Enables parameter sweeps from a single file driven by env vars.

The `qrun` workflow handler (`workflow()` in `qlib/cli/run.py`) signature: `workflow(config_path, experiment_name="workflow", uri_folder="mlruns")`. If `exp_manager` is present under `qlib_init`, it is used directly; otherwise the MLflow URI is set to `file:<cwd>/<uri_folder>`.

---

## 5. Security

Qlib's core ships **no network service, no auth, and no secret management** — its threat surface is the data pipeline and the dependency graph, not an exposed endpoint. Security review should focus on the following.

### 5.1 Data provenance and integrity

- Bundled/collected market data is sourced from third parties (Yahoo Finance, Coingecko, baostock, East Money). The project explicitly warns this data "might not be perfect" and may contain anomalies even after price adjustment. **Do not trust downloaded data blindly** — run `scripts/check_data_health.py` and consider the cross-validated `crowd_source` data to reduce survivorship bias.
- The community data source (`chenditc/investment_data`) is a third-party GitHub release fetched over `wget`. Verify checksums/releases before ingestion in a controlled environment.

### 5.2 Code-execution surfaces

> [!CAUTION]
> Qlib's config system is **powerful by design and trusting by default**. Several mechanisms execute or import arbitrary code from configuration:
> - `init_instance_by_config` imports any `module_path` and instantiates any `class` named in a config — including loading a **pickle object** from a `file://` path (`InstConf` in `qlib/typehint.py`).
> - `R.save_objects` / `load_object` and model serialization use Python pickle. **Never load a recorder artifact or a config from an untrusted source** — pickle deserialization is arbitrary code execution.
> - The `sys` section of a workflow config appends arbitrary paths to `sys.path` (`sys_config` in `qlib/cli/run.py`).
> - Jinja2 config rendering pulls values from the process environment.
>
> Treat workflow YAML files and saved artifacts as **trusted code**, not data. Source them only from your own pipeline.

### 5.3 Network and credentials

- **NFS auto-mount** (`_mount_nfs_uri`) shells out to `sudo mount.nfs` (Linux) or `mount -o anon` (Windows). In shared deployments this requires elevated privileges; scope it carefully and prefer pre-mounting volumes over `auto_mount: True`.
- **Redis** (`redis_host`/`redis_port`/`redis_password`) and **MongoDB** (`mongo.task_url`) connection details live in plain config. In production, supply credentials via environment/secret injection and bind these services to private networks — they are unauthenticated by default in the example configs (`mongodb://localhost:27017/`, `redis 127.0.0.1`).
- Data collectors make outbound HTTP requests to financial data providers; some (Yahoo) are geo-blocked from China.

### 5.4 Recommended hardening

- [ ] Run model sweeps (`run_all_model.py`) in isolated virtualenvs (the default) and ideally isolated containers.
- [ ] Pin and audit per-model `requirements.txt` — many pin old `torch`/`tensorflow`/`numpy` with known CVEs.
- [ ] Never deserialize third-party recorder artifacts or configs.
- [ ] Bind Redis/MongoDB to localhost or a private subnet; set `redis_password` and Mongo auth.
- [ ] Validate downloaded data with the health checker before training.

---

## 6. Operations

### 6.1 Deployment modes

| Mode | Data location | Caching | Use case |
|------|---------------|---------|----------|
| **Offline** (default) | Local disk per machine | Optional simple/disk caches | Single-researcher, reproducible |
| **Online** | Shared Qlib-Server, NFS-mounted | Disk caches shared across clients | Team, higher cache-hit rate, less disk |

Online mode is documented under the separate [Qlib-Server](https://qlib-server.readthedocs.io/) project and supports one-click Azure deployment. In online mode, multiple clients share data and cache, improving cache-hit rate and reducing per-client disk usage.

### 6.2 The experiment store (MLflow)

All run state — params, metrics, tags, and artifacts (models, predictions, reports) — is persisted to the MLflow tracking URI (default `file:./mlruns`). Operationally:

- The URI can be a local `file:` path or a remote MLflow tracking server. **For file URIs, use absolute paths** (the backend rejects `~`-style strings via `set_uri`).
- Concurrency: experiment creation is guarded by a `filelock` on file URIs (`_get_or_create_exp`) because MLflow does not lock multi-process experiment creation. For non-file schemes, Qlib double-checks via `ExpAlreadyExistError`.
- MLflow version compatibility is handled (`search_experiments` for v2+, `list_experiments` for v1).

```python
from qlib.workflow import R
# Audit all runs of an experiment as a DataFrame
df = R.search_records([exp_id], order_by=["metrics.ic DESC"])
# Inspect a specific run's artifacts
rec = R.get_recorder(recorder_id=rid, experiment_name="my_exp")
```

> [!NOTE]
> MLflow's `search_runs` caps results at 50,000 (`UNLIMITED = 50000` in `qlib/workflow/exp.py`). For very large experiment stores, page or filter with `filter_string`.

### 6.3 Parallel & distributed training

Two trainer families (`qlib/model/trainer.py`):

| Trainer | Backend | Behavior |
|---------|---------|----------|
| `TrainerR` | Recorder only | Linear, in-process training of a task list. |
| `TrainerRM` | Recorder + MongoDB `TaskManager` | Tasks pushed to MongoDB; multiple `worker()` processes/machines pull and train in parallel. |
| `DelayTrainerR` / `DelayTrainerRM` | — | Split into a cheap `train` (prepare) step and a heavy `end_train` (fit) step — useful for "prepare on CPU VM, fit on GPU VMs". |

For `TrainerRM`, run the orchestrator (`train`) on one node and `worker()` on others, all sharing one `task_pool` (MongoDB collection). This is Qlib's mechanism for scaling a model sweep across machines.

### 6.4 Data lifecycle operations

| Task | Command |
|------|---------|
| Download data | `python scripts/get_data.py qlib_data --target_dir <dir> --region <cn\|us\|in>` |
| Health check | `python scripts/check_data_health.py check_data --qlib_dir <dir>` |
| Incremental daily update | `python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <dir>` |
| Rebuild from raw | collect → `normalize_data` → `python scripts/dump_bin.py dump_all` |
| Point-in-time dump | `python scripts/dump_pit.py dump --data_path <dir> --qlib_dir <dir> --interval quarterly` |
| Automated cron update | `* * * * 1-5 python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <dir>` |

> [!TIP]
> `dump_bin.py dump_all` parallelizes with `--max_workers` (default 16) and selects fields via `--include_fields` / `--exclude_fields`. For high-frequency data, set `maxtasksperchild: 1` in the runtime config to bound per-process memory.

### 6.5 Logging

Logging is fully configurable through the `logging_config` runtime key (a `logging.dictConfig`). Notable built-ins (`qlib/log.py`):

- A custom `LogFilter` suppresses noisy "data not found" warnings by regex.
- `set_global_logger_level(level)` and the `set_global_logger_level_cm` context manager adjust handler levels at runtime.
- `TimeInspector` (`logt` context manager) instruments code blocks for timing — useful for profiling data-loading bottlenecks.

---

## 7. Repository Structure

```text
qlib/
├── __init__.py            init(), auto_init(), NFS mount logic
├── config.py              QlibConfig, _default_config, MODE_CONF, region config, QSettings
├── constant.py            REG_CN/US/TW, EPS, INF
├── log.py                 logging, LogFilter, TimeInspector
├── typehint.py            InstConf / InstDictConf (config-as-instance contract)
├── cli/
│   ├── run.py             `qrun` workflow runner (fire-based)
│   └── data.py            `python -m qlib.cli.data` data downloader
├── data/                  Data layer: providers, expression engine, caches, _libs (Cython)
│   └── dataset/           DatasetH, DataHandler(LP), processors, samplers
├── model/                 base.py (Model/ModelFT), trainer.py (TrainerR/RM)
├── workflow/              R recorder, ExpManager, RecordTemp, task/ (TaskManager)
├── strategy/              BaseStrategy, RLStrategy, RLIntStrategy
├── backtest/              Exchange, Executor, Position, Account, decision
├── rl/                    Reinforcement-learning framework (interpreters, trainer, order_execution)
├── contrib/               Batteries-included: models, data handlers (Alpha158/360),
│                          strategies (TopkDropout), evaluation, reporting, rolling, tuner
├── utils/                 init_instance_by_config, serialization, parallel, time
└── tests/                 GetData, canned configs (CSI300_GBDT_TASK, CSI300_BENCH)

scripts/                   get_data.py, dump_bin.py, dump_pit.py, check_data_health.py,
└── data_collector/        per-source collectors (yahoo, cn_index, us_index, crypto, fund, pit, ...)

examples/                  workflow_by_code.py, run_all_model.py, benchmarks/<MODEL>/,
                           benchmarks_dynamic/, rl_order_execution/, nested_decision_execution/,
                           highfreq/, tutorial/

docs/                      Sphinx documentation (readthedocs)
```

### 7.1 Module reference (HIGH-priority)

| Module | Lines (≈) | Responsibility |
|--------|-----------|----------------|
| `qlib/data` | 173K chars | Providers, expression engine, calendar/instrument/feature access, caches. |
| `qlib/utils` | 116K chars | `init_instance_by_config`, serialization, parallelism, time utilities. |
| `qlib/workflow` | 106K chars | Recorder/experiment management (MLflow wrapper), record templates. |
| `qlib/data/dataset` | 98K chars | `DatasetH`, `DataHandlerLP`, processors, weight/reweighter, samplers. |
| `qlib/contrib/data` | 76K chars | `Alpha158`/`Alpha360` handlers and feature definitions. |
| `qlib/workflow/task` | 51K chars | `TaskManager` (MongoDB), task generation, collection. |
| `qlib/rl` (+ utils) | 62K chars | RL interpreters, trainer, order-execution environment. |
| `qlib/backtest` | 227K chars | Exchange, executors (incl. `NestedExecutor`), position, account, decisions. |
| `qlib/model` | 27K chars | `Model`/`ModelFT` base, `Trainer` family. |
| `qlib/strategy` | 11K chars | `BaseStrategy`, RL strategy bases. |
| `qlib/contrib/rolling` | 28K chars | Rolling-retrain driver (`python -m qlib.contrib.rolling`). |
| `qlib/contrib/tuner` | 16K chars | Hyperparameter tuning launcher. |

---

## 8. Internals & Extension Points

### 8.1 Config-driven instantiation (the core idiom)

`init_instance_by_config(config, accept_types=..., default_module=...)` is the universal factory. A config can be:

- A dict `{class, module_path, kwargs}` (the common case).
- A class object directly, with `kwargs`.
- A string: a class name, a dotted `a.b.c.ClassName`, or a `file://.../obj.pkl` pickle path.
- An already-constructed instance (passed through).

The `task` executor (`_exe_task` in `qlib/model/trainer.py`) shows the full lifecycle: instantiate model + dataset, `auto_filter_kwargs(model.fit)(dataset, reweighter=...)`, save the model and (config-only) dataset, fill the `<MODEL>`/`<DATASET>` placeholders via `fill_placeholder`, then run each `record` template's `generate()`.

### 8.2 Writing a custom model

Subclass `qlib.model.base.Model` and implement `fit` and `predict`:

```python
from qlib.model.base import Model
from qlib.data.dataset import Dataset
from qlib.data.dataset.handler import DataHandlerLP

class MyModel(Model):
    def fit(self, dataset: Dataset, reweighter=None):
        df_train, df_valid = dataset.prepare(
            ["train", "valid"], col_set=["feature", "label"], data_key=DataHandlerLP.DK_L
        )
        x_train, y_train = df_train["feature"], df_train["label"]
        # ... train self ...

    def predict(self, dataset: Dataset, segment="test"):
        x = dataset.prepare(segment, col_set="feature", data_key=DataHandlerLP.DK_I)
        return ...  # pandas.Series indexed by (datetime, instrument)
```

> [!NOTE]
> Learned attributes must **not** start with `_` so the model serializes cleanly (`Serializable` contract). For fine-tunable models, subclass `ModelFT` and add `finetune(dataset)`.

### 8.3 Writing a custom strategy

Subclass `qlib.strategy.base.BaseStrategy` and implement `generate_trade_decision(execute_result)`. The base provides `executor`, `trade_calendar`, `trade_position`, and `trade_exchange` via the shared `level_infra` / `common_infra` infrastructure objects. For RL strategies, subclass `RLStrategy` (carries a `policy`) or `RLIntStrategy` (adds `state_interpreter` / `action_interpreter` that translate between Qlib execution results and the RL env).

### 8.4 Custom expression operators

Pass `custom_ops` to `qlib.init()` — each element is either an `ExpressionOps` subclass or a `{class, module_path}` dict. They are registered by `register_all_ops` at init, extending the `$field`/operator expression language.

### 8.5 Record templates

Custom analysis = subclass `RecordTemp` (`qlib/workflow/record_temp.py`) and implement `generate()`. Templates declare `artifact_path` and an optional `depend_cls` so dependent records (e.g. `SigAnaRecord` depending on `SignalRecord`) can locate upstream artifacts. The built-in chain is `SignalRecord → SigAnaRecord → PortAnaRecord`.

---

## 9. Code Quality & Conventions

- **Consistent factory pattern.** Nearly every component is built through `init_instance_by_config`, giving a uniform, declarative, swap-anything design. This is the platform's defining architectural strength.
- **MLflow wrapper rationale.** `qlib/workflow/__init__.py` documents *why* Qlib wraps MLflow rather than using it directly: object-oriented recorder handles instead of bare `run_id`s, code-diff logging at run start, and `log_object`/`load_object` for Python objects. The cost is the up-front need to create an experiment before getting a recorder.
- **Abstract base + concrete impl** is used consistently (`Experiment`/`MLflowExperiment`, `ExpManager`/`MLflowExpManager`, `Trainer`/`TrainerR`/`TrainerRM`), keeping the backend swappable.
- **Pydantic settings** (`QSettings`) provide typed, env-overridable configuration.
- **Cython hot paths.** Rolling/expanding window operators are compiled C++ (`qlib/data/_libs/*.pyx`) for the expression engine — the source of Qlib's data-throughput advantage.

### 9.1 Performance characteristics

Qlib's own storage benchmark (14 features over ~800 stocks/day, 2007–2020) demonstrates the compact-binary + cache design:

| Storage | HDF5 | MySQL | MongoDB | InfluxDB | Qlib −E −D | Qlib +E −D | Qlib +E +D |
|---------|------|-------|---------|----------|-----------|-----------|-----------|
| 1 CPU (s) | 184.4 | 365.3 | 253.6 | 368.2 | 147.0 | 47.6 | **7.4** |
| 64 CPU (s) | — | — | — | — | 8.8 | **4.2** | — |

(`+E` = ExpressionCache, `+D` = DatasetCache.) The takeaway for operators: **enable both caches in any throughput-sensitive deployment**.

---

## 10. Known Issues & Compatibility

These are confirmed from the project's issue tracker and release notes — track them when pinning dependencies.

| Issue | Impact | Mitigation |
|-------|--------|------------|
| **LightGBM 4.0+ incompatibility** | `TypeError` when `early_stopping_rounds=None`; affects model training and DDG-DA. | Pin LightGBM 3.x (e.g. `3.1.0`) or use a patched Qlib build. |
| **DDG-DA chained failures** | Dynamic workflow fails on LightGBM 4.0+, unhashable-list in `InternalData.setup()`, and pandas indexing. | Use the example's pinned deps in an isolated env. |
| **`SeriesDFilter` redundant execution** | Filter pipe re-runs per `fields_group`, slowing data loading. | Reduce distinct field-group filters; enable caches. |
| **`FileCalendarStorage.index()` defect** | Reported calendar-storage bug. | Track upstream fix; validate calendar data. |
| **pandas 1.5 → 2.0 `group_keys` change** | A few scripts (`gen_training_orders.py`, TRA dataset, `tft.py`) may misbehave. | Qlib sets `group_key=False`; pin a compatible pandas. |
| **Missing benchmark index (`SH000300`)** | Backtest aborts if the benchmark isn't in the data set. | Ensure region/data alignment. |
| **Official dataset disabled** | Built-in downloader may return empty data. | Use community `chenditc/investment_data`. |

---

## 11. Potential Improvements

For maintainers and contributors, the following are visible technical-debt and enhancement areas (several flagged in-repo):

- **LightGBM 4.0+ support** across all GBDT models and the DDG-DA pipeline — the most impactful compatibility gap.
- **`run_all_model.py` portability** — currently Linux-only and cannot run the same model in parallel across iterations; both are noted as planned.
- **`SeriesDFilter` optimization** — deduplicate filter-pipe execution across field groups to fix the slow-loading path.
- **GeneralPtNN consolidation** — migrate existing per-architecture NN models onto the unified `GeneralPtNN` design so swapping a PyTorch module is the only change needed.
- **Data robustness** — the project itself recommends users supply their own high-quality data; the crowd-sourced/cross-validated pipeline reduces survivorship bias and could become the default.
- **Find rough edges** — the README explicitly suggests `rg 'TODO|FIXME' qlib` to surface incomplete implementations.

---

## 12. Contributing

- **Repo:** `https://github.com/microsoft/qlib` (MIT, Microsoft Open Source CoC). A **CLA** is required on first PR.
- **Dev install:** `pip install -e .[dev]`; see `docs/developer/code_standard_and_dev_guide.rst`.
- **Good entry points:** the `good first issue` label, doc fixes, new models (`examples/benchmarks/#contributing`), new datasets, and bug fixes.
- **Build docs:** `cd docs/ && make html` (requires `sphinx` + `sphinx_rtd_theme`).
- **CI reference:** if local install/examples fail, compare against `.github/workflows/test_qlib_from_source.yml`.

> [!TIP]
> When adding a model, follow the existing `examples/benchmarks/<MODEL>/` convention: a `workflow_config_*.yaml`, a `requirements.txt` with pinned deps, and a `README.md` citing the source paper. This keeps the model zoo reproducible and `run_all_model.py`-compatible.

---

## 13. Quick Reference

### Initialization

```python
import qlib
from qlib.constant import REG_CN
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)
```

### Key globals & factories

| Symbol | Module | Role |
|--------|--------|------|
| `C` | `qlib.config` | Global `QlibConfig` singleton. |
| `D` | `qlib.data` | Data access (calendar/instruments/features). |
| `R` | `qlib.workflow` | Experiment/recorder manager (MLflow wrapper). |
| `init_instance_by_config` | `qlib.utils` | Build any component from a config dict. |
| `Model` / `ModelFT` | `qlib.model.base` | Model base classes. |
| `TrainerR` / `TrainerRM` | `qlib.model.trainer` | Linear / MongoDB-parallel trainers. |
| `BaseStrategy` | `qlib.strategy.base` | Strategy base. |
| `SignalRecord` / `SigAnaRecord` / `PortAnaRecord` | `qlib.workflow.record_temp` | Analysis record templates. |

### Operational commands

| Command | Purpose |
|---------|---------|
| `qrun <config.yaml> [exp_name]` | Run a workflow. |
| `python scripts/get_data.py qlib_data --target_dir <dir> --region cn` | Download data. |
| `python scripts/check_data_health.py check_data --qlib_dir <dir>` | Validate data. |
| `python scripts/dump_bin.py dump_all --data_path <dir> --qlib_dir <dir> --freq day` | Build binary data. |
| `python run_all_model.py run <n>` | Benchmark sweep (Linux). |
| `python -m qlib.contrib.rolling base --conf_path <yaml> run` | Rolling retrain. |

### Service endpoints (optional, default insecure)

| Service | Default | Used by |
|---------|---------|---------|
| Redis | `127.0.0.1:6379`, db `1`, no password | Disk caches (server mode). |
| MongoDB | `mongodb://localhost:27017/`, db `default_task_db` | `TrainerRM` / `TaskManager`. |
| MLflow | `file:<cwd>/mlruns` | Experiment tracking. |

---

_Generated June 2026 from microsoft/qlib at d5379c5. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._