# Qlib User Guide

Qlib is an open-source, AI-oriented quantitative investment platform. It lets you take a machine-learning idea — from a tree model to a deep network or a reinforcement-learning agent — all the way through the full quant pipeline: data processing, model training, signal generation, portfolio construction, back-testing, and analysis. This guide shows you how to install Qlib, prepare data, run complete research workflows, and build your own strategies.

> [!NOTE]
> Qlib is distributed on PyPI under the name **`pyqlib`**, but you import it in Python as **`qlib`**. Throughout this guide, "install `pyqlib`" and "import `qlib`" refer to the same package.

---

## 1. Quick Start

This section gets you from nothing to a back-tested model in a few commands. It assumes a working Python 3.8–3.12 environment.

### 1.1 Install Qlib

```bash
pip install pyqlib
```

```text
Successfully installed pyqlib-<version>
```

### 1.2 Download market data

The example data set is China A-share daily data. Download it into the default data directory:

```bash
python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn
```

```text
[INFO] Downloading qlib_data to ~/.qlib/qlib_data/cn_data ...
[INFO] Data is downloaded and extracted.
```

> [!IMPORTANT]
> The official hosted dataset is **temporarily disabled** due to a stricter data-security policy. If the download above fails or returns empty data, use the community-maintained data set described in [Section 3.3](#33-community-data-source-recommended).

### 1.3 Run a complete workflow

Qlib ships ready-to-run model configurations. From the `examples/` directory, run a LightGBM workflow with the `qrun` command:

```bash
cd examples
qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
```

`qrun` builds the dataset, trains the model, generates predictions, runs a back-test, and prints a performance summary:

```text
'The following are analysis results of the excess return without cost.'
                       risk
mean               0.000708
std                0.005626
annualized_return  0.178316
information_ratio  1.996555
max_drawdown      -0.081806
'The following are analysis results of the excess return with cost.'
                       risk
mean               0.000512
std                0.005626
annualized_return  0.128982
information_ratio  1.444287
max_drawdown      -0.091078
```

> [!TIP]
> Run `qrun` from inside `examples/` (or any directory that does **not** contain a folder named `qlib`). Running from the repository root can cause Python to import the local source tree instead of the installed package.

That's the entire loop. The rest of this guide explains each piece so you can swap in your own models, data, and strategies.

---

## 2. Installation

### 2.1 Supported Python versions

| Python version | Install with `pip` | Install from source | Plotting |
|----------------|:-----------------:|:-------------------:|:--------:|
| 3.8  | ✅ | ✅ | ✅ |
| 3.9  | ✅ | ✅ | ✅ |
| 3.10 | ✅ | ✅ | ✅ |
| 3.11 | ✅ | ✅ | ✅ |
| 3.12 | ✅ | ✅ | ✅ |

### 2.2 Install methods

=== "pip (stable)"

    ```bash
    pip install pyqlib
    ```

    Installs the latest stable release. Recommended for most users.

=== "From source (latest dev)"

    ```bash
    pip install numpy
    pip install --upgrade cython
    git clone https://github.com/microsoft/qlib.git && cd qlib
    pip install .          # or: pip install -e .[dev] for development
    ```

    The `main` branch is under active development and may contain features not yet released to PyPI.

=== "Docker"

    ```bash
    docker pull pyqlib/qlib_image_stable:stable
    docker run -it --name qlib_container -v <local_dir>:/app pyqlib/qlib_image_stable:stable
    ```

    A self-contained environment with Qlib and its build dependencies preinstalled.

> [!TIP]
> **Conda is recommended** for managing your environment. Building Qlib outside of Conda sometimes fails because of missing C/C++ header files needed to compile Qlib's Cython extensions.

> [!WARNING]
> **Apple Silicon (M1/M2):** building the LightGBM wheel may fail due to a missing OpenMP dependency. Install it first with `brew install libomp`, then re-run `pip install .`.

### 2.3 Optional dependency groups

Some features require extra packages installed via "extras":

```bash
# Graphical report analysis (notebooks, plotting)
python -m pip install .[analysis]
```

> [!NOTE]
> Individual benchmark models under `examples/benchmarks/` declare their own `requirements.txt` (for example, specific `torch` or `tensorflow` versions). Install those before running a particular model. The TFT model, for instance, requires `tensorflow-gpu==1.15.0` and only supports Python 3.6–3.7.

---

## 3. Data Preparation

Qlib stores market data in a compact binary format optimized for fast scientific computation. You can download a ready-made data set or build your own from raw sources.

### 3.1 Download ready-made data

=== "As a module"

    ```bash
    # Daily (1d) China A-share data
    python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

    # 1-minute data
    python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min
    ```

=== "From the source script"

    ```bash
    # Daily (1d) China A-share data
    python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

    # 1-minute data
    python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min
    ```

Common parameters for the `qlib_data` downloader:

| Parameter | Meaning | Default |
|-----------|---------|---------|
| `--target_dir` | Where to save the data | `~/.qlib/qlib_data/cn_data` |
| `--region` | Market region: `cn`, `us`, `in` | `cn` |
| `--interval` | Bar interval: `1d` or `1min` | `1d` |
| `--version` | Data set version (`v1`/`v2`) | `v1` |
| `--delete_old` | Remove existing data in `target_dir` first | `True` |
| `--exists_skip` | Skip download if data already exists | `False` |

> [!NOTE]
> The published `examples/benchmarks` results were produced with the **`v1`** data set. If you want to reproduce benchmark numbers exactly, use `--version v1`.

### 3.2 Download a smaller sample

For a quick test you can pull a reduced "simple" data set:

```bash
python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --region cn
```

### 3.3 Community data source (recommended)

While the official dataset is paused, the community provides a compatible build:

```bash
wget https://github.com/chenditc/investment_data/releases/latest/download/qlib_bin.tar.gz
mkdir -p ~/.qlib/qlib_data/cn_data
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
rm -f qlib_bin.tar.gz
```

```text
qlib_bin.tar.gz                100%[==============================>]  ...  MB/s
```

### 3.4 Check data health

Before training, verify the data is complete and free of obvious anomalies:

```bash
python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data
```

You can tune the thresholds the health check uses:

```bash
python scripts/check_data_health.py check_data \
  --qlib_dir ~/.qlib/qlib_data/cn_data \
  --missing_data_num 30055 \
  --large_step_threshold_volume 94485 \
  --large_step_threshold_price 20
```

### 3.5 Keep daily data up to date

Once you have a base data set, you can incrementally append new trading days from Yahoo Finance:

```bash
python scripts/data_collector/yahoo/collector.py update_data_to_bin \
  --qlib_data_1d_dir ~/.qlib/qlib_data/cn_data \
  --trading_date 2021-05-25 \
  --end_date 2021-06-25
```

> [!CAUTION]
> You **cannot** incrementally update the offline data set provided by Qlib — some fields were removed to reduce its size. To maintain a continuously updated data set, download raw data from scratch with the Yahoo collector and then update it incrementally. See [Section 8](#8-building-your-own-data-set) for the full collect → normalize → dump pipeline.

---

## 4. Running Models with `qrun`

`qrun` is the one-command entry point for the entire automatic quant research workflow. It reads a single YAML file that declares your data, model, training segments, strategy, and analysis records, then executes them end to end.

### 4.1 Anatomy of a workflow config

Here is the complete LightGBM/Alpha158 configuration, annotated. Every Qlib workflow follows this same shape.

```yaml
qlib_init:                          # passed to qlib.init()
    provider_uri: "~/.qlib/qlib_data/cn_data"
    region: cn
market: &market csi300              # the stock universe
benchmark: &benchmark SH000300      # index used for excess-return analysis
data_handler_config: &data_handler_config
    start_time: 2008-01-01
    end_time: 2020-08-01
    fit_start_time: 2008-01-01
    fit_end_time: 2014-12-31
    instruments: *market
port_analysis_config: &port_analysis_config
    strategy:
        class: TopkDropoutStrategy
        module_path: qlib.contrib.strategy
        kwargs:
            signal: <PRED>
            topk: 50                # hold the top 50 ranked stocks
            n_drop: 5               # replace 5 each rebalance
    backtest:
        start_time: 2017-01-01
        end_time: 2020-08-01
        account: 100000000          # starting capital
        benchmark: *benchmark
        exchange_kwargs:
            limit_threshold: 0.095  # price-limit filter (CN market)
            deal_price: close       # execution price
            open_cost: 0.0005       # buy commission
            close_cost: 0.0015      # sell commission
            min_cost: 5             # minimum commission per trade
task:
    model:
        class: LGBModel
        module_path: qlib.contrib.model.gbdt
        kwargs:
            loss: mse
            learning_rate: 0.2
            max_depth: 8
            num_leaves: 210
            num_threads: 20
            colsample_bytree: 0.8879
            subsample: 0.8789
            lambda_l1: 205.6999
            lambda_l2: 580.9768
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: Alpha158
                module_path: qlib.contrib.data.handler
                kwargs: *data_handler_config
            segments:
                train: [2008-01-01, 2014-12-31]
                valid: [2015-01-01, 2016-12-31]
                test:  [2017-01-01, 2020-08-01]
    record:
        - class: SignalRecord       # save model predictions
          module_path: qlib.workflow.record_temp
          kwargs:
            model: <MODEL>
            dataset: <DATASET>
        - class: SigAnaRecord        # signal quality (IC, rank IC)
          module_path: qlib.workflow.record_temp
          kwargs:
            ana_long_short: False
            ann_scaler: 252
        - class: PortAnaRecord       # portfolio back-test analysis
          module_path: qlib.workflow.record_temp
          kwargs:
            config: *port_analysis_config
```

The top-level sections are:

| Section | Purpose |
|---------|---------|
| `qlib_init` | Arguments forwarded to `qlib.init()` (data path, region, caches). |
| `market` / `benchmark` | The stock universe and the comparison index. |
| `data_handler_config` | Time ranges and the instrument universe for feature generation. |
| `port_analysis_config` | The trading strategy and back-test settings. |
| `task.model` | Which model to train, by class + module path + hyperparameters. |
| `task.dataset` | The dataset and feature handler, with train/valid/test splits. |
| `task.record` | What artifacts to produce (signals, signal analysis, portfolio analysis). |

> [!NOTE]
> Qlib resolves every `class` + `module_path` + `kwargs` block into an object at runtime. The placeholders `<MODEL>`, `<DATASET>`, and `<PRED>` are filled in automatically — `<MODEL>` and `<DATASET>` with the trained model and dataset, `<PRED>` with the model's prediction. You don't set these yourself.

### 4.2 Run, debug, and reuse configs

Run a workflow:

```bash
qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
```

Run under the Python debugger:

```bash
python -m pdb qlib/cli/run.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
```

Name the experiment (otherwise it defaults to `workflow`):

```bash
qrun config.yaml my_experiment_name
```

**Reuse a base config.** Point `BASE_CONFIG_PATH` at another file and override only the fields you change:

```yaml
qlib_init:
    provider_uri: "~/.qlib/qlib_data/cn_data"
    region: cn
BASE_CONFIG_PATH: "workflow_config_lightgbm_Alpha158_csi500.yaml"
market: csi300
```

> [!TIP]
> Config files are rendered as Jinja2 templates before parsing. Any `{{ VAR }}` placeholder is filled from a matching environment variable, so you can parameterize a single config across runs without editing it.

### 4.3 Run one or many benchmark models

Qlib bundles a large "model zoo" of published architectures, each with a ready config. Use `run_all_model.py` to run them:

```bash
# Run a single named model
python run_all_model.py run --models=lightgbm

# Run every model for 10 iterations each
python run_all_model.py run 10
```

> [!NOTE]
> `run_all_model.py` creates an isolated virtual environment per model (because each model pins different dependency versions) and removes it afterward, keeping only the results. It currently supports **Linux only**, and does not yet run the same model in parallel.

The available models include gradient-boosted trees and a broad set of neural architectures:

| Category | Models |
|----------|--------|
| Gradient boosting | XGBoost, LightGBM, CatBoost, DoubleEnsemble |
| Linear / MLP | Linear, MLP |
| Recurrent | LSTM, GRU, ALSTM, SFM, TCN, KRNN, Sandwich |
| Attention / Transformer | Transformer, Localformer, TFT, TRA, TabNet |
| Graph-based | GATs, HIST, IGMTF |
| Other | ADARNN, ADD, TCTS |

---

## 5. Building Workflows in Code

The automatic `qrun` workflow won't fit every research need. Qlib is fully modular, so you can assemble the same pipeline step by step in Python. This is the recommended path when you want to inspect intermediate data, customize the loop, or integrate Qlib into a larger program.

```python
import qlib
from qlib.constant import REG_CN
from qlib.utils import init_instance_by_config, flatten_dict
from qlib.workflow import R
from qlib.workflow.record_temp import SignalRecord, PortAnaRecord, SigAnaRecord
from qlib.tests.config import CSI300_BENCH, CSI300_GBDT_TASK

# 1. Initialize Qlib
provider_uri = "~/.qlib/qlib_data/cn_data"
qlib.init(provider_uri=provider_uri, region=REG_CN)

# 2. Build the model and dataset from config dicts
model = init_instance_by_config(CSI300_GBDT_TASK["model"])
dataset = init_instance_by_config(CSI300_GBDT_TASK["dataset"])

# 3. (Optional) inspect the prepared data
example_df = dataset.prepare("train")
print(example_df.head())

# 4. Train, predict, and analyze inside an experiment
with R.start(experiment_name="workflow"):
    R.log_params(**flatten_dict(CSI300_GBDT_TASK))
    model.fit(dataset)
    R.save_objects(**{"params.pkl": model})

    recorder = R.get_recorder()

    # Save predictions
    sr = SignalRecord(model, dataset, recorder)
    sr.generate()

    # Analyze signal quality (IC, rank IC)
    sar = SigAnaRecord(recorder)
    sar.generate()

    # Portfolio back-test
    par = PortAnaRecord(recorder, port_analysis_config, "day")
    par.generate()
```

```text
                          feature                                 ...
                            KMID      KLEN      KMID2  ...    LABEL0
datetime   instrument                                 ...
2008-01-02 SH600000     0.013... -0.005...  0.002...  ...  0.011...
```

> [!TIP]
> The `with R.start(...)` block is the heart of every code-driven workflow. Everything you log or save inside it is grouped under one **recorder**, which you can reload later by id. See [Section 6](#6-tracking-experiments-with-the-recorder).

### 5.1 The Model interface

Every Qlib model implements two methods:

| Method | Signature | Purpose |
|--------|-----------|---------|
| `fit` | `model.fit(dataset)` | Train on the dataset's `train`/`valid` segments. |
| `predict` | `model.predict(dataset, segment="test")` | Produce predictions for a segment (default `test`). |

Models that subclass `ModelFT` also support `finetune(dataset)` for continued training from a saved checkpoint.

---

## 6. Tracking Experiments with the Recorder

Qlib's experiment manager (`R`) wraps MLflow with a more intuitive, object-based API. An **experiment** groups related runs; each run is a **recorder** that stores parameters, metrics, and artifacts (models, predictions, reports).

### 6.1 Common recorder operations

```python
from qlib.workflow import R

# Start an experiment and an auto-named recorder
with R.start(experiment_name="my_exp", recorder_name="run_1"):
    R.log_params(learning_rate=0.01, max_depth=8)   # log hyperparameters
    R.log_metrics(train_loss=0.33, step=1)          # log metrics over time
    R.save_objects(**{"pred.pkl": predictions})     # save Python objects
    rid = R.get_recorder().id                        # remember the run id

# Later: reload an artifact from a finished run
recorder = R.get_recorder(recorder_id=rid, experiment_name="my_exp")
pred = recorder.load_object("pred.pkl")
```

### 6.2 Recorder API reference

| Method | What it does |
|--------|--------------|
| `R.start(experiment_name=..., recorder_name=..., resume=False)` | Context manager that starts (or resumes) an experiment + recorder. |
| `R.log_params(**kwargs)` | Log hyperparameters/config values. |
| `R.log_metrics(step=None, **kwargs)` | Log numeric metrics, optionally over training steps. |
| `R.save_objects(local_path=None, artifact_path=None, **kwargs)` | Persist Python objects or files as artifacts. |
| `R.load_object(name)` | Load a saved artifact from the active recorder. |
| `R.set_tags(**kwargs)` | Attach searchable tags to a run. |
| `R.get_recorder(recorder_id=..., experiment_name=...)` | Retrieve a specific recorder. |
| `R.list_experiments()` | Dictionary of all experiments. |
| `R.list_recorders(experiment_name=...)` | Dictionary of all recorders in an experiment. |
| `R.search_records(experiment_ids, order_by=[...])` | Query runs as a pandas DataFrame. |
| `R.delete_exp(...)` / `R.delete_recorder(...)` | Remove experiments or runs. |

> [!IMPORTANT]
> Do **not** call `qlib.init()` again while a recorder is active — Qlib raises a `RecorderInitializationError` to prevent silently moving your experiment's storage location.

> [!NOTE]
> To resume a finished run, pass the **exact same** `experiment_name` and `recorder_name` (or `recorder_id`) plus `resume=True`. By default, experiment artifacts are stored under `./mlruns` in your working directory.

### 6.3 Record templates

The `record` section of a workflow (and the classes you call in code) are **record templates** that compute and save standard research outputs:

| Record template | Produces |
|-----------------|----------|
| `SignalRecord` | The raw model predictions (the trading "signal"). |
| `SigAnaRecord` | Signal-quality metrics: Information Coefficient (IC), rank IC, long-short returns. |
| `PortAnaRecord` | A full portfolio back-test: returns, risk, and the analysis tables printed by `qrun`. |

---

## 7. Querying Data with the `D` Interface

`D` is Qlib's data-access object. After `qlib.init()`, use it to load calendars, instrument lists, and computed features.

```python
import qlib
from qlib.data import D
from qlib.constant import REG_CN

qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)

# Trading calendar
print(D.calendar(start_time="2010-01-01", end_time="2017-12-31", freq="day")[:2])

# Parse a market name into an instrument universe, then list members
instruments = D.instruments("csi500")
print(D.list_instruments(instruments=instruments,
                         start_time="2010-01-01", end_time="2017-12-31",
                         as_list=True)[:6])

# Load features (raw fields and on-the-fly expressions)
fields = ["$close", "$volume", "Ref($close, 1)", "Mean($close, 3)", "$high-$low"]
df = D.features(["SH600000"], fields,
                start_time="2010-01-01", end_time="2017-12-31", freq="day")
print(df.head())
```

### 7.1 The `D` methods

| Method | Signature | Returns |
|--------|-----------|---------|
| `D.calendar` | `D.calendar(start_time, end_time, freq="day", future=False)` | List of trading timestamps. |
| `D.instruments` | `D.instruments(market="all", filter_pipe=None)` | A stock-pool configuration. |
| `D.list_instruments` | `D.list_instruments(instruments, start_time, end_time, freq="day", as_list=False)` | The concrete instrument list/dict. |
| `D.features` | `D.features(instruments, fields, start_time, end_time, freq="day")` | A pandas DataFrame of computed features. |

### 7.2 The expression language

Feature fields are written as expressions. A field prefixed with `$` is a raw column; operators compute derived signals on the fly:

| Expression | Meaning |
|------------|---------|
| `$close`, `$volume`, `$high`, `$low` | Raw daily fields. |
| `Ref($close, 1)` | Yesterday's close (lag by 1 bar). |
| `Mean($close, 3)` | 3-bar moving average of close. |
| `$high - $low` | Intraday range (arithmetic on fields). |

> [!TIP]
> The expression engine is backed by caches. With `ExpressionCache` and `DatasetCache` enabled, repeated feature queries are dramatically faster — Qlib's own benchmarks show feature construction dropping from ~184s (HDF5) to ~7s with both caches on a single CPU.

### 7.3 Pre-built feature sets: Alpha158 and Alpha360

You rarely write all your features by hand. Qlib ships two curated feature handlers used throughout the benchmarks:

| Dataset | US Market | China Market | Description |
|---------|:---------:|:------------:|-------------|
| `Alpha158` | ✅ | ✅ | 158 engineered factors (price/volume patterns). |
| `Alpha360` | ✅ | ✅ | 360 raw normalized price/volume features over a lookback window. |

Both live in `qlib.contrib.data.handler` and are plugged into a `DatasetH` via the `handler` block shown in [Section 4.1](#41-anatomy-of-a-workflow-config).

---

## 8. Building Your Own Data Set

If the ready-made data doesn't meet your needs, Qlib provides a three-stage pipeline — **collect → normalize → dump** — to turn raw CSVs into the Qlib binary format. Each market has a collector script under `scripts/data_collector/`.

### 8.1 Collect, normalize, dump (Yahoo example)

```bash
# 1. Download raw CSVs (one file per symbol)
python scripts/data_collector/yahoo/collector.py download_data \
  --source_dir ~/.qlib/stock_data/source/cn_data \
  --start 2020-01-01 --end 2020-12-31 --delay 1 --interval 1d --region CN

# 2. Normalize (adjust prices; scale first valid close to 1)
python scripts/data_collector/yahoo/collector.py normalize_data \
  --source_dir ~/.qlib/stock_data/source/cn_data \
  --normalize_dir ~/.qlib/stock_data/source/cn_1d_nor \
  --region CN --interval 1d

# 3. Dump to Qlib binary format
python scripts/dump_bin.py dump_all \
  --data_path ~/.qlib/stock_data/source/cn_1d_nor \
  --qlib_dir ~/.qlib/qlib_data/cn_data \
  --freq day --exclude_fields date,symbol --file_suffix .csv
```

> [!WARNING]
> Yahoo Finance blocks access from within China. Switch your network if you need the Yahoo crawler. The collected data may also contain anomalies even after price adjustment — validate it with the health check from [Section 3.4](#34-check-data-health).

### 8.2 Build an index/instrument universe

To track which stocks belong to an index over time (additions and removals), use the index collectors:

```bash
# China indices: CSI300, CSI100, CSI500
python scripts/data_collector/cn_index/collector.py \
  --index_name CSI300 --qlib_dir ~/.qlib/qlib_data/cn_data --method parse_instruments

# US indices: SP500, NASDAQ100, DJIA, SP400
python scripts/data_collector/us_index/collector.py \
  --index_name SP500 --qlib_dir ~/.qlib/qlib_data/us_data --method parse_instruments
```

### 8.3 Other data sources

Qlib includes collectors beyond daily equities:

| Collector | Data | Notes |
|-----------|------|-------|
| `yahoo` | Global equities (CN/US/IN/BR), 1d & 1min | Most common source. |
| `baostock_5min` | China 5-minute bars | High-frequency CN data. |
| `crypto` | Coingecko crypto prices | Data-retrieval only — no back-test (no OHLC). |
| `fund` | China mutual-fund NAVs | From 天天基金网 (East Money). |
| `pit` | Point-in-Time quarterly fundamentals | Use `dump_pit.py` to dump. |
| `crowd_source` | Cross-validated community data | Mitigates survivorship bias. |

---

## 9. Advanced Workflows

### 9.1 Adapting to market dynamics (rolling & DDG-DA)

Financial markets are non-stationary, so a model trained on old data decays over time. Qlib provides two adaptation strategies:

- **Rolling Retraining (RR):** periodically retrain on the most recent data.
- **DDG-DA:** a meta-learning method that forecasts the future data distribution and adapts to it.

Run the rolling baseline:

```bash
cd examples/benchmarks_dynamic/baseline
python rolling_benchmark.py run

# Use a different forecasting model
python rolling_benchmark.py --conf_path=workflow_config_lightgbm_Alpha158.yaml run
```

The rolling entry point is also exposed as a module:

```bash
python -m qlib.contrib.rolling base --conf_path <path/to/workflow.yaml> run
```

### 9.2 Reinforcement learning for order execution

Qlib includes an RL framework for modeling continuous trading decisions — most notably **order execution**, where the goal is to fill an order while minimizing market impact. Example configurations under `examples/rl_order_execution/` cover TWAP, PPO, and OPDS methods.

> [!NOTE]
> The RL framework integrates with the back-test executor and supports **nested execution** — for example, optimizing an intraday order-execution agent *inside* a daily portfolio strategy. See [Section 9.3](#93-nested-decision-execution).

### 9.3 Nested decision execution

Real trading involves decisions at multiple time scales: a daily portfolio strategy decides *what* to hold; an intraday executor decides *how* to trade it. Qlib's `NestedExecutor` lets you stack strategies and executors at different granularities and optimize them together. The example lives at `examples/nested_decision_execution/workflow.py`.

### 9.4 Hyperparameter tuning

The bundled Optuna example tunes LightGBM hyperparameters with a live dashboard:

```bash
# Terminal 1: create the study and launch the dashboard
optuna create-study --study LGBM_158 --storage sqlite:///db.sqlite3
optuna-dashboard --port 5000 --host 0.0.0.0 sqlite:///db.sqlite3

# Terminal 2: run the search
python hyperparameter_158.py
```

---

## 10. Analyzing Results

After a workflow finishes, run the graphical report notebook to visualize signal quality and portfolio performance:

```bash
python -m pip install .[analysis]
jupyter notebook examples/workflow_by_code.ipynb
```

The analysis covers two areas:

**Forecasting-signal analysis** (how good are the predictions?):

- Cumulative return by prediction-ranked groups
- Return distribution (long/short)
- Information Coefficient (IC) and Monthly IC
- Auto-correlation of the signal

**Portfolio analysis** (how does the strategy perform?):

- Back-test cumulative return vs. benchmark
- Risk metrics: annualized return, information ratio, max drawdown

The same numbers are printed in the `qrun` summary you saw in [Section 1.3](#13-run-a-complete-workflow): `mean`, `std`, `annualized_return`, `information_ratio`, and `max_drawdown`, reported both with and without trading cost.

> [!TIP]
> Always read the **with-cost** table. The example LightGBM strategy's annualized return drops from `0.178` to `0.129` once commissions are applied — costs matter.

---

## 11. How Qlib Works (Concepts)

You can use Qlib productively from the configs above, but understanding the component layers helps you customize. Qlib is built as a stack of loosely-coupled modules; each can be used stand-alone.

```text
   Data Layer        Raw fields + expression engine (D, ExpressionCache)
        │
   Dataset / Handler  Feature sets (Alpha158/Alpha360) split into train/valid/test
        │
   Model Layer        fit() / predict() — trees, neural nets, RL agents
        │
   Workflow Layer     R recorder + record templates (Signal/SigAna/PortAna)
        │
   Strategy Layer     Turn signals into trade decisions (TopkDropoutStrategy)
        │
   Backtest / Executor  Simulate trading, optionally nested across time scales
        │
   Analysis           IC, returns, risk, drawdown
```

Key concepts:

- **Signal:** a model's per-stock score for a future period. Higher score = more attractive.
- **Strategy:** a rule that converts the signal into trades. `TopkDropoutStrategy` holds the top-`topk` stocks and swaps out `n_drop` of them each rebalance.
- **Recorder:** the unit that stores one run's parameters, metrics, and artifacts.
- **Handler:** the component that builds features and labels for a time range.
- **Offline vs. online mode:** by default Qlib runs **offline** (data lives on your machine). In **online mode**, a shared data server (Qlib-Server) serves data and caches to multiple clients for higher cache-hit rates and lower disk use.

---

## 12. Troubleshooting

### Error: `TypeError when early_stopping_rounds=None` (LightGBM 4.0+)

**Cause:** LightGBM 4.0 changed the early-stopping API; Qlib model code that passes `early_stopping_rounds=None` triggers a `TypeError`.
**Solution:** Pin a LightGBM 3.x version (for example `lightgbm==3.1.0` as used by several benchmark configs), or upgrade to a Qlib build that handles LightGBM 4.0+.
**Prevention:** Match the `requirements.txt` shipped with the specific benchmark model you are running.

### Error: `The benchmark SH000300 does not exist. Please provide the right benchmark`

**Cause:** The benchmark index symbol referenced in your config isn't present in your downloaded data set.
**Solution:** Confirm your data region matches the benchmark (`SH000300` is the CSI300 index in CN data), and that your data download completed. Re-download with the correct `--region`.
**Prevention:** Keep `market`, `benchmark`, and `qlib_init.region` consistent in your config.

### Error: `unhashable type: 'list'` in `InternalData.setup()` (DDG-DA)

**Cause:** A known chain of compatibility bugs in the DDG-DA dynamic workflow involving newer pandas indexing and LightGBM 4.0+.
**Solution:** Use the pinned dependency versions for the DDG-DA example, and a pandas version compatible with the example's `requirements.txt`.
**Prevention:** Run dynamic-benchmark examples in their own virtual environment with the example's exact requirements.

### Issue: Data loading is very slow

**Cause:** Feature computation without caches, or redundant filter execution across field groups (a known inefficiency in `SeriesDFilter`).
**Solution:** Enable `ExpressionCache`/`DatasetCache` (server mode), and reduce the number of distinct `fields_group` filters where possible.
**Prevention:** Use the cache configuration and load only the fields you need.

### Error: `Invalid provider uri: <path> ... This path does not exist.`

**Cause:** `qlib.init()` was given a `provider_uri` that doesn't exist on disk.
**Solution:** Download data to that path first (Section 3), or point `provider_uri` at your actual data directory.
**Prevention:** Verify the directory exists before calling `qlib.init()`.

### Issue: `qrun` imports the wrong Qlib

**Cause:** Running `qrun` from a directory that contains a folder named `qlib` (such as the cloned repo root) makes Python import the source tree instead of the installed package.
**Solution:** Run from `examples/` or any other directory.
**Prevention:** Keep your working directory separate from the source checkout.

### Issue: pandas `group_key` break change

**Cause:** From pandas 1.5 to 2.0 the default of `groupby(group_keys=...)` changed, which can break some scripts.
**Solution:** Qlib sets `group_key=False` internally, but a few scripts (e.g. `gen_training_orders.py`, the TRA dataset, `tft.py`) may still need attention with newer pandas.
**Prevention:** Use a pandas version compatible with your target model's requirements.

### Issue: LightGBM build fails on Apple Silicon

**Cause:** Missing OpenMP runtime when compiling the LightGBM wheel.
**Solution:** `brew install libomp`, then re-run `pip install .`.
**Prevention:** Install `libomp` before building on macOS M1/M2.

---

## 13. FAQ

**Q: What's the difference between `qrun` and `workflow_by_code`?**
They do the same thing. `qrun config.yaml` runs a workflow declared entirely in YAML; `workflow_by_code` assembles the identical pipeline in Python so you can customize each step. Start with `qrun`, switch to code when you need control.

**Q: The official data download fails — what do I do?**
The official hosted dataset is temporarily disabled. Use the community data set (`chenditc/investment_data`) described in [Section 3.3](#33-community-data-source-recommended).

**Q: How do I change the stock universe?**
Set `market` (e.g. `csi300`, `csi500`, `all`) in your config, and make sure `benchmark` and your data region match.

**Q: Where are my trained models and predictions stored?**
Under the experiment-tracking URI, which defaults to `./mlruns` in your working directory. Reload artifacts with `R.get_recorder(...).load_object(name)`.

**Q: Can I use my own model?**
Yes. Subclass `qlib.model.base.Model` (implement `fit` and `predict`) and reference it in the `task.model` block with its `class` and `module_path`.

**Q: Which data set version reproduces the published benchmark numbers?**
The `v1` Yahoo data set. Pass `--version v1` to the downloader.

**Q: Does Qlib support intraday / high-frequency trading?**
Yes. Download `1min` data, and use the high-frequency examples (`examples/highfreq/`) and nested execution. Note that some operations require both 1d and 1min data.

**Q: Can I back-test crypto?**
You can retrieve crypto data, but the crypto collector provides retrieval only — back-testing isn't supported there because it lacks full OHLC data.

**Q: How do I run many models to compare them?**
Use `python run_all_model.py run <iterations>` (Linux only). It isolates each model's dependencies in its own virtual environment.

**Q: What does `TopkDropoutStrategy` actually do?**
Each rebalance it holds the `topk` highest-ranked stocks by the model's signal and replaces the `n_drop` lowest-ranked currently-held positions — a simple, effective long-only ranking strategy.

**Q: How do I adapt a model to changing markets?**
Use rolling retraining (`examples/benchmarks_dynamic/baseline`) or the DDG-DA meta-learning approach (`examples/benchmarks_dynamic/DDG-DA`).

**Q: Can I share data across machines?**
Yes — deploy Qlib in **online mode** with Qlib-Server, which serves data and caches to multiple clients.

**Q: My run prints warnings about Redis / cache not used. Is that a problem?**
No. If Redis isn't reachable, Qlib disables the disk caches that depend on it and continues. Performance is lower but results are unaffected.

---

## 14. Command Reference

| Command | Purpose |
|---------|---------|
| `pip install pyqlib` | Install Qlib. |
| `python -m qlib.cli.data qlib_data --target_dir <dir> --region cn` | Download data (as a module). |
| `python scripts/get_data.py qlib_data --target_dir <dir> --region cn` | Download data (from source). |
| `python scripts/check_data_health.py check_data --qlib_dir <dir>` | Validate data integrity. |
| `qrun <config.yaml>` | Run a full workflow from a config. |
| `qrun <config.yaml> <exp_name>` | Run a workflow under a named experiment. |
| `python -m pdb qlib/cli/run.py <config.yaml>` | Debug a workflow. |
| `python run_all_model.py run --models=<name>` | Run a single benchmark model. |
| `python run_all_model.py run <n>` | Run all benchmark models for `n` iterations. |
| `python -m qlib.contrib.rolling base --conf_path <yaml> run` | Run a rolling-retrain workflow. |
| `python scripts/dump_bin.py dump_all --data_path <dir> --qlib_dir <dir> --freq day` | Convert CSVs to Qlib binary format. |
| `python scripts/data_collector/<source>/collector.py download_data ...` | Collect raw data from a source. |

### Key Python entry points

| Object | Import | Purpose |
|--------|--------|---------|
| `qlib.init` | `import qlib` | Initialize Qlib with a data path and region. |
| `D` | `from qlib.data import D` | Query calendars, instruments, and features. |
| `R` | `from qlib.workflow import R` | Manage experiments and recorders. |
| `init_instance_by_config` | `from qlib.utils import init_instance_by_config` | Build a model/dataset/strategy from a config dict. |
| `SignalRecord`, `SigAnaRecord`, `PortAnaRecord` | `from qlib.workflow.record_temp import ...` | Generate predictions and analysis. |

---

_Generated June 2026 from microsoft/qlib at d5379c5. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._