Benchmark weighting: default backtest compares an equal-weighted portfolio against a cap-weighted index

[taro0915]

1. Summary

In the default backtest configuration, TopkDropoutStrategy allocates cash equally across
holdings while the reported excess return subtracts a cap-weighted index (SH000300) as a plain
difference. Over 2017-01-03..2020-07-31 that weighting mismatch alone is worth -4.13% per year
(information ratio -0.723) with no alpha involved, measured by simply holding the point-in-time
CSI300 constituents equally weighted. Because equal weighting underperformed the cap-weighted
index in this window, the offset makes reported model excess returns lower than a
weighting-consistent comparison would show, so this is a measurement-convention question rather
than a result that flatters Qlib.

2. Evidence in code

Repository commit 79633dd9506ea689e5400dea0197717b5b3d74b7 (2026-07-23).

• Portfolio is equal-weighted by cash. qlib/contrib/strategy/signal_strategy.py:266

  value = cash * self.risk_degree / len(buy) if len(buy) > 0 else 0
  

• Excess return is a plain difference against the benchmark series.
  qlib/workflow/record_temp.py:507-512

  analysis["excess_return_without_cost"] = risk_analysis(
      report_normal["return"] - report_normal["bench"], freq=_analysis_freq
  )
  analysis["excess_return_with_cost"] = risk_analysis(
      report_normal["return"] - report_normal["bench"] - report_normal["cost"], freq=_analysis_freq
  )
  

• The shipped example configs set a cap-weighted index as that benchmark.
  examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml:5 (benchmark: &benchmark SH000300)
  and line 23 inside backtest:. The same default appears in qlib/workflow/record_temp.py:410.

• An equal-weight benchmark is anticipated but not implemented.
  qlib/backtest/profit_attribution.py:238

  :param bench: The benchmark for comparing. TODO: if no benchmark is set, the equal-weighted is used.
  

I searched the GitHub issue tracker via the search API for existing discussion of benchmark
weighting and did not find a matching issue; apologies if I missed one.

3. Measurement

3a. Model-independent control

To isolate the weighting convention from any alpha, I held the point-in-time CSI300 constituents
equally weighted and compared them with SH000300 over the same window the example configs
backtest.

• Window: 2017-01-03 .. 2020-07-31, 871 trading days
• Universe: D.instruments(market="csi300"), so Qlib's own membership spans apply
• Constituents with a usable return per day: median 291, max 300
• Statistics from Qlib's own qlib.contrib.evaluate.risk_analysis

Series	Daily mean	Daily std	Annualized return	Information ratio	Max drawdown
Equal-weight CSI300	0.000355	0.012662	+8.44%	0.432	-39.96%
SH000300 (cap-weighted)	0.000467	0.012298	+11.10%	0.585	-37.05%
Equal-weight minus cap-weighted	-0.000174	0.003708	-4.13%	-0.723	-19.50%

No model is involved in this table.

3b. Same effect on an actual model run

I then ran examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml (single seed,
seed=0) and re-measured the identical daily return series against both benchmarks. As a sanity
check, my recomputed SH000300 daily return matched the bench column Qlib produced to
max absolute difference 0.00e+00.

Metric	vs SH000300 (cap-weighted)	vs equal-weight CSI300	Difference
Annualized excess, without cost	+17.33%	+22.51%	+5.18 pt
Annualized excess, with cost	+12.80%	+17.96%	+5.16 pt
Information ratio, without cost	2.005	3.347	+1.342
Information ratio, with cost	1.481	2.670	+1.189
Daily std of excess	0.005605	0.004360	-22.2%
Max drawdown, with cost	-9.94%	-4.25%	—

Single-seed IC figures for that run: IC 0.0475, ICIR 0.3892, Rank IC 0.0506, Rank ICIR 0.4213.
examples/benchmarks/README.md:45 reports IC 0.0448±0.00, ICIR 0.3660±0.00, Rank IC 0.0469±0.00,
Rank ICIR 0.3877±0.00, annualized return 0.0901±0.00, IR 1.0164±0.00, max drawdown -0.1038±0.00
over 20 seeds. My IC-family numbers are close but not inside the quoted ±0.00 bands, and my
backtest annualized return (0.1280 with cost) differs from the README's 0.0901. I did not chase
that gap down; it may come from the seed count, the dataset snapshot, or environment differences,
and it does not affect the comparison in the table above, which uses one return series measured
two ways.

4. Implication

For any window in which equal weighting and cap weighting diverge, excess_return_with_cost and
excess_return_without_cost carry a systematic component that reflects the weighting convention
rather than the model. In this window that component is about -4.13% per year, i.e. reported
excess returns for equal-weighted strategies are understated by roughly that amount. The sign is
window-dependent: in a period where equal weighting outperforms, the same mismatch would overstate
them instead.

5. Reproduction

Dataset: v2/qlib_data_cn_1d_latest.zip, 196.5 MB

  python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

  import qlib
  from qlib.constant import REG_CN
  from qlib.data import D
  from qlib.contrib.evaluate import risk_analysis

  qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)

  START, END = "2017-01-01", "2020-08-01"

  px = D.features(D.instruments("csi300"), ["$close"],
                  start_time=START, end_time=END, freq="day")["$close"].unstack("instrument")
  # NOTE: pandas defaults pct_change to fill_method="pad", which turns a suspended name into a
  # 0% day and drags the cross-sectional mean toward zero. With the default the median number
  # of constituents contributing a return per day is 376; with fill_method=None it is 291.
  ew = px.pct_change(fill_method=None).mean(axis=1, skipna=True)

  bm = D.features(["SH000300"], ["$close"], start_time=START, end_time=END,
                  freq="day")["$close"].unstack("instrument")["SH000300"].pct_change(fill_method=None)

  idx = ew.index.intersection(bm.index)
  print(risk_analysis(ew.reindex(idx).dropna()))
  print(risk_analysis(bm.reindex(idx).dropna()))
  print(risk_analysis((ew.reindex(idx) - bm.reindex(idx)).dropna()))

Environment: Windows 11, Python 3.12.8 in a clean venv, pyqlib 0.9.7 from PyPI,
numpy 1.26.4, pandas 2.1.4, lightgbm 4.7.0.

Two environment notes that cost me a couple of runs and may help others, neither of which is a
Qlib defect:

• On Windows, a script that builds an Alpha158 dataset must sit behind
  if __name__ == "__main__":, otherwise the joblib workers re-import the module and re-enter
  dataset construction.
• Recent MLflow releases refuse the filesystem tracking backend by default; setting
  MLFLOW_ALLOW_FILE_STORE=true restores the current recorder behaviour.

6. Suggested fix

Either of these would resolve the ambiguity; the second is much cheaper:

1. Implement the equal-weight benchmark already sketched at
   qlib/backtest/profit_attribution.py:238, and let benchmark in the backtest config accept
   something like "equal_weight" in addition to an instrument code, so the benchmark weighting
   can be matched to the strategy's weighting.
2. Document the convention where excess_return_* is defined (record_temp.py, and the
   report/benchmark pages in docs/), stating that the default benchmark is cap-weighted while
   TopkDropoutStrategy is equal-weighted, and that the resulting excess return therefore
   contains a weighting component. A one-line note next to the examples/benchmarks/README.md
   results table would reach the most readers.

I am happy to open a PR for either if the direction is agreeable.

7. What I checked and found sound

I looked at these while investigating and want to record that they held up, so this issue is not
read as a broader criticism:

• Point-in-time index membership is real in the shipped data.
  instruments/csi300.txt has 820 rows for 690 unique symbols, 109 symbols carry more than one
  membership span, and the maximum for a single symbol is 5 (SH601991: 2005-01-01..2010-12-31,
  2012-01-04..2012-12-31, 2013-07-01..2014-12-12, 2015-06-15..2016-12-09, 2017-12-11..2019-06-14).
  520 of 820 rows (63.4%) end before the dataset's last calendar day (2020-09-25). Of the 441
  symbols whose CSI300 membership ended early, 434 still have a directory under features/
  (3,875 symbol directories in total). The spans are enforced at qlib/data/data.py:624-628, and
  scripts/data_collector/index.py:204-238 reconstructs them from index change history.
  Separately, scripts/data_collector/crowd_source/README.md:4 already documents that the
  Yahoo-sourced data itself can miss delisted names.
• No look-ahead in the signal-to-execution path. The signal is read from the previous step at
  qlib/contrib/strategy/signal_strategy.py:140-143 (get_step_time(trade_step, shift=1)), and
  the Alpha158/Alpha360 label Ref($close, -2)/Ref($close, -1) - 1
  (qlib/contrib/data/handler.py:90,152) spans T+1 close to T+2 close, consistent with execution
  at the T+1 close. Normalizers fit only on the training window
  (qlib/data/dataset/processor.py:197-205), with an explicit in-code warning that
  fit_end_time must not include test information.
• Transaction costs are on by default and included in the published table. Defaults are
  open_cost=0.0015, close_cost=0.0025, min_cost=5.0
  (qlib/backtest/exchange.py:48-50), and examples/run_all_model.py:155-160,178-180 populates
  the README table from 1day.excess_return_with_cost.*.

8. Limitations of this report

• Measured on CSI300 over 2017-01-03..2020-07-31 only. I did not test the US region, other index
  universes, or other windows, and the sign and size of the offset are window-dependent.
• The model run is a single seed, and as noted in 3b it does not match the README's 20-seed
  backtest figures; I did not investigate that difference.
• The equal-weight benchmark I built rebalances daily and ignores transaction costs on the
  benchmark side, which is the same convention the cap-weighted index comparison uses but is
  worth stating.
• I did not test whether an equal-weight benchmark would change the ranking of the models in
  examples/benchmarks/README.md, only the level of a single model's reported excess return.

[taro0915]

Opened #2316 with the documentation-only part of this. It adds notes to TopkDropoutStrategy, PortAnaRecord, and docs/component/strategy.rst; no behavior change, so it can be considered independently of whether the comparison itself should be changed or parameterized. If the direction here is agreed, I am still happy to follow up with an equal-weighted benchmark option.