Skip to content

Lecture data: fix live hosting risks and bring high_dim_data into shape #337

Description

@mmcky

Part of #336. A full audit of dataset usage across the 7 Python lecture repos (2026-07-15, each repo's main) surfaced two live breakage risks plus a backlog of shape-up work on the data repos. This issue documents them so the fixes land as part of the consolidation project rather than ad hoc.

Live risk 1 — a lecture reads a file that exists only on a feature branch

lecture-python-intro/lectures/mle.md (line 93) reads SCF_plus_mini_no_weights.csv from the update_scf_noweights branch of high_dim_data. The audit confirmed this file does not exist on main at all — the feature branch is its only home. Deleting or rebasing that branch is an instant 404 for the lecture, so this is sharper than the "latent breakage" noted in #336.

Live risk 2 — eight URLs depend on the dormant pre-MyST repo

QuantEcon/lecture-python (the old Sphinx repo, last pushed January 2022, not archived) still serves data to two current series. Deleting it today would break 3 lectures. In every case the consuming repo already commits its own copy of the same file — the fetch just points at the wrong repo.

Correction, 2026-07-16. This originally read "Archiving or deleting it today would break 3 lectures". Archiving turned out to be harmless: archived repositories keep serving raw content indefinitely, verified live against QuantEcon/lecture-python-website (archived, Pages still built and serving) and again after this repo was archived. What actually breaks consumers is deletion, or reuse of the freed name after a rename — the latter tested on the twin, where lecture-python-programming was renamed to .rst and the MyST repo took the vacated name, after which old URLs resolve against the new occupant instead of the archived content.

Consuming repo Lecture Refs Currently fetched from
lecture-python-programming pandas_panel 3 (realwage, countries, employ) lecture-python master
lecture-python.myst pandas_panel 2 (realwage, countries; employ already repointed) lecture-python master
lecture-python.myst ols 3 (maketable1/2/4 in the body; the solutions already read the own-repo copies) lecture-python master via blob/…?raw=true
  • Repoint all 8 references (interim: own-repo copies; final: the canonical data repo per Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336). Interim repointing is churn-tolerant — moving twice beats breaking when someone archives the old repo
  • Archive QuantEcon/lecture-pythondone 2026-07-16: renamed to lecture-python.rst, description corrected, README warning box added, then archived. Details in QuantEcon/workspace-lectures#17. The "once nothing references it" precondition proved unnecessary — archived repos keep serving raw content, so the 15 references that remain (lecture-python.zh-cn 9, lecture-stats 6) still resolve. Re-verified after archiving: all three URL forms in use return 200.

Shape-up backlog — high_dim_data

Related cleanup — orphaned data files in the lecture repos

The audit found 20 of 50 committed data files are dead weight: orphans with no reference, copies superseded by cross-repo fetches, or files shadowed by %%file cells that regenerate them at build time. Notably lecture-dp carries 10 unused files inherited from lecture-python.myst, and lecture-python-intro still commits two World Bank CSVs whose canonical copies moved to QuantEcon/data in Feb 2025.

  • Sweep the orphans per repo (intro 3, programming 7, python.myst 5, dp 10 — full list available from the audit registry)

None of the checklist items above depend on the #336 design decisions except the two "final home" repoints — everything else can proceed now.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions