Merge code for conf27/inductive-nlg-for-shared-tasks/ - #1
Merged
Conversation
Directory structure, .gitignore rules for logs/ and data/, and empty .env.example, so the pipeline stages have a clean base to build on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fetches each volume's CEUR-WS index page (published table of contents) and its DBLP working-notes record (cross-check source for Stage 2), caching both under data/raw/ so re-runs never re-fetch static published documents. Closes #1 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Parses each cached CEUR-WS index page into lab sections and their papers (title, authors, pdf_url taken from the href, position in section), preserving published order and section boundaries. Each paper is cross-checked against the cached DBLP working-notes record by normalized title; on a match, DBLP's more consistently formatted author spelling is preferred. Unmatched papers are logged as warnings, not treated as errors — CEUR's TOC stays authoritative for which papers exist. Closes #3 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 1: fetch and cache CEUR-WS + DBLP volume pages
Stage 2: parse volume index pages into sections and papers
Groups each parsed section's papers into shared-task entries linking one overview paper to its participants. Deviates from PLAN.md's rule 1 in an important way discovered while building this: CEUR-WS volumes front-load all of a lab's overview papers before listing any participants, and participants are not grouped contiguously per task (verified across all 8 CLEF volumes). So title-keyword matching (PLAN.md's rule 3) is the primary assignment path for any multi-overview section, not a rare fallback — those groups are always confidence=medium and written to needs_review.jsonl, never silently promoted. Single-overview sections are unaffected and stay confidence=high via positional grouping. Also excludes "Best of Labs" re-publications and rejects zero- participant groups per PLAN.md's known pitfalls. Closes #6 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 3: group papers into shared-task entries
Manually reviewed all 163 title_heuristic task groups from Stage 3
one by one and found three real assignment bugs, not just noise:
1. tokenize() treated compound acronyms ("BirdCLEF", "GeoLifeCLEF") as
single opaque tokens, so they only ever collided on their shared,
non-discriminative "CLEF" suffix. Now splits camelCase boundaries
before lowercasing.
2. Genuine title-score ties were resolved arbitrarily via max() dict
iteration order, silently inventing an assignment with no real
evidence behind it. Ties are now dropped, same as a zero score.
3. The "must be section-locally unique" cutoff for discriminative
keywords discarded real signal whenever two related overviews
shared genuine topical vocabulary (e.g. GeoLifeCLEF and PlantCLEF
both saying "plant"), letting an unrelated one-off token collision
("Zero-Shot" vs "Few-Shot" both tokenizing to "shot") win by
default. Replaced with soft 1/frequency weighting across the
section's overviews, plus a small stopword set for generic
shared-task vocabulary ("detection", "classification", "text", ...)
that was previously winning matches on zero real topical signal.
Verified against real cases before/after: PAN 2021 "Hate Speech
Spreader Detection" papers no longer wrongly land under "Style Change
Detection"; LifeCLEF 2025 PlantCLEF papers no longer wrongly land
under FungiCLEF; PAN 2024 "Conspiracy Theory" papers no longer wrongly
land under "Multilingual Text Detoxification" (this last one has no
lexical overlap with its correct overview title at all — an inherent
limit of title-only matching that needs_review.jsonl exists to catch,
not a bug to fix in code).
Corpus regenerated: 198 candidate tasks (44 high-confidence, 154
flagged for review), all 18 tests passing.
Closes #8
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix three title-matching bugs found in full needs_review audit
Scoped to the 44 high-confidence task groups from Stage 3, per user
decision to build the benchmark from that subset. Downloads each
overview PDF, parses once with liteparse, and searches a window
restricted to the abstract + introduction (ending at the first
numbered task subsection) for team/run count patterns.
Three real bugs found and fixed while spot-checking extracted counts
against the source PDFs:
1. A per-task participation sentence buried in a task subsection
("37 runs from 10 participating teams") outscored the actual
lab-wide total in the introduction ("received results from 20
teams") because it matched a higher-priority pattern. Fixed by
bounding the search window to end at the first task subsection.
2. Some overviews report runs as an un-totaled per-task breakdown
("37 runs for Task 1, 48 for Task 2, 20 for Task 3") rather than a
stated total. Now summed when 2+ such fragments are found, instead
of silently keeping just the first task's count.
3. "CENTRE@CLEF 2019 teams up with ..." used "teams" as a verb, and
the generic fallback pattern grabbed the adjacent year (2019) as if
it were a team count. Fixed the specific idiom and added a general
plausibility bound (team/run counts are never 4-digit year-shaped
numbers), plus a coverage_ratio sanity check per PLAN.md section 5:
if notebook_papers/teams > 1.5, the count is discarded as
mis-parsed rather than trusted (caught an elliptical "14 and 4
teams participated in Task 1 and Task 2, respectively" sentence
that regex genuinely cannot parse correctly).
Verified against all 44 tasks: 24 got team counts, 20 got run counts;
Touché 2020 extraction (17 teams, 41 runs) matches PLAN.md's own
worked example in pitfall 5 exactly.
Closes #10
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 4: extract claimed team/run counts from overview PDFs
Scoped to the 44 high-confidence task groups. For each participant paper, downloads and parses the PDF (cached), regexes for GitHub/ GitLab/Zenodo/Hugging Face URLs and TIRA references (tira.io URLs, docker.io image names), and HEAD-validates each candidate code URL — recording status rather than silently dropping a dead link, since a dead GitHub link is still evidence the team had a repo. Closes #12 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Joins the 44 high-confidence candidate tasks from Stage 3 with Stage 4 counts and Stage 5 code links, computes coverage_ratio, and runs the PLAN.md section 5 validation gate (unique task_id/pdf_url, coverage_ratio in [0, 1.5], zero-participant rejection, 5-task spot-check printout) before writing any deliverable — a failed validation blocks output entirely rather than writing a corpus known to have a problem. Writes data/final/shared_tasks.jsonl (full records), shared_tasks.csv (flattened for review), and report.md (per-venue/year counts, coverage stats, code-link resolution rate). Result: 44 tasks, 460 participant papers, spanning CLEF 2018-2025. Spot-checked against known ground truth: Touché 2020 shows 17 claimed teams / 41 runs, matching PLAN.md's own worked example in pitfall 5 exactly. Closes #13 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 5: resolve participant code links
Assemble final benchmark: shared_tasks.jsonl / .csv / report.md
Requested a spot-check of PLAN.md against the implementation. Rather
than re-running tests, manually verified extracted counts and task
groupings against actual source PDF text for a fresh random sample,
which surfaced three real defects:
1. extract_counts.py: "Each team could submit up to 5 runs... We
received 45 contributions from 11 different institutions" (eRisk
2018) — the per-team submission cap (5) was read as the total,
since "45 contributions" doesn't use the word "runs" at all now
excluded via negative lookbehind.
2. extract_counts.py: the abstract/intro window boundary assumed the
*first* numbered heading is always "1. Introduction" and cut at
the *second* heading found overall. eRisk 2025's PDF has no
numbered "1." heading at all, so its first numbered heading is
already "2. Task 1" — the old logic took the *third* heading
("3. Task 2") as the cutoff, pulling an entire wrong Task-1-only
subsection into the window and misreading both teams (17 vs true
25) and runs (117 vs true 128, missing a non-numbered "pilot task"
fragment in the per-task sum). Now searches explicitly for the
first heading number >= 2, and the per-task run pattern matches
non-numeric task names too.
3. group_tasks.py: CLEF eHealth 2021 had two real overview papers,
but the second ("Consumer Health Search at CLEF eHealth 2021")
never contains the word "overview" or "extended abstract", so it
was invisible to overview detection. The section looked
single-overview and got trusted as confidence=high, silently
dumping the real second task's overview and 2 of its participants
under the wrong (SpRadIE) task. Added a generalizable detector:
when the lone detected overview declares its own task number
("Task 1"), a participant declaring a *different* task number is
strong evidence of a hidden second task — downgrades to
confidence=medium for review rather than trusting it blindly.
Verified this does not false-positive on 6 other CLEF labs with
legitimate umbrella overviews covering multiple internal subtasks.
Result: final corpus is now 43 high-confidence tasks (down from 44 —
the eHealth 2021 contamination correctly moved to needs_review), with
every other task's counts and groupings verified against source PDF
text.
Closes #16
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix three real bugs found in full pipeline audit
…team_name
Audit found the ELOQUENT 2024 section publishes three organizer task
papers but only one uses the word "Overview", so the other two were
filed as participant submissions — contaminating clef2024-eloquent-
hallucigen and losing the Topical Quiz and Robustness tasks entirely.
Added detection for the lab-branded organizer title shape
("ELOQUENT 2024 - Robustness Task"), verified across all 8 volumes to
add exactly those 2 real detections with zero false positives.
An author-overlap rule was evaluated for the same purpose and
rejected: lab organizers routinely also submit competing systems
("DPRL Systems in the CLEF 2021 ARQMath Lab", "Organiser Team at
ImageCLEFlifelog 2020"), so it flagged ~48 genuine participant papers.
Also implemented two schema fields that were previously dead:
- is_umbrella (PLAN.md section 1) was hardcoded false on every record;
now true when one overview serves several sub-tasks, detected from a
plural "Tasks" title or participants declaring 2+ distinct task
numbers. Best-effort: a lab whose titles omit task numbers entirely
still reads as non-umbrella.
- team_name was null on all 450 participants; now extracted from the
two attribution shapes CEUR titles actually use, returning null
rather than guessing on descriptive titles.
Closes #18
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PLAN.md documents code_urls as the participants' code repositories,
but ~294/609 stored URLs were third-party libraries and pretrained
models (huggingface/transformers, nltk, keras, usnistgov/trec_eval,
meta-llama, sentence-transformers x25, even fasterxml — a Java JSON
library). A participant reading the corpus would have been pointed at
Meta's Llama repo as though it were a team submission.
Root cause: Stage 5 regexed the whole PDF including the bibliography,
where papers cite the tools they used. PLAN.md Stage 5 had specified
that code links live in a footnote or a "Reproducibility"/
"Availability" section; that scoping was never implemented.
- Bibliography is dropped before matching (last references heading,
and only past the first third of the document so an in-body mention
of the word cannot truncate the paper).
- URLs under known third-party namespaces are classified out of
code_urls and kept per participant in third_party_urls, so the
exclusion is auditable rather than a silent delete.
- Each retained link records whether it appeared alongside an explicit
code-availability statement, so consumers can prefer released
artifacts over passing mentions.
- HEAD-check status now reaches the deliverable via code_url_details.
Nothing is filtered by status — a dead repo is still evidence the
team published code — but a consumer can finally tell a live repo
from a 404 or an inconclusive 429 rate-limit without re-fetching.
Also implements the Stage 6 ranking that PLAN.md section 3 specifies
and that build_corpus never had (coverage_ratio desc, confidence,
is_umbrella, participant count, with null ratios sorting last), adds
--target, and surfaces team_names/live-link counts in the CSV and
report. Adds two safe team-count participation phrasings ("received
results coming from N distinct teams") while continuing to refuse
registration-only counts, which would inflate coverage_ratio.
Closes #19
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Repair corpus: organizer-paper contamination, code_urls quality, dead schema fields, Stage 6 ranking
Participants need the pre-parsed papers, not just links. PLAN.md section 7 had listed this as out of scope for the corpus builder; it is added now by request as its own re-runnable, cached stage. Parses the overview plus every participant paper for all 42 tasks (486 documents, 18.4M characters) to Markdown, which preserves the heading structure a generation task needs. Writes a per-document manifest recording source URL, chars, pages, and a needs_ocr verdict so parse quality is auditable rather than assumed, plus a README describing the layout for whoever receives the directory. OCR: liteparse cannot load a HuggingFace model directly — it delegates OCR to an HTTP server via --ocr-server-url — so PaddleOCR-VL is supported by serving it and passing that flag, with --only-needs-ocr to re-parse just the flagged documents. Measured across all 504 cached PDFs: 0 are garbled, 0 are full-page images, and exactly 1 lacks a usable text layer, so OCR is not on the default path (PLAN.md section 3 Stage 2: "Avoid OCR unless absolutely necessary"). Two real bugs found and fixed while building this: - `lit is-complex` signals its verdict through the exit code, non-zero meaning the document IS complex, the way grep exits non-zero on no-match. Treating that as a command failure discarded valid JSON, so the page count came back null precisely for documents needing OCR, which then skipped the thin-text-layer check entirely — the one genuinely unparseable paper passed silently. Stdout is now always parsed, and an unknown page count counts as suspicious. - That same is-complex verdict is not used as the OCR trigger: it fires on ordinary tables and figures, flagging 471 of 486 well-extracted papers. The character-based metrics match the independent scan exactly. Closes #21 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parsed text was already one folder per task, but nothing in the
corpus files pointed at it, so a consumer had to derive filenames from
pdf_url or join through the manifest themselves.
- Notebook papers now sit under {task_id}/participants/, keeping the
overview (the target output) visibly separate from the inputs.
- shared_tasks.jsonl carries fulltext_path on the overview and on every
participant; null rather than a fabricated path when a document was
not parsed.
- shared_tasks.csv gains overview_fulltext_path and
participant_fulltext_paths, joined by "; " in the same order as
participant_pdf_urls so the columns line up positionally.
- parse_fulltext.py now reads all_candidates.jsonl instead of
shared_tasks.jsonl, breaking the circular dependency created by
build_corpus.py needing to join this stage's manifest. It takes the
same --confidence flag so both stages select the same tasks.
Existing parsed files were moved into place rather than re-parsed, so
all 486 documents came back as cache hits.
Closes #23
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 504 PDFs plus the CEUR/DBLP HTML were being committed, pushing the repository past half a gigabyte. They are fully re-fetchable by running fetch_volumes.py / extract_counts.py / find_code.py, and AGENT.md specifies that data/ stays out of git. Untracked from the index only — every file remains on disk, and the deliverables in data/final/ plus the small joinable artifacts in data/intermediate/ stay tracked so the corpus is still versioned. Note: this stops future growth but does not shrink existing history, which still holds the previously pushed blobs. Removing those needs a history rewrite and a force-push, which is destructive to anyone who has cloned, so it is left as a separate deliberate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Figures are the raster images embedded in each paper, pulled out via
liteparse and grouped per document. liteparse writes bare filenames
into the markdown (), which only resolve when the
images sit beside the .md, so the references are rewritten to point at
the figures directory and stay clickable from the document.
Tables come from the parsed markdown, which liteparse already renders
as GitHub-style pipe tables, rather than being re-derived from the PDF.
Each is written verbatim as .md, plus a .csv when the table is
rectangular — a ragged table is left without a CSV, since silently
misaligned columns are worse than no CSV at all. Tables and figures
both stay inline in the markdown as well; the extracted files are an
additional view for analysis, not a removal.
Layout stays parallel to the existing per-task folders:
{task_id}/figures/{doc}/img_p4_1.png
{task_id}/tables/{doc}/table-01.md + .csv
with {doc} being "overview" or the notebook paper's stem, and the
per-document counts (n_figures, n_tables) plus directories published on
the corpus records so they are queryable from shared_tasks.jsonl.
Known limitation, documented in the generated README: figures drawn as
vector graphics — many plots and diagrams — are not raster images and
so are not extracted as files; their captions remain in the text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tables are now treated like figures: verbatim markdown plus an image of the table as it actually appears in the paper. A flattened CSV loses the column layout, spanning headers and alignment that make a results table readable, so it is dropped rather than kept alongside. Locating a table to crop it went through three approaches: - Matching cell text and taking the union of matched items over-ran into the body text below the table, because a long cell string recurs in prose. - Clustering those matches by vertical gap then under-ran, cutting the table off, because only a few cells match and the gaps between them exceed any sane threshold. - Using the paper's own ruling lines works: a LaTeX table is bounded by a top rule, optional mid rules and a bottom rule, so a run of nearby rules sharing an x-extent brackets exactly one table. Verified visually against a rendered crop. Short rules are filtered out (the separator above a footnote block is ~56pt against ~345pt for a real table rule), and a lone rule is treated as a separator rather than a table. Where a page has no rules at all, a text-block fallback grows from the matched cells through vertically adjacent lines, and refuses when the result exceeds half the page — that case is body text or a block the parser mis-rendered as a table, and a mispositioned crop is worse than none. Those tables keep their markdown, and the manifest records n_table_images beside n_tables so the gap stays visible. Images are cropped at 150 DPI and stored grayscale: these are black-on-white tables, so it roughly halves the file size, and rendering lower then downscaling was actually *larger*, since resampling blurs the glyphs and defeats PNG compression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pairing each markdown table to an image by matching its cell text was unsound. liteparse's text reconstruction is unreliable for large tables — in the eRisk 2018 overview it collapsed a 34-team results table into one markdown row — so the text match anchored that markdown table to a different, smaller table elsewhere in the paper, and the image was filed under a table it did not show. Table images are now found purely from the page geometry: every ruled region in the document is cropped and named for the page it came from (page011-table01.png). An image therefore always shows the table it claims to show, and no longer depends on the parser having reconstructed that table correctly. It also means large tables the parser mangles are still captured faithfully as images — verified against a full-page 34-row results table, caption included. The markdown tables remain as the text view, in document order, and the README states plainly that the two are independent and that the image wins where they disagree. The text-matching localisation and its text-block fallback are removed rather than left unused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 7: parsed full text, aligned with shared_tasks.jsonl / .csv
A guide for colleagues using the corpus: where the data comes from, how tasks were identified and why 156 of 198 candidates are held back for review, the file layout and record schema, what each field can and cannot be trusted for, and how to reproduce the pipeline. Documents the judgement calls rather than only the outputs — including the two grouping failure modes that were found by reading the data, the rejected author-overlap heuristic, why coverage_ratio is null for half the corpus, and why the table images should be trusted over the parsed markdown tables. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Makes the corpus guide the repository's landing page, so colleagues find it without being pointed at a filename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- .env.example was empty and stays empty: nothing in src/ or tests/ reads an environment variable or credential, since CEUR-WS and DBLP are public. AGENT.md asks for the file to list every key the project reads, and that list is empty, so the file carried no information. - SKILL.md is now untracked and stays on disk alongside AGENT.md and PLAN.md. .gitignore already intended this but named it "SKILLS.md", so the singular filename never matched. - data/final/.gitkeep and data/intermediate/.gitkeep are redundant now that both directories hold real tracked content. pytest.ini is deliberately kept. It looks removable because `python -m pytest` passes without it, but that only works because the -m form puts the working directory on sys.path; a bare `pytest` fails with six collection errors, since the tests import from src/. logs/.gitkeep and notebooks/.gitkeep are kept too: both directories are otherwise empty and gitignored, and those markers are what preserve the layout AGENT.md prescribes across a clone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
requirements.txt pins the five packages the code actually imports. Three of them are invisible to a plain import scan and were easy to miss: lxml is never imported, only named as the parser BeautifulSoup is constructed with; PIL is imported inside a function; pytest is the runner rather than an import. A note records that liteparse is an npm package, so `pip install -r` alone is not enough to run the pipeline. python-dotenv is deliberately absent — nothing reads a credential, which is the same reason .env.example was removed. .python-version is now ignored and kept local, since the pyenv environment name is a machine-local choice rather than part of the project. The Python .gitignore template already carried the line commented out; it is now enabled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moves the setup notes that lived in its comments into the README's Reproducing section, so the liteparse npm dependency is still stated somewhere — pip install alone does not give a working pipeline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pierre-achkar
approved these changes
Aug 17, 2026
pierre-achkar
approved these changes
Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.