Skip to content

feat(detect): index Jupyter notebooks with kernel-language fenced code blocks - #1

Draft
jimwhite with Copilot wants to merge 2 commits into
v8from
copilot/fix-markdown-fenced-code-type
Draft

feat(detect): index Jupyter notebooks with kernel-language fenced code blocks#1
jimwhite with Copilot wants to merge 2 commits into
v8from
copilot/fix-markdown-fenced-code-type

Conversation

Copilot AI commented Jun 28, 2026

Copy link
Copy Markdown

Ports notebook indexing from Graphify-Labs#1498 with the core fix: fenced code blocks use the actual kernel language from notebook metadata instead of hardcoded code.

Changes

graphify/detect.py

  • NOTEBOOK_EXTENSIONS = {'.ipynb'} — kept separate from CODE_EXTENSIONS/DOC_EXTENSIONS since notebooks go through sidecar conversion
  • ipynb_to_markdown(path) — converts cells to markdown; resolves fence language via metadata.language_info.namekernelspec.language"code"; handles source as either str or list[str] (both valid per Jupyter spec)
  • convert_notebook_file(path, out_dir) — sidecar logic mirroring convert_office_file(); uses content-equality check so output-only re-runs don't churn sidecar mtime
  • classify_file().ipynbDOCUMENT
  • detect() — converts notebooks to markdown sidecars before indexing

tests/test_detect.py — 14 new tests covering language resolution priority, mixed cells, mtime stability on output-only changes, detect()/detect_incremental() integration, and string vs. list source formats.

Language resolution

meta = nb.get("metadata", {})
lang = (
    meta.get("language_info", {}).get("name")   # set by kernel at runtime
    or meta.get("kernelspec", {}).get("language") # set at notebook creation
    or "code"                                     # fallback
)

A standard Python notebook produces ```python blocks; the "code" fallback applies only when metadata is absent entirely.

Copilot AI added 2 commits June 28, 2026 18:53
… code blocks

- Add NOTEBOOK_EXTENSIONS = {'.ipynb'}
- Add ipynb_to_markdown() using metadata.language_info.name / kernelspec.language
  for fenced code type (falls back to 'code' when metadata absent)
- Add convert_notebook_file() sidecar logic mirroring office files
- classify_file() now classifies .ipynb as DOCUMENT
- detect() converts notebooks to markdown sidecars
- Tests for all new behaviour including language metadata resolution
The Jupyter format allows 'source' to be either a str or list[str].
Also adds a test for the string-source case.
Copilot AI changed the title feat(detect): add Jupyter notebook support with language-aware fenced code blocks feat(detect): index Jupyter notebooks with kernel-language fenced code blocks Jun 28, 2026
Copilot AI requested a review from jimwhite June 28, 2026 18:55
jimwhite pushed a commit that referenced this pull request Aug 2, 2026
…languages (Graphify-Labs#1581)

Cross-file name resolution folded case for every language, so `from pathlib
import Path` resolved to a shell script's `export PATH=...` node — one variable
becoming the corpus's #1 god-node (266 false incoming edges on a real repo),
polluting god-node rankings, affected blast-radius, and clustering. Reported
with a precise diagnosis by @sheik-hiiobd.

Case is semantic in Python/Rust/Go/Java/C#/Kotlin/Swift/Ruby/C/C++/JS/TS: `Path`
(class), `PATH` (env var), `path` (variable) are distinct. Fix gates folding by
language at the two resolution sites the repro exercised:

- global cross-file CALL resolver: index by exact case; a folded index is built
  only for case-insensitive-language nodes (PHP/SQL/Nim) and consulted only when
  the calling file is such a language.
- type-reference STUB rewire (_rewire_unique_stub_nodes): match stubs to real
  defs by exact case, with a folded fallback restricted to case-insensitive-
  language definitions — so a case-sensitive `PATH` can never absorb a `Path`.

For case-sensitive languages this only ever removes false edges. Concept/doc
dedup (dedup.py, guarded to non-code nodes) is intentionally left folding.
Regression tests: Python `Path` no longer hits shell `PATH`; a case-differing
cross-file ref doesn't resolve; exact-case resolution still works; PHP fold
preserved. Full suite 2777.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jimwhite pushed a commit that referenced this pull request Aug 2, 2026
…raphify-Labs#1749)

The extraction spec forbids cross-language `calls` edges, and build already
dropped cross-language INFERRED `calls`. But `imports`/`references` had no such
guard: an unresolved Python `import time` resolved by bare stem (the Graphify-Labs#1504
old-stem alias) onto a `src/time.ts` file node, welding a polyglot repo's two
language halves together. In the reporter's repo three such edges were the only
bridge between 2409 Python and 1403 TS nodes, so every backend<->frontend
shortest path routed through time.ts, inflating its betweenness ~90x and making
it the #1 reported god node.

Hoist the interop-family map to a module constant and extend the edge-loop
guard to `imports`/`imports_from`/`references`. For these relations the edge is
dropped only when BOTH endpoints are known code languages of different families,
so a config/manifest -> code reference (unknown ext) is never mistaken for a
phantom. `calls` behavior is unchanged (still INFERRED-only, still drops when
either family differs). Regression tests: py->ts import dropped, ts->ts import
kept, config->code reference kept.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants