docs(specs): normative spec of Diagram.expand, with a traversal shape catalog

[dimitri-yatsenko]

This PR replaces the draft "Traversal algebra" in reference/specs/diagram.md with a normative specification of Diagram.expand, the one traversal behind cascade and trace, and adds a catalog of worked examples that the test suite will mirror one to one.

What the spec defines

The new Traversal specification section of diagram.md:

• Base principles. P1 referential integrity, P2 compositional integrity, P3 reproducibility, P4 minimality.
• Model. Foreign-key references ref_e, with SQL's MATCH SIMPLE rule for NULLs (A2); parallel foreign keys as separate references; owned parts (A1): a part is owned when a foreign key in its primary key references the master or an owned part of the same master, renames allowed, and its owning attributes (what that chain maps onto the master's key) define its master row and its instance; the row relations down = child ∪ inst and up = parent ∪ inst. A part that is not owned is a plain table, with a warning at declaration.
• Operations. D(S) and U(S) as closures under down and up; expand(seed, "both") = U(D(S)), and why neither D(U(S)) nor D(S) ∪ U(S) works for export; the three part_integrity modes in terms of D and D1; trace = U; self.upstream for a row that does not exist yet.
• Derived results T1–T8, each with a short justification: closure properties, additivity, the duality between D and U, delete safety, the read boundary, the properties of both, the enforce equivalence D1 lift-closed ⇔ D1 = D, and topo_sort succeeding exactly when the group quotient is acyclic.
• Computation. The edge rule R1 downward (restriction, with the narrow conditions under which a condition may be copied) and upward (always a restriction); the group rule R2 (an existential, one-step lift that renames the owning attributes onto the master's key, then re-expansion); one pass over the group quotient; the exact condition under which a group needs iteration; seeds and when materialized keys may be relabeled; the plan; materialization only at delete time; concurrency.
• The contract for expanded diagrams: node set, d[T], counts(), iteration, copying, +, the operators that raise, and prune().
• Retirement of restrict (deprecated in 2.4, removed in 3.0), with migration recipes.

New and changed 2.4 behavior is marked with version-added / version-changed / version-deprecated admonitions.

What it removes

• The draft's mechanism-only derivation, including the relabel discussion that did not say when it was sound.
• restrict as an operation of the model, AND-convergence, and the "Rule 1–3" propagation section in Operational Methods.
• In cascade.md: the multi-pass loop, the part-to-master path walk, the old worked examples built on that walk, and the internal function names. It also removes the contradiction about cost: the preview never materializes keys; materialization happens only at delete time.

Shape catalog

New page reference/specs/traversal-shapes.md (added to the nav under Query Algebra, after Diagram). One subsection per shape, each with a tiny schema, a few rows, seeds, and the exact rows expand returns down, up, and both where that is informative:

• foreign-key shapes S1–S7 (identity, secondary, nullable with a shared column, renamed, parallel, diamond, diamond through a secondary foreign key);
• group shapes G1–G7 (lift, part-of-part with two chains, a second renamed foreign key to the master, a renamed sibling foreign key chained across instances, merge table, a part its master does not own, and the owned part-of-part chain with renamed attributes from Cascade may fail when Part table references another Part with renamed foreign keys datajoint-python#1429 Case 1);
• seed shapes K1–K6 (restricted by a descendant, self-referencing, seed kinds including the NOT IN NULL trap, dj.Top, empty, unrestricted);
• graph shapes X1–X2 (unloaded downstream schema, group cycle refused at declaration).

Every result was checked against a small executable model of D and U written from the definitions, and independently against the reference model in datajoint/datajoint-python#1621, whose shape tests copy each row from this catalog by hand.

cascade.md and trace.md

• cascade.md is now the down case: what each part_integrity mode deletes, the row-level enforce check before any DELETE, the symbolic preview, the delete steps, delete-time materialization, concurrency, and a worked example of a seed restricted by a descendant.
• trace.md is now the up case: upward crossings are always restrictions, a trace includes the parts of ancestor masters and their ancestors (Diagram.trace: descend through master-part boundaries (Merge-table upstream tracing) datajoint-python#1481), and self.upstream is built from the parents of the row being computed. The published user-facing material is kept.

What reviewers should check

1. The definitions. Do down, up, D, U, and D1 say what a delete, a trace, and an export should select? Do the ownership rule (A1) and its owning attributes match the declaration check and ordering in fix(declare): refuse a part reference that closes a master-part group cycle datajoint-python#1620?
2. The derived results. Does each justification of T1–T8 hold? T7 and T8 rely on every owned part being reachable from its master along its owning chain.
3. The catalog rows. Each expected row set should follow from the definitions alone. Please try a few by hand, especially G3, G4, G5, G7, and K3.
4. The 2.4 behavior changes the spec implies, which the release notes will carry:
   • traces and self.upstream include the parts of ancestor masters and their ancestors;
   • deletes whose seed is restricted by a descendant now delete (Cascade delete: a restriction referencing an earlier-deleted (downstream) table is silently invalidated by reverse-topological delete order — materialize (independent of MySQL 1093) datajoint-python#1496). delete() fails when the table is restricted by another query datajoint-python#1610 (2.3.5) already replaces a seed whose conditions contain a subquery by its keys; the spec generalizes this to delete-time materialization of every non-literal seed and every lift;
   • enforce is checked per part row before deleting, which is stricter;
   • + on expanded diagrams takes the union of their rows, and the other operators raise;
   • restrict is deprecated, and expand results are larger than restrict results because they take unions;
   • parts belong to a master's group by ownership rather than by table name (A part table that references a table downstream of its own master breaks delete, drop, trace and list_tables on the connection datajoint-python#1619): a part-of-part chain with renamed attributes lifts to its master, and a part that references its master only through a secondary or nullable foreign key gets a warning and is traversed as a plain table.
5. Wording. Propagation is described as restriction (&) throughout. Terms that are not DataJoint canon (closure, least fixed point, group quotient, lift-closed) are defined where first used.

Decisions for reviewers to confirm

• The enforce preview. Diagram.cascade(seed) with enforce holds D1(S) and does not raise; delete() runs the check and raises before deleting anything.

• + on expanded diagrams. d1 + d2 takes the union of rows for two expanded diagrams of the same direction. It raises for different directions, for an expanded diagram and one that is not expanded, and for two cascade() diagrams with different part_integrity modes.

• Owning attributes when a part has several owning chains. They come from an unrenamed -> master if the part has one, and otherwise from the first owning reference in its definition.

• Groups that iterate. A group iterates exactly when some foreign key inside it does not map the referencing part's owning attributes onto the referenced table's owning attributes. This covers both a renamed sibling reference (G4) and a part's second, renamed reference to its own master (G3).

• When a condition may be copied downward. The attributes a condition names are every key of a dict and the whole heading of a query expression. With that reading, a dict with keys the parent lacks, or a query expression that shares more attributes with the child than with the parent, is never copied.

• Relabeling materialized keys downward. Every key names, among the parent's attributes, only attributes the foreign key carries; if the foreign key has a nullable column, every key names all of its attributes, none of them NULL.

• The warning for a dict seed that names no attribute (K6). restrict() discards a dict that names none of the table's attributes before the seed reaches the traversal, so the planner (feat(diagram): Diagram.expand on a traversal plan datajoint-python#1637) cannot see it. Raising the warning needs restrict() to record that it discarded the dict. Should the spec keep the warning, or drop it?

Pages that use the old wording elsewhere (the delete how-to, the master-part spec §4, autopopulate.md §4.4, what's new) are updated in a follow-up PR once the engine changes merge.

[dimitri-yatsenko]

Revised the derivation to expand-only: dropped restrict and the additive/subtractive framing. The model is now one operation — expand(seed, direction in {up,down,both}) (cascade = down, trace = up, export = both) — over R1 (edge) + R2 (group). Multi-condition filtering reduces to seeding a common descendant or a union of expands, so it's a UI-layer composition, not a core primitive. (Note: the Operational Methods section still documents the shipped Diagram.restrict(); it's subsumed by expand in the model and would be removed when the redesign lands.)