Repository navigation
fix(compilers/openapi): bound or remove quadratic key scans - #774
Merged
Merged
Conversation
With external references allowed, settle resumes a schema chain that stalled on another document's cached bytes (GitHub #761). The library finds each key of a pointer by comparing the keys of the mapping that holds it, in order, so a resumed hop costs the width of every mapping it passes, and settle read each such hop twice: once in ends, to rule out a cycle, then in the resumed resolution. Nothing charged that work, so n chains into one mapping of n keys took time quadratic in n: 40,000 took 23 s where the build before the resumption existed took 5 s. settle now charges every resumption to a budget of 2^28 steps per resolution pass, before it reads anything: a step for the resumption, and for each hop the resumed resolution reads in the document's tree, a step plus the keys the library's read compares there, once for each time the hop is read. Past the budget a resolution is left where it stalled, its $ref reports that the budget stopped it, and the crossing is reported once, at the document, as openapi/budget-exceeded. The keys are counted by treeReads, a model of jsonpointer.GetTarget's walk of a YAML tree that answers each mapping from an index built once, so pricing a read costs its depth rather than the widths it scans. It follows the library's reading, not nodeview's: the first of two equal keys, a key read through an alias, and the `<<` fallback with its backtracking, since a charge along any other path could be steered past. GetTarget still makes every read; the model only prices them, and a test holds it to GetTarget on generated trees.
This was referenced Oct 6, 2026
The library finds a key in raw YAML by comparing a mapping's keys in order, so a read costs the width of every mapping it passes. Four more places paid that, or a quadratic of their own, per reference: - annotation.checkUniqueKeys compared every pair of a mapping's keys, so one extension of n keys took n squared steps to compile (GitHub #776). One pass now finds the same pair: the earliest key written again. - mappings charged a read of raw YAML along nodeview's reading, which keeps the last of two equal keys where the library keeps the first. A wide mapping under the first of two equal keys was charged as the empty second, and never reached maxMappingWork (GitHub #777). Targets and chain hops are now priced by treeReads, along the library's path. - loops read each hop in the source's tree twice and charged it as one hop, whatever the width (GitHub #775). It now reads once, and charges the keys the read compares against maxLoopWork, 2^28 steps. Past it, chains are left as they were past the hop bound: unresolved where the source is held, with the cycle-scan-failed warning. - resolve.Scope.ModelAt read a pointer whole, scanning each raw mapping below an extension for a node it then discarded (GitHub #778). It now walks a token at a time and stops where the model leaves for raw YAML. maxHeldRefs no longer derives from the loop bound, so it is stated on its own, at the value it had.
OmarAlJarrah
changed the title
This was
linked to
issues
Oct 6, 2026
BudgetExceeded's doc listed the budgets openapi.Limits sets, and said each refuses the document. Load's work budgets report it too, for the mapping targets and now for resuming chains through other documents, which no Limits field raises and which leave what they did not reach unresolved; validation over what references reach stops rather than refusing. CycleScanFailed's doc named the pre-parse scan and reach, not load's reading of the chains that name the source by its file name. maxScanMerges said a read past it refused a resumption, which was true only while treeReads priced nothing else.
resolve.Step read every token as the second of two, under a map keyed by the empty string, because the library reads the one-token pointer "/" as the root. Only the empty token needs that. Any other is now read alone: the same call, without a map and a second token to parse. ModelAt now steps every pointer it reads, so on a six-token pointer into the model this takes it from 16.3 to 14.9 microseconds and from 96 allocations to 68, against 13.5 and 33 for one read of the whole pointer.
The library's pointer walk reads the last token of a field the model holds by value as the value, and any earlier one from the field's address, where the type's methods navigate it. resolve.Step reads every token as the last, so a walk built from steps stopped at an operation's responses wherever only their methods answer, as for default: - Scope.passed missed a reference under a default response, so Scope.At read content another document holds as the source's own (GitHub #779). - ModelAt, now a walk of steps, answered nil there where reading the whole pointer finds the schema. A differential over every position in the corpus found it in the golden petstore spec. - modelCost priced a read below a default response as ending there. Step now hands such a struct on as a pointer to a copy, which every caller walks on from. The corpus differential is kept as a test.
A mapping target under an operation's default response, a reference into another document, made load walk that document's content as the source's, so a $ref in it was resolved against the source and a finding about the source's own schema was reported there (GitHub #779). Step's fix keeps it the other document's; this pins that at load, against the same document with the response under a status code.
The library reads on from a field's address only where the field holds a model, which navigates by its core's keys; any other struct value it reads on from as itself. Step addressed every struct value, so a walk of steps could navigate one by methods the library would not reach. Only models are held by value in the model today, so no read changed; Step now applies the library's own rule. Also: maxResumeWork's doc names the indexing it charges, and a comment in checkUniqueKeys says which repeat it keeps.
The ModelAt corpus differential read less than it said. Two of its four globs matched no file, so a spec at another depth or with a .yml extension was never read. It re-parsed each file rather than reading the tree the model was built from, so a stream whose first document is empty gave it no tree paths. Its depth and count caps were bare literals that dropped what lay past them without a word. And its oracle, the whole read ModelAt is held to, was copied into a second package, so editing one copy would compare the two tests against different readings with both green. openapitest now holds the walk once: SpecFiles lists every spec under a root with the filter internal/harness sweeps with, and Positions reads the model's own tree, failing past MaxTreeDepth rather than truncating. The corpus test reads all of testdata through them, and both ModelAt tests compare against one wholeModelAt.
Step moves from resolve into a package of its own, navigation, which defs can import as well: defs reads the model a token at a time with a lone GetTarget, the read Step exists to replace, and it sits below resolve. The package also says where the library's read of a pointer leaves the model. The library reads a token in the mapping an object was built from when none of the object's fields or embedded maps answers it, and it finds the key there by comparing the mapping's keys in order. Asked once per reference, that scan costs the square of a wide mapping, and a reader stepping through the model paid it even where it then stopped. - Leaves follows the library's dispatch for one token (getStructTarget, navigateModel in v1.25.2): stand-ins are read through, a model's embedded maps are asked through their own NavigateWithKey, and its core is read by key tag. Where nothing answers, it returns the mapping the library would scan, without scanning it. - Walk steps until a token leaves the model, and hands back that mapping with the tokens left, which the library reads together there, so a `<<` merge holding a decoy reads as the library reads it. - Tokens splits a pointer as the library's navigation stack holds it. Nothing calls them yet. They are held to the library's whole read by a differential over a document built to break each rule (merge decoys at the root and in a schema, value-held fields, maps embedded by pointer and by value, a resolved reference, a key written twice) and over every position in the corpus, and each rule has a mutant a test reddens.
defs.Reader read the model a token at a time with a lone GetTarget, the read Step replaced in #779. For the last token of a field the model holds by value, the library hands back the value, and an operation's responses answer default only through their pointer's methods. So the reader found no ancestor at all for a schema under a default response: a $ref to a local definition there reported "definition not found" and lowered to any, and a discriminator mapping value to one failed the same way, where the library resolves both and a status code's response compiled. Each position is now read with navigation.Step. The reader also read every pointer whole in the document first. An OpenAPI document's "$defs" is raw YAML, which the library scans for the key, so each local $defs reference cost a scan of the document's root mapping, for an answer that can be no schema. The reads now stop where navigation.Leaves says a token leaves the model: at the document, at a position in raw YAML and at a $defs written as raw YAML, none of which holds a schema, so no answer changes. n local $defs references beside n root keys, `morphic validate`: 2.93 s to 2.27 s at 10,000, and 7.78 s to 4.51 s at 20,000.
The library reads an index token on a struct that is no model as a key first, with the rest of the pointer below it, and tries the index only once that read fails, so what answers the token turns on the tokens after it. A walk of single steps reads the key alone and goes on from whatever it finds, so it could miss what the whole read reaches through the index. Walk now leaves such a read to the library whole. No model type in v1.25.2 is read by index, so no read of a parsed document changes; this keeps Walk equal to the whole read for any type the library dispatches on, as a test with synthetic types holds it.
The corpus differentials asked about each place the model's walk reaches and each path of the tree, but never a position past a $ref: a reference's site holds only the $ref, so no tree path continues below it, and no stray token appended there named a key its target holds. A read through a reference, which the library makes by reading what the reference resolved to, was compared nowhere in the corpus, though that is where an operation's default response broke the stepped walk. Positions now also reads, for each internal $ref the tree writes, every path below its target from the $ref's own site. The ModelAt and navigation differentials pass over the added positions unchanged.
ModelAt and Scope.At stop where a pointer leaves the model, but the step that left it was itself the library's read of the token in the mapping the object was built from, which scans that mapping. A pointer naming one of n top-level x- keys cost a scan of the root, so n such $refs cost n squared in ModelAt and At alone, for a node both then discarded. Both now walk through navigation and stop where Leaves says a token leaves the model, before any scan. Where the walk leaves at a reference, it still asks the library whether the target holds the token, which decides whether the reference is passed and which the resolver's own read of the pointer makes too. A hoisted $ref read its pointer twice from the root, once for ModelAt and once for At. Scope.Locate walks it once for ModelAt, DeclaredAt and At together, and the hoisting and typePosition's first hop use it. lowering.Ctx.At has no caller left, so it is gone; InScope takes the scope Locate answers. ModelAt also read "/" as the entry keyed by the empty string, where the library reads the root, which a schema document's model can answer. n $refs to n top-level keys, `morphic validate`: 4.71 s on main, 4.65 s before, 3.04 s now at 20,000; 15.04, 15.20 and 7.64 s at 40,000, the rest being the library's own read of each $ref. 40,000 $refs to the properties of a default response's schema, `morphic compile`: 181.1G instructions before, 175.8G now, 175.4G on main, with identical IR.
With external references on, loops reads every schema $ref's chain, an internal $ref's included, to find a cycle through the source's file name before the resolver follows one. It read each hop with GetTarget, which finds a key in raw YAML by comparing a mapping's keys in order, and charged maxLoopWork the keys compared. n $refs into one mapping of n keys therefore crossed the bound near 23,000, and since a chain the guard cannot read could close through the held source, every $ref past it was left unresolved: 40,000 $refs into one extension failed with 16,841 unresolved-ref where main compiles them, as did 20,000 file-name chains (#775's own generator) and 40,000 $refs to top-level keys. The cost was the guard's reads, not the resolver's. loops now reads each hop with navigation.Walk, and the raw YAML the model leaves for, or the tree a file-name hop names, with treeReads.read: the walk the price followed, answering from its index, so a hop costs its depth and a mapping its width once. maxLoopWork charges that own work. A read is given the budget left and stops within it, though a mapping it indexes is charged whole. modelCost, which still prices the library's reads of mapping targets, now prices them exactly: Walk says where the read leaves the model, so a field holding raw YAML is not charged a scan of its object's mapping, a decoy `<<` cannot steer the price off the mapping the library scans, and a pointer the library refuses is free. joinTokens keeps a byte that is no UTF-8, which jsontext spelled as U+FFFD. Walk now returns the tokens left at a failed step, so a price can count the steps taken. `morphic validate --opt allow-external-refs=true`, 40,000 $refs into one extension: exit 0 in 8.50 s, against 14.86 s on main and 16,841 errors before. #775's generator at 20,000: 5.16 s, against 13.07 s and 10,811 errors.
loops now answers a hop from treeReads' read rather than only pricing the library's, so a read that ended elsewhere than GetTarget's would be a chain the guard follows apart from the resolver, a cycle it misses. The generated differential held both walks to GetTarget on eight seeds; the fuzz target draws the generator's seed, so the search covers trees those eight never built. A minute of it reads about two million trees, 25 pointers each, without a difference.
CycleScanFailed's doc named load's chain read among the checks that report it, as reading the chains that name the source by its file name, and said the code is never a refusal. The read takes every schema chain of a held source, to find one that closes through the file name, and past its bound a $ref it did not read is left unresolved, since its chain could close through the held file. The doc now says both.
treeReads counted the library's read of raw YAML through a recursive walk of its `<<` fallback, capped at 1,024 merged mappings along one pointer, and priced a read past the cap past any limit. With references on, loops then cut the chain, and a $ref the library resolves was reported unresolved: a spec with 1,100 nested inline merges, or 45,000 through aliases at the default alias budget, exited 1 where main exits 0. The library has no such cap. The depth it reads is bounded only by the budgets the source is parsed within, which a caller sets and may turn off, so no fixed cap could be right. The fallback now keeps the reads it is inside on a stack instead of the call stack. Each is pushed at a step, so the read's limit bounds them as it bounds its steps. A read entering a mapping it is already reading for the same part would never end, in the library as here. No parsed tree holds such a cycle, since the pre-parse refusals reject an anchor naming its own ancestor; a hand-built one is priced past the limit where it closes. Below the old cap the counts are unchanged: 1.95 million generated reads match the recursive walk's steps and targets, or pass the limit on both. A chain 2^17 mappings deep is read and counted as a transcription of the library's walk reads and counts it.
The library reads an index token on a struct that is no model as a key first, with the rest of the pointer below it, and as an index only once that read fails, so what answers the token turns on the tokens after it. Walk already leaves such a read to the library. Scope.Locate, which ModelAt, DeclaredAt and At read through, walked it a step at a time and read the key alone, so ModelAt could answer nil where the whole read it replaced finds a schema through the index. ModelAt is meant to answer as that whole read did for every pointer (#778). navigation now exports the reading of one token (ReadingOf), and Locate takes what the pointer names from the library's whole read of the rest once it meets a retried index. It still notes references a step at a time, as At's own walk always has. No model type in v1.25.2 is read by index, so no read of a parsed document changes. A test with a synthetic document holds ModelAt to the whole read there.
Where Scope.Locate's walk leaves the model at a reference, the reference is passed only if its target holds the next token, and the walk asked the library, whose read compares the keys of the mapping the target was built from in order. For a $ref the resolver's own read of the pointer makes the same scan. A discriminator mapping value is never read by the resolver, so there that scan was the only one: n values into one reference's mapping of n keys cost n squared steps in the lowering's typePosition, as in main's ModelAt. A CPU profile of a 10,000-value compile put 0.19 s of the compiler's 0.33 s there. Scope now takes the question as a function, Holds, the way it already takes Ends, Mapped and Built. MappingTargets answers it from the treeReads index the mapping targets were read through, so each mapping is indexed once, and the mappings pass's own scope asks the same index. A scope without Holds asks the library as before. holds is held to GetTarget's read of the one-token pointer across generated trees, and Holds fakes that answer the opposite way show the walk passes what Holds says rather than asking the library.
Where a walk leaves the model at a reference, Scope.Holds answers whether the target's mapping holds the next token, in place of the library. The corpus differentials never exercise that answer: at 6a9f53d no corpus position left the model at a reference into a token its target holds, so a Holds that always said no would have passed them, and one unit test, on a response, was all that said otherwise. The new test builds a reference of each kind ReferenceEnd names besides a schema, each to a target holding extensions, merged keys, a key written twice and keys that need escaping. It holds the walk asking the loader's Holds to the walk asking the library at every one of them. An inverted Holds fails it at 101 assertions, and a Holds that leaves its token unescaped at 18. It shares the corpus test's reading of every reference as ending in another document.
maxNavigableHops still named Leaves as the reader it bounds, though the loop it bounds is ReadingOf's now, and treeReads' overview listed its uses without the one resolve.Scope.Holds makes of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #773, resolves #775, resolves #776, resolves #777, resolves #778, resolves #779.
They share one mechanism. The library finds a key in raw YAML by comparing the keys of the mapping that holds it, in order:
jsonpointer.GetTargeton ayaml.Node, and the read of a model object that none of its fields answers, which the library makes in the mapping the object was built from. A read therefore costs the width of every mapping it passes. Wherever morphic reads, or makes the library read, once per reference, n references into one wide mapping cost n squared. This PR removes that scan from morphic's own reads, and bounds the library reads morphic adds where it cannot avoid them.Reading a pointer as the library does:
navigationA new package,
compilers/openapi/internal/navigation, reads a pointer through the parsed model a token at a time, asGetTarget's walk does, and says where that walk leaves the model:Step(moved fromresolve, sodefscan import it) reads one token as the library's walk reads on. A model the model holds by value comes back as a pointer to a copy (openapi: Scope.At misses a reference under an operation's default response #779).ReadingOffollows the library's dispatch for one token (getStructTargetandnavigateModel, v1.25.2). It reads through stand-ins (GetNavigableNode), asks each map a model embeds through the map's ownNavigateWithKey, and reads a model's core by key tag. It says which of three ways the read goes on: a field or an entry answers the token; the read leaves the model, for the mapping the library would scan, which it returns unscanned; or the library retries an index token.Leavesasks whether it leaves.Walksteps until a token leaves the model, and returns that mapping with the tokens left, which the library reads together there. So a<<merge holding a decoy is read as the library reads it. An index token on a struct that is no model is left to the library whole, since the library tries it as an index only once the rest of the pointer fails below it as a key.Tokenssplits a pointer as the library's navigation stack holds it.ModelAt,Scope.At,defs.Reader,loopsandmodelCostall read through it now.Reading raw YAML through an index:
treeReadstreeReads(treereads.go) mirrorsGetTarget's YAML walk: the first of two equal keys, a key read through an alias, and the<<fallback with its backtracking. It answers each mapping from an index built once. Its<<fallback keeps the reads it is inside on a stack rather than the call stack, each pushed at a step, so the read's limit bounds them. It has no depth bound of its own, as the library has none: the parse budgets bound a chain's depth, and a caller may lift them. It walks two ways:costcounts the library's steps, to price the reads the library makes.readtakes the same walk but counts its own steps: one for each node it visits, key it looks up and<<key it tries. A mapping's width then costs it once, when indexed.loopsanswers fromread.A generated differential holds both walks to
GetTarget, andcost's counts to a transcription of the library's walk. A fuzz target drives that generator's seed.An earlier version of this branch capped the
<<recursion at 1,024 merged mappings along a pointer, and priced a deeper read past any limit. With references on,loopsthen left a$refthe library resolves unresolved: 1,100 nested inline merges, or 45,000 through aliases at the default alias budget, exited 1 wheremainexits 0. A read entering a mapping it is already reading for the same part would never end, in the library as here. No parsed tree holds that cycle, since the pre-parse refusals reject an anchor naming its own ancestor, and a hand-built one is priced past the limit where it closes.modelCostprices the library's read of a pointer in the source's model exactly:Walksays the read leaves the model, a step into the raw YAML;cost's price of the tokens left there.#773: resuming a chain in another document
With
allow-external-refs=true,settleresumes a schema chain that stalled on another document's cached bytes (#761). Each resumed hop was read twice, once byendsto rule out a cycle and then by the resumed resolution, and nothing charged either read.settlenow charges every resumption, before it reads anything, tomaxResumeWork(2²⁸ steps per resolution pass, the sizemaxMappingWorkalready uses). The charge is:ends, then the library), and once for other kinds, whichendsdoes not check;Past the budget, the resolution is left where it stalled. Its
$refreportsresolving it further takes more than the 268435456 steps budgeted for chains through referenced documents, in place of the bytes artifact (invalid path -- expected index, got key). The crossing is reported once, at the document, asopenapi/budget-exceeded.#777: the mapping-target charge followed the wrong reading
keysHoldingcharged a read of raw YAML alongnodeview's reading, which keeps the last of two equal keys. The library keeps the first. So a wide mapping under the first of two equal keys was charged as the empty second, and 40,000 targets took 15 s without ever reachingmaxMappingWork.mappings.pricedreplaceskeysHolding. It prices each resolved target, and each hopchainEndsreads, which was charged one step whatever its width, withmodelCostandcost.#775:
loopsWith external references on,
loopsreads every schema chain of a held source, an internal$ref's included. It is looking for a chain that closes through the source's file name, before the resolver follows it (#768). It read each hop withGetTarget, twice for a tree hop. The first version of this PR then chargedmaxLoopWork(2²⁸ steps) the keys those reads compared, wheremaincounted hops againstmaxLoopReads.That made the bound reachable by valid documents. Past it, a held source's unread chains are left unresolved, since one could close through the held file. Each of these came back with more than 10,000
unresolved-referrors wheremaincompiles it clean:$refs into one extension;$refs to as many top-level keys;loopsnow reads each hop withnavigation.Walk. It reads the raw YAML the model leaves for, or the tree a file-name hop names, withtreeReads.read.maxLoopWorkis charged that work:So a hop costs its depth, and a mapping its width once. A read is given the budget left and stops within it, though a mapping it indexes is charged whole.
30,000 ten-schema chains (about 270,000 hops in model positions) stopped
main's hop-counting guard with its warning. Here they are read in full, with nothing reported, in 18.0 s againstmain's 17.6 s.maxHeldRefsderived frommaxLoopReads, so it is now stated on its own, at the value it had (2¹⁶).#776:
checkUniqueKeyscheckUniqueKeyscompared every pair of a mapping's keys. One pass now finds the same pair: the earliest key written again, and where it is written next.#778:
ModelAt,Scope.Atanddefs.Readerresolve.Scope.ModelAtread a pointer whole. For a pointer into an extension, that scanned each raw mapping below, for a node it then discarded. Stepping a token at a time stopped the scans below. But the step that left the model was itself the library's read of the token in the object's own mapping. So a pointer naming one of n top-levelx-keys still cost a scan of the root, inModelAtand inScope.Atalike.Both now walk through
navigation, and stop where a token leaves the model, before any scan. Where a walk leaves at a reference, the reference is passed only if its target holds the token:$ref's pointer costs the resolver that scan anyway, but the resolver never reads a discriminator mapping value. So n mapping values into one reference's mapping of n keys cost the lowering'stypePositionn scans, asmain'sModelAtdid.Scope.Holdsnow answers it from the loader'streeReadsindex.MappingTargetshands the lowering that index, the one the mapping targets were read through, and the mappings pass's own scope asks the same one. A scope withoutHoldsasks the library, as before.Past an index the library retries,
Locatetakes what the pointer names from the library's whole read of the rest, asWalkdoes, which the issue requires ofModelAtfor every pointer. No v1.25.2 type is read by index, so no parsed document reaches it.A hoisted
$refread its pointer twice from the root, forModelAtand forScope.At.Scope.Locatewalks it once for both, and forDeclaredAt. The hoisting andtypePosition's first hop use it.lowering.Ctx.Athad no caller left;InScopetakes the scopeLocateanswers.defs.Readerread each#/$defs/...pointer whole in the document before any ancestor. An OpenAPI document's$defsis raw YAML, so each local$defsreference cost a scan of the root mapping, for an answer that can be no schema. Its reads now stop where a token leaves the model:$defswritten as raw YAML.None of these holds a schema, so no answer changes.
ModelAtread/as the entry keyed by the empty string, where the library reads the root.#779: a step past a struct the model holds by value
The library reads the last token of a field the model holds by value as the value. It reads an earlier one from the field's address, where the type's own methods navigate it.
Stepread every token as the last. An operation holds its responses by value and answersdefaultonly through those methods, so every walk built from single reads stopped there:Scope.Atmissed a reference under adefaultresponse, and read content another document holds as the source's own. This was already inmain.defs.Reader, which reads each position from its parent, found no$defsancestor under adefaultresponse. A$refto a local definition there reporteddefinition not foundand lowered toany, and so did a discriminator mapping value naming one. The library resolves both, and a status code's response compiled. This was already inmain.ModelAtanswered nil there, where the whole read finds the schema. This PR introduced it, and a differential over every position in the corpus found it in the golden petstore spec.modelCostpriced the read as ending there.Stepnow hands a model it lands on by value on as a pointer to a copy, which every caller walks on from. As in the library, any other struct stays a value.defs.Readerreads each position withStep.The
Scope.Atmiss showed at load. Take a mapping target under adefaultresponse that points into another document. Its content was walked as the source's, so a$refin it was resolved against the source, and a finding about the source's own schema was reported there. Under a status code, nothing was.Design notes
A mirror of the library's dispatch, rather than its read. The library's own read cannot drift from the resolver's, and every reader here used to make it. But the scan is inside it, at the step that leaves the model, so a reader asking it once per reference pays the square of a wide mapping, wherever it stops after.
ReadingOfmirrors only the dispatch for one token, and asks each map through the library's own method. Where it cannot place a token,Walklets the library read it.$ref's target; and a mutant per rule.loopsanswers fromtreeReads.read, notGetTarget. The guard must answer as the resolver does, since a cycle it misses overflows the stack. Three ways out of the refusals lose:maincompiles failing, with a test pinning the over-broad price as intended.treeReadsalready had to end whereGetTargetends for its prices to hold. It is now held there by the fuzz target as well.endskeeps its ownGetTargetread (charged), rather than reading throughtreeReads.endsguards a resumption fix(compilers/openapi): resolve mappings and cross-document refs #763 added, whose work is the library's own resumed reads. The budget bounds those. Crossing it declines the resumption, which leaves the chain failing as it did before fix(compilers/openapi): resolve mappings and cross-document refs #763.ends' read is a constant share of that work. Reading it throughtreeReadswould move the bound, not remove it, and give up the library's own read in a guard. Reading through the model roughly doubled the chains openapi: chains into one wide mapping of another document take quadratic time #773's budget admits, with no change in wall time (6.57 s against 6.30 s at 40,000).loopsis the other way round. Its own reads were the only quadratic work its bound refused, and the resolver resolves those chains by itself.Holdsis a function the scope takes, not an index it holds. MovingtreeReadsintonavigationwould give that package the compiler's YAML helpers, where its rule is that it reaches only the library. The loader already hands the lowering what it resolved as functions (Mapped,Built), andEndsreaches the scope the same way. A scope withoutHoldsanswers exactly as before, and only its cost differs.A bound that ignores the path (pointer length × the widest mapping) would need no model. But it charges a read through narrow mappings as if it crossed the widest, so a component file with one large mapping would hit the budget on chains that never touch it.
What stays uncharged: the library's own first hop into a document, and a hop that leaves the document. Those are a reference's own read, which nothing charged before fix(compilers/openapi): resolve mappings and cross-document refs #763. A resumed hop the library answers from its object cache is still charged, which overcharges it. A hop the library has resolved once never stalls, so it is never resumed.
Known limits
Past
maxResumeWork, chains fail thatmainresolved slowly. In the issue's shape for openapi: chains into one wide mapping of another document take quadratic time #773, 10,000 chains into one 20,000-key mapping of another document, 809 are left unresolved.mainresolves them in 2.5 s. The bound is on the library's own resumed reads, which nothing here can make linear (see design notes).The library's own read of each
$refstays quadratic for one wide raw mapping. It reads each$ref's pointer in the model withGetTarget:$refs to as many top-level keys still take 7.8 s, down from 15.5 s onmain;Morphic could avoid that read only by seeding the library's referenced-object cache with schemas, which load deliberately does not do (openapi: a $ref cycle closing through the source's own file name overflows the stack #768, openapi: resolve schema $refs in the compiler so the library never recurses #558).
Memory near the bound: a mapping-target layout that crosses
maxMappingWorknow resolves more targets before it does. openapi: a key written twice in raw YAML steers the mapping-target charge off the scanned mapping #777's price follows the target's position, wherekeysHoldingcharged every mapping's full width. So its peak resident memory is higher: 1,080 MB against 648 MB at 40,000 targets. Below the bound it is close: 494 MB against 469 MB at 8,000.Indexing is charged after the read that does it.
loopsandtreeReadscharge a mapping's keys once, after the read that indexed them. So the last read before a bound can pass it by the width of the mappings it met.Test plan
navigation:TestWalk_AnswersAsTheWholeReadDoesholdsWalk, plus the library's read of what it leaves, to the whole read. Its document puts<<decoys at the root and in a schema, and holds value-held fields, maps embedded by pointer and by value, a resolved reference and a key written twice.TestWalk_AnswersAsTheWholeReadDoesAcrossTheCorpusdoes the same at every corpus position.TestWalk_ScansNoMappingWhereItLeavesTheModelgives each mapping a nil first key, which a scan faults on.TestLeaves_*covers each dispatch path, including a fake model whose by-value map lacks the library's map methods, and a model's core read by tag alone.TestWalk_LeavesAnIndexTheLibraryRetriesToIt,TestWalk_ReturnsTheTokensLeftWhereAStepFails,TestTokens_DecodesAsTheLibraryReadsandTestStep_*.openapitest.Positionsgives every walk location and every tree path. For each internal$ref, it adds every path below its target, read through the$ref. Each position is also asked with stray tokens appended.ModelAtandWalkare held to the library's whole read there.Locate's scope is held toAt's, with every resolved reference read as one into another document.defs:TestTarget_ReadsPastResponsesTheOperationHoldsByValue, which fails onmain, andTestTarget_ReadsNoMappingTheModelLeavesFor.resolve:TestScope_ReadsNoRawMappingAPointerPasses,TestScope_ModelAt_ReadsThePointerAsTheLibraryDoes,TestScope_Locate_ReadsTheScopeAsAtDoes,TestScopeAt_PassesAReferenceTheWalkLeavesTheModelFrom, and themaxRefChainbound forLocate.TestScopeAt_PassesAReferenceTheWalkLeavesTheModelFromalso runs with aHoldsthat reads the target's mapping as the library does, and with fakes answering the opposite way, which the walk follows.TestScope_Locate_ReadsAnIndexTheLibraryRetriesWholeholdsModelAtto the whole read past a retried index, on a synthetic document.TestScope_Locate_HoldsAnswersAsTheLibraryAtEveryReferenceKindholds a walk asking the loader'sHoldsto one asking the library, at a reference of each of the nine kindsReferenceEndnames besides a schema. It asks for tokens the target holds itself, through a merge, escaped or written twice, and for tokens it does not hold. At6a9f53d, no corpus position left the model at a reference into a token its target holds, so this spec is built for that read. An invertedHoldsfails it, and so does one that leaves its token unescaped.Holdswiring:TestMappingTargets_HoldsReadsThroughTheTargetsIndex,TestPositionScope_ReadsThroughTheGivenIndexandTestRefScope_CarriesTheMappingTargetsItWasGiven.loops:TestLoops_AReadCostsItsDepthNotItsWidthreads n$refs into one mapping three ways: by file name, past an extension, and at top-level keys. Doubling n at most doubles the work.TestLoops_ReadAtChargesItsOwnSteps.TestLoops_AReadStopsAtTheBoundItHasLeft.treeReads:TestTreeReads_CountsWhatTheLibraryReadTakescovers 3,200 generated trees × 25 pointers: duplicate keys, alias keys and values,<<naming an alias, an inline mapping, a sequence or a scalar, several merges in one mapping, document nodes, odd pairs, escaped, empty and leading-zero tokens, and invalid pointers.costandreadboth end whereGetTargetends, andcostcounts what a transcription of the library's walk counts.FuzzTreeReads_ReadEndsWhereGetTargetEndsdraws the generator's seed. A minute reads about two million trees without a difference.TestTreeReads_ReadsAMergeChainAsDeepAsTheLibraryreads a chain 2^17 merged mappings deep whereGetTargetreads it, counted as the transcription counts it, in steps linear in the depth._PricesAMergeCyclePastTheLimitrefuses a hand-built cycle where it closes, and with no limit counts the most an int holds rather than wrapping negative._ReadsAMergeCycleTheLibraryEndsreads a cyclic tree whose read the library does end.TestTreeReads_HoldsAsTheLibraryReadsOneTokenholdsholdstoGetTarget's read of the one-token pointer on 1,600 generated trees.TestTreeReads_ReadCostsItsDepthNotItsWidth,TestTreeReads_Cost,_StopsPastTheLimitand_IndexesAMappingOnce.modelCost: a refused pointer is free, a decoy merge and an enum member are charged, and nothing is indexed past the limit.TestJoinTokens_KeepsTheBytesTheLibraryReadsandTestPartsOf.<<merges, and five aliased chains of 9,000 (45,000 merges). Diagnostics and exit code are identical tomain's on each, and so is the IR at 9,000 and 45,000.ceb1e00exits 1 on each with references on. With the alias budget off, 108,000 and 450,000 merges compile in-process with diagnostics identical tomain's.d0f84f8adds acycle-scan-failedwarning to each, and with references on anunresolved-ref.<<keys, each naming one 1,000-key mapping. At 4,000 chainsmaintakes 2.25 s; here the budget stops it at about 1,110 chains, in 1.03 s. At 1,000 chains both resolve everything.TestSettle_TheResumedWorkIsBoundedtries every limit short of the work two resumed chains take. Past it, no chain is resumed, each one left reports the budget at its$ref, and oneopenapi/budget-exceededlands at the document. AlsoTestSettle_AResumedReadIsChargedTheKeysItScans,TestResumeWork_ChargeandTestResumable_TheBudgetStopsWhatItWouldResume.TestMappings_AKeyWrittenTwiceIsPricedAsTheLibraryReadsItandTestMappings_PricedCountsTheLibrarysRead(model, tree and both).TestMappings_TheKeysTheResolverScansAreChargedrequires each target charged twice, once bychainEndsand once to resolve it.TestCheckUniqueKeys_NamesTheEarliestKeyWrittenAgain, including a later key repeated first and one repeated last.0f65658, each mutant built before it counted:navigation, 4 indefs, 9 inresolveand its callers, and 12 inload.d0f84f8: 15 in the<<stack, all red. 12 in the retried read and theHoldswiring: 10 red and 2 equivalent. One equivalent is the guard that keeps the whole read's answer once a step-by-step walk completes, which then ends where the whole read does. The other isretriedRead's error check: the library returns no target with an error. One of the 10 first survived, and its test now buildsMappingTargetsthrough the mappings pass.morphic compileon the 154 specs undertestdata/, references off and on (308 runs): IR, diagnostics and exit code identical tomain, at6a9f53d.morphic-harness testdata(no panic, invariants, round-trip, determinism, order invariance): output byte-identical tomain's.FuzzTreeReads_ReadEndsWhereGetTargetEnds,FuzzCompileandFuzzLowerSchemafor 60 s each,FuzzCycleDetectorfor 45 s, andFuzzInternalPointer_ScanAndLoweringReadFragmentsAlikefor 30 s, at6a9f53d;go test -raceonnavigation,defs,resolve,load,lowering,schema,openapitestandcompilers/openapi.BenchmarkCompile_Petstore, 12 interleaved runs each: a median of 1.57 ms here against 1.63 ms onmain, with allocations within 0.3%.go test ./...andmake fmt vet lint nolint-grammar nolint coverage-count coveragepass, and the coverage gate finds every statement covered. The full gate is left to CI.Timings:
/usr/bin/time morphic validate root.yamlon macOS, with each generator in its issue. Every column was run at the same time, on builds whose revisiongo version -mreports, unmodified.2191ed0ismainbefore #763, and0f65658is this branch beforenavigation. This branch is6a9f53d; the two commits after it add a test and change doc comments only.2191ed0main(25bdace)0f65658$refs into one extension$refs to as many top-level keys$defsrefs beside 20,000 root keys$refs to adefaultresponse's properties,compile, instructions$refpast 9,000 nested inline<<merges, refs onThe #773 shapes exit 1 on every build, because the lowering does not lower a
$refinto another document (#74). "Bound crossed" there means load left chains unresolved pastmaxResumeWork, as the known limit counts them. A$defsreference under adefaultresponse, with a discriminator mapping value to it:mainexits 1 with twounresolved-ref; this branch exits 0, as under"200".The last two rows have no issue of their own. The first nests 9,000
{<<: …}mappings underx-maround{D: {type: object}}, with one schema$refing#/x-m/D. The second gives a parameterQ10,000x-k{i}keys, makesPa$reftoQ, and maps a discriminator's values to#/components/parameters/P/x-k{i}. It exits 1 on every build, with onepass/discriminator-missing-variantper value, since no target is a variant of the base.