path_compiler: clamp the pre-allocation hint in ParsePath - #289
Conversation
ParsePath sizes its slice from the separator count of the caller's path,
before any component has been validated:
parts := make([]string, 0, 1+strings.Count(jsonPath, ".")+strings.Count(jsonPath, "["))
A path consisting only of separators passes the cheap pre-checks and
reserves 16 bytes per separator, then is rejected on the first component,
so none of the reserved memory is used.
Clamp the hint. Capacity is only a hint to append, which still grows as
needed, so this cannot change which paths are accepted or what a
successful parse returns.
ParsePath on 200000 separators, rejected on the first component:
before 127667 ns/op 3203087 B/op 1 allocs/op
after 11761 ns/op 9472 B/op 1 allocs/op
Paths below the clamp are unaffected. The added test uses literal
integers rather than the new constant, so it passes with and without
this change.
Change-Id: I0cb10f6d6266a2650d323d4f39aa8e2522b3e753
The constant ceiling in the previous commit regressed valid deep paths. I
benchmarked it rather than assuming, on a well-formed 2000-component path:
upstream, no clamp 32781 B/op 1 alloc/op
constant 512 ceiling 113177 B/op 5 allocs/op <- 3.45x worse
proportional len/2+1 32788 B/op 1 alloc/op
A 512-element ceiling under-reserves any path with more components than that, so
append regrows repeatedly and the common case pays for the hostile one. Deep
paths are unusual but they are legal, and a defensive bound should not make valid
input worse.
The shortest component that can contribute a separator is two bytes ("k."), so
len(jsonPath)/2+1 can never under-reserve a well-formed path, while still
refusing to size the allocation from a long run of separators. The hostile case
is unaffected by the change:
200000 separators, rejected on the first component:
upstream 1606185 B/op -> 803352 B/op
The existing correctness test could not catch this: it passed with the constant
too, because clamping never changes what ParsePath returns. So the property now
has an allocation assertion of its own,
TestParsePathHintDoesNotRegressDeepPaths, which fails on the constant version
with "used 5 allocations, want 1" and passes here.
Verified that ./... behaves identically to unpatched upstream: the two
TestOracleSetPr286Regression subtests fail on a clean checkout as well, so they
are pre-existing and unrelated to this change.
Change-Id: I43e5ed1e205c70fb55dde128a4ab817e3d359d23
|
Correcting my own patch before you spend time on it: the constant ceiling I Well-formed 2000-component path, same clone and machine,
A 512 ceiling under-reserves any path with more components than that, so The hostile case is essentially unchanged by the correction — 200,000 separators Worth flagging why this slipped past my own test: the existing correctness test One note for reproducing: |
Problem
ParsePathsizes its slice from the separator count of the caller's string, before any component has been validated:A path consisting only of separators passes the two cheap pre-checks and reserves 16 bytes per separator —
stringis a two-word header — and is then rejected on the first component, so none of the reserved memory is used.Change
Clamp the hint. Capacity is only a hint to
append, which still grows as needed, so this cannot change which paths are accepted or what a successful parse returns.Measurements
ParsePathon 200000 separators, rejected on the first component (same clone, same machine,-benchtime 300x, Apple M3 Pro, go1.22.5):(Two runs on different clones gave 144736/8219 ns and 127667/11761 ns; the byte
figures were stable at 3203085-3203087 and 9472.)
Paths below the clamp are unaffected.
Tests
TestParsePathHintClampPreservesResultsasserts identical results for 1, 2, 511, 512, 513 and 2600 dot-separated components — well past the clamp, whereappendmust grow — plus a 1500-element bracket-notation path, and that malformed paths ("",".","..",".a","a..b","a[","a]") are still rejected. It uses literal integers rather than the new constant, so it compiles and passes on an unpatched tree as well; I checked that, so it is testing behaviour rather than the patch.gofmtis clean. The package has the same 728 passing tests before and after the change. Two things I should mention rather than have you discover them:go vetreports oneunreachable codewarning atparser.go:2023, andTestOracleSetPr286Regressionfails — both reproduce identically on an unpatched tree here and are unrelated to path parsing.Notes
512is a judgement call, not derived from anything in the path grammar — happy to change it, inline the comparison, or drop the helper if you would prefer a smaller diff. I have not tried to show a practical denial of service in a real caller; the claim is limited to what the benchmark shows, which is allocation proportional to caller-supplied input on a parse that is rejected immediately.Disclosure: this was found by a small static checker I wrote for this pattern, and the patch was written with AI assistance. I ran and verified every number above myself, including the before/after on the same clone and the check that the new test passes without the change.