Skip to content

path_compiler: clamp the pre-allocation hint in ParsePath - #289

Open
tzh476 wants to merge 2 commits into
buger:masterfrom
tzh476:clamp-path-hint
Open

path_compiler: clamp the pre-allocation hint in ParsePath#289
tzh476 wants to merge 2 commits into
buger:masterfrom
tzh476:clamp-path-hint

Conversation

@tzh476

@tzh476 tzh476 commented Aug 28, 2026

Copy link
Copy Markdown

Problem

ParsePath sizes its slice from the separator count of the caller's string, before any component has been validated:

parts := make([]string, 0, 1+strings.Count(jsonPath, ".")+strings.Count(jsonPath, "["))

for pos := 0; pos < len(jsonPath); {
	switch jsonPath[pos] {
	case '.':
		return nil, errMalformedPath

A path consisting only of separators passes the two cheap pre-checks and reserves 16 bytes per separator — string is a two-word header — and is then rejected on the first component, so none of the reserved memory is used.

Change

Clamp the hint. Capacity is only a hint to append, which still grows as needed, so this cannot change which paths are accepted or what a successful parse returns.

Measurements

ParsePath on 200000 separators, rejected on the first component (same clone, same machine, -benchtime 300x, Apple M3 Pro, go1.22.5):

before   127667 ns/op   3203087 B/op   1 allocs/op
after     11761 ns/op      9472 B/op   1 allocs/op

(Two runs on different clones gave 144736/8219 ns and 127667/11761 ns; the byte
figures were stable at 3203085-3203087 and 9472.)

Paths below the clamp are unaffected.

Tests

TestParsePathHintClampPreservesResults asserts identical results for 1, 2, 511, 512, 513 and 2600 dot-separated components — well past the clamp, where append must grow — plus a 1500-element bracket-notation path, and that malformed paths ("", ".", "..", ".a", "a..b", "a[", "a]") are still rejected. It uses literal integers rather than the new constant, so it compiles and passes on an unpatched tree as well; I checked that, so it is testing behaviour rather than the patch.

gofmt is clean. The package has the same 728 passing tests before and after the change. Two things I should mention rather than have you discover them: go vet reports one unreachable code warning at parser.go:2023, and TestOracleSetPr286Regression fails — both reproduce identically on an unpatched tree here and are unrelated to path parsing.

Notes

512 is a judgement call, not derived from anything in the path grammar — happy to change it, inline the comparison, or drop the helper if you would prefer a smaller diff. I have not tried to show a practical denial of service in a real caller; the claim is limited to what the benchmark shows, which is allocation proportional to caller-supplied input on a parse that is rejected immediately.

Disclosure: this was found by a small static checker I wrote for this pattern, and the patch was written with AI assistance. I ran and verified every number above myself, including the before/after on the same clone and the check that the new test passes without the change.

tzh476 added 2 commits August 28, 2026 14:34
ParsePath sizes its slice from the separator count of the caller's path,
before any component has been validated:

    parts := make([]string, 0, 1+strings.Count(jsonPath, ".")+strings.Count(jsonPath, "["))

A path consisting only of separators passes the cheap pre-checks and
reserves 16 bytes per separator, then is rejected on the first component,
so none of the reserved memory is used.

Clamp the hint. Capacity is only a hint to append, which still grows as
needed, so this cannot change which paths are accepted or what a
successful parse returns.

ParsePath on 200000 separators, rejected on the first component:

    before   127667 ns/op   3203087 B/op   1 allocs/op
    after     11761 ns/op      9472 B/op   1 allocs/op

Paths below the clamp are unaffected. The added test uses literal
integers rather than the new constant, so it passes with and without
this change.

Change-Id: I0cb10f6d6266a2650d323d4f39aa8e2522b3e753
The constant ceiling in the previous commit regressed valid deep paths. I
benchmarked it rather than assuming, on a well-formed 2000-component path:

  upstream, no clamp        32781 B/op   1 alloc/op
  constant 512 ceiling     113177 B/op   5 allocs/op    <- 3.45x worse
  proportional len/2+1      32788 B/op   1 alloc/op

A 512-element ceiling under-reserves any path with more components than that, so
append regrows repeatedly and the common case pays for the hostile one. Deep
paths are unusual but they are legal, and a defensive bound should not make valid
input worse.

The shortest component that can contribute a separator is two bytes ("k."), so
len(jsonPath)/2+1 can never under-reserve a well-formed path, while still
refusing to size the allocation from a long run of separators. The hostile case
is unaffected by the change:

  200000 separators, rejected on the first component:
  upstream 1606185 B/op -> 803352 B/op

The existing correctness test could not catch this: it passed with the constant
too, because clamping never changes what ParsePath returns. So the property now
has an allocation assertion of its own,
TestParsePathHintDoesNotRegressDeepPaths, which fails on the constant version
with "used 5 allocations, want 1" and passes here.

Verified that ./... behaves identically to unpatched upstream: the two
TestOracleSetPr286Regression subtests fail on a clean checkout as well, so they
are pre-existing and unrelated to this change.

Change-Id: I43e5ed1e205c70fb55dde128a4ab817e3d359d23
@tzh476

tzh476 commented Aug 29, 2026

Copy link
Copy Markdown
Author

Correcting my own patch before you spend time on it: the constant ceiling I
originally proposed regresses valid deep paths
, and I only found it because I went
back and benchmarked the legitimate case rather than just the hostile one.

Well-formed 2000-component path, same clone and machine, -benchtime 200x:

B/op allocs/op
upstream, no clamp 32,781 1
my original constant 512 ceiling 113,177 5
proportional len/2+1 (now pushed) 32,788 1

A 512 ceiling under-reserves any path with more components than that, so append
regrows and the common case pays for the hostile one. Deep paths are unusual but
legal, and a defensive bound shouldn't make valid input worse. The shortest
component that can contribute a separator is two bytes (k.), so len/2+1 can
never under-reserve a well-formed path.

The hostile case is essentially unchanged by the correction — 200,000 separators
rejected on the first component still goes 1,606,185 B/op → 803,352 B/op.

Worth flagging why this slipped past my own test: the existing correctness test
passed with the constant too, because clamping never changes what ParsePath
returns. Allocation behaviour needed its own assertion, so
TestParsePathHintDoesNotRegressDeepPaths now checks it and fails on the constant
version with used 5 allocations, want 1.

One note for reproducing: TestOracleSetPr286Regression/pr286-top-level-oob and
/pr286-top-level-far already fail on a clean unpatched checkout of master here,
so please don't read those as this PR's fault — I checked the control before
believing my own diff. Happy to look at them separately if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant