Skip to content

gh-153568: Use FIRST sets to speed up parser choices - #156209

Open
pablogsal wants to merge 1 commit into
python:mainfrom
pablogsal:gh-153568-first-set-dispatch
Open

gh-153568: Use FIRST sets to speed up parser choices#156209
pablogsal wants to merge 1 commit into
python:mainfrom
pablogsal:gh-153568-first-set-dispatch

Conversation

@pablogsal

@pablogsal pablogsal commented Aug 21, 2026

Copy link
Copy Markdown
Member

The parser currently tries every alternative in a choice, even when the next token rules most of them out. This uses FIRST sets from the grammar to skip that work. When only one alternative can match, it jumps straight there. When several can match, a small bitmask keeps them in grammar order. The optimization leaves nullable choices alone and is disabled for the invalid-rule pass, so syntax errors stay the same.

On my machine, parsing every Python file in Lib dropped from 2.180 seconds to 1.868 seconds, about 14.6%. That costs roughly 45 KB more text in the parser extension, or 7.4%. Part of gh-153568.

@bedevere-app bedevere-app Bot mentioned this pull request Aug 21, 2026
@pablogsal
pablogsal marked this pull request as ready for review August 22, 2026 12:45
@pablogsal
pablogsal requested a review from lysnikolaou as a code owner August 22, 2026 12:45
@pablogsal

pablogsal commented Aug 22, 2026

Copy link
Copy Markdown
Member Author

I grabbed some perf counters while parsing every parseable Python file under Lib. I alternated the two builds on the same pinned CPU. main is 999a046 and the PR build is d9cca76.

Metric main this PR Change
Parse time per Lib pass 2.215 s 1.891 s -14.8%
CPU cycles 33.91 B 28.83 B -14.8%
Instructions 106.15 B 91.88 B -13.5%
Branches 25.36 B 21.86 B -13.8%
Branch misses 82.05 M 80.85 M -1.5%
Branch miss rate 0.324% 0.370% +0.046 pp
Instructions per cycle 3.13 3.19 +1.8%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant