gh-153568: Use FIRST sets to speed up parser choices - #156209
Open
pablogsal wants to merge 1 commit into
Open
Conversation
Member
Author
|
I grabbed some perf counters while parsing every parseable Python file under
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The parser currently tries every alternative in a choice, even when the next token rules most of them out. This uses FIRST sets from the grammar to skip that work. When only one alternative can match, it jumps straight there. When several can match, a small bitmask keeps them in grammar order. The optimization leaves nullable choices alone and is disabled for the invalid-rule pass, so syntax errors stay the same.
On my machine, parsing every Python file in Lib dropped from 2.180 seconds to 1.868 seconds, about 14.6%. That costs roughly 45 KB more text in the parser extension, or 7.4%. Part of gh-153568.