Improve MetalLexer Performance Using Combined Regular Expression

[pracheeeeez]

The current implementation of MetalLexer iterates through a list of token patterns (TOKENS) and applies individual regex matches sequentially. While functional, this approach can lead to suboptimal performance due to multiple regex evaluations for each position in the input code.

To enhance performance, I propose combining all token patterns into a single master regex using named capturing groups. This approach reduces the overhead of repeated regex evaluations and improves the maintainability of the code.

Current Implementation

for token_type, pattern in TOKENS:
regex = re.compile(pattern)
match = regex.match(self.code, pos)
# Logic for handling the match

Proposed Improvement
Combine all token patterns into a single regex using named capturing groups. Each pattern will be associated with its corresponding token type:
TOKENS_PATTERN = "|".join(
f"(?P<{token_type}>{pattern})" for token_type, pattern in TOKENS
)
MASTER_REGEX = re.compile(TOKENS_PATTERN)

This allows the lexer to match all token types in one pass and determine the matched token using match.lastgroup.

@NripeshN Can I work on this?

[Manya123-max]

I want to work on this

[ManeeshDevarasetty]

Can u assign me this work?

[raghulchandramouli]

@CrossGL-issue-bot assign me

[CrossGL-issue-bot]

The issue you are trying to assign to yourself is already assigned.

[huntermarchi]

@CrossGL-issue-bot assign me this issue i will fix this easily

[CrossGL-issue-bot]

The issue you are trying to assign to yourself is already assigned.

[KeerthiKeswaran]

I'll work on this, assign this to me.