StackOverflowError while parsing large nested queries.

[manoharg]

Antlr4 fails to parse large nested queries like employee where city in ('a',b', 'c'...) or team in ('aa', 'bb', 'cc'...)' when the size of in clause list is > 500. From other forums and discussions, a solution is to increase the thread stack size but that's not an option for me in production. Is this fixed in the latest versions of antlr4 or any alternate solution?

Atlr4 version: 4.5.3

StackTrace

java.lang.StackOverflowError
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1529)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)

[KvanTTT]

Have you tried to upgrade ANTLR to the latest version? Antlr 4.5.3 is very old.

[manoharg]

yes, I tried upgrading to 4.7.3 and facing the same issue. After 4.7.3, I'm facing issues with grammar which I'm trying to fix. But from releasenotes, I couldn't find anything related to this.

[kaby76]

Also, what grammar are you using? Stack overflow is as much a problem with the grammar as the version of Antlr. For example, whether the grammar use kleene closure operators versus recusion?

[manoharg]

Our grammar uses recursion.

[ericvergnaud]

I suspect increasing the stack size is the only clean option. Recursion is key to Antlr’s parsing algorithm, and what you’re experiencing is not an Antlr bug, rather a deployment bug.I suspect your grammar has a rule like: … IN LPAR literal (COMMA literal)* RPARA dirty workaround worth trying could be to create a _10_literals rule as follows:_10_literals :literal COMMA literal COMMA literal COMMA literal COMMA literal COMMA literal COMMA literal COMMA literal COMMA literal COMMA literal COMMA;and change the enclosing rule as follows:… IN LPAR ( _10_literals* literal (COMMA literal)* RPARThis would divide the recursion depth by 10 and could solve your problem.Envoyé de mon iPhoneLe 4 nov. 2022 à 10:41, Manohar Golla ***@***.***> a écrit : yes, I tried upgrading to 4.7.3 and facing the same issue. After 4.7.3, I'm facing issues with grammar which I'm trying to fix. But from releasenotes, I couldn't find anything related to this. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[coderextreme]

How do i set stack size with antlr4-parse.exe?

[kaby76]

How do i set stack size with antlr4-parse.exe?

You can't. The code hardwires the order of the arguments to the call. https://github.com/antlr/antlr4-tools/blob/10a090aab6ec70217d1d8aa0c2f16c9f1e425861/antlr4_tool_runner.py#L137

$ antlr4-parse abbLexer.g4 abbParser.g4 module_ ../examples/robdata.sys; echo $?
0
04/03-08:46:53 ~/issues/g4-4040/abb/Generated-Java
$ antlr4-parse -Xmx500M abbLexer.g4 abbParser.g4 module_ ../examples/robdata.sys; echo $?
Exception in thread "main" java.nio.file.NoSuchFileException: -Xmx500M
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:236)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
        at java.base/java.nio.file.Files.readAllBytes(Files.java:3289)
        at java.base/java.nio.file.Files.readString(Files.java:3367)
        at java.base/java.nio.file.Files.readString(Files.java:3326)
        at org.antlr.v4.gui.Interpreter.interp(Interpreter.java:146)
        at org.antlr.v4.gui.Interpreter.main(Interpreter.java:277)
0

You'll have to call the tool the "old fashioned way".

Seems like a bug that if the tool can't open a file, it should set the exit code to 1.