Yarn.Compiler installed from NuGet crashes on CompileFile

[mgilski]

What is the current behavior?

Yarn installed from nuget crashes when trying to compile file.

Please provide the steps to reproduce, and if possible a minimal demo of the problem:

Install Yarn.Compiler from nuget and call Compiler.CompileFile with a sample yarn file.

Please tell us about your environment:

• Yarn Spinner Version: I was able to replicate the issue on both 1.1.0 and 1.2.0
• Unity Version: none, using API (with Godot 3.2.1, probably irrelevant)

Other information

Got rid of the error by compiling Yarn.Compiler project from source.

Error and stack trace:

E 0:00:01.933   void YarnSpinnerLexer.CreateIndentIfNeeded(Int32 ): System.MissingMethodException: Method not found: void Antlr4.Runtime.CommonToken..ctor(System.Tuple`2<Antlr4.Runtime.ITokenSource, Antlr4.Runtime.ICharStream>,int,int,int,int)
  <C++ Error>   Unhandled exception
  <C++ Source>  :0 @ void YarnSpinnerLexer.CreateIndentIfNeeded(Int32 )()
  <Stack Trace> :0 @ void YarnSpinnerLexer.CreateIndentIfNeeded(Int32 )()
                :0 @ void YarnSpinnerLexer.HEADER_NEWLINE_action(Antlr4.Runtime.RuleContext , Int32 )()
                :0 @ void YarnSpinnerLexer.Action(Antlr4.Runtime.RuleContext , Int32 , Int32 )()
                :0 @ void Antlr4.Runtime.Atn.LexerCustomAction.Execute(Antlr4.Runtime.Lexer )()
                :0 @ void Antlr4.Runtime.Atn.LexerActionExecutor.Execute(Antlr4.Runtime.Lexer , Antlr4.Runtime.ICharStream , Int32 )()
                :0 @ void Antlr4.Runtime.Atn.LexerATNSimulator.Accept(Antlr4.Runtime.ICharStream , Antlr4.Runtime.Atn.LexerActionExecutor , Int32 , Int32 , Int32 , Int32 )()
                :0 @ Int32 Antlr4.Runtime.Atn.LexerATNSimulator.FailOrAccept(Antlr4.Runtime.Atn.SimState , Antlr4.Runtime.ICharStream , Antlr4.Runtime.Atn.ATNConfigSet , Int32 )()
                :0 @ Int32 Antlr4.Runtime.Atn.LexerATNSimulator.ExecATN(Antlr4.Runtime.ICharStream , Antlr4.Runtime.Dfa.DFAState )()
                :0 @ Int32 Antlr4.Runtime.Atn.LexerATNSimulator.Match(Antlr4.Runtime.ICharStream , Int32 )()
                :0 @ Antlr4.Runtime.IToken Antlr4.Runtime.Lexer.NextToken()()
                :0 @ Antlr4.Runtime.IToken YarnSpinnerLexer.NextToken()()
                :0 @ Int32 Antlr4.Runtime.BufferedTokenStream.Fetch(Int32 )()
                :0 @ Boolean Antlr4.Runtime.BufferedTokenStream.Sync(Int32 )()
                :0 @ void Antlr4.Runtime.BufferedTokenStream.Consume()()
                :0 @ Antlr4.Runtime.IToken Antlr4.Runtime.Parser.Consume()()
                :0 @ Antlr4.Runtime.IToken Antlr4.Runtime.Parser.Match(Int32 )()
                :0 @ YarnSpinnerParser+HeaderContext YarnSpinnerParser.header()()
                :0 @ YarnSpinnerParser+NodeContext YarnSpinnerParser.node()()
                :0 @ YarnSpinnerParser+DialogueContext YarnSpinnerParser.dialogue()()
                :0 @ Yarn.Compiler.Status Yarn.Compiler.Compiler.CompileString(System.String , System.String , Yarn.Program& , System.Collections.Generic.IDictionary`2[System.String,Yarn.Compiler.StringInfo]& )()
                :0 @ Yarn.Compiler.Status Yarn.Compiler.Compiler.CompileFile(System.String , Yarn.Program& , System.Collections.Generic.IDictionary`2[System.String,Yarn.Compiler.StringInfo]& )()
                TestDialogue.cs:10 @ void TestDialogue.Test()()
                Game.cs:14 @ void Game._Ready()()

[desplesda]

Can you provide the Yarn file you were using?

[mgilski]

Sorry for the nonsense

title: Start
tags: 
colorID: 0
position: 583,233
---
Dog: Hello
Cat: Hi
Dog: Dog!
Cat: Cat!

[[ Dog |Dog]] 
[[ Cat |Cat]]
===
title: Dog
tags: 
colorID: 0
position: 196,285
---
Dog: Yaaaay, dog
Cat: Oh no
===
title: Cat
tags: 
colorID: 0
position: 889,319
---
Cat: Yaaaay, cat
Dog: Oh no
===

[desplesda]

Thank you! Will investigate.

[JoBiden]

Unity is known to have problems importing the .dll of nuget packages installed through visual studio - you have to manually add the .dll to unity

[mgilski]

Well, I'm not using Unity

[fcingolani]

Can confirm on the same scenario that @mgilski mentioned.

My dialog:

title: start
tags: 
colorID: 0
position: 361,205
---
omg stuff
===

Exception:

System.MissingMethodException: Method not found: void Antlr4.Runtime.CommonToken..ctor(System.Tuple`2<Antlr4.Runtime.ITokenSource, Antlr4.Runtime.ICharStream>,int,int,int,int)
  at YarnSpinnerLexer.CreateIndentIfNeeded (System.Int32 type) [0x00054] in <9d419cb6180b453c97c5942b14e16443>:0 
  at YarnSpinnerLexer.HEADER_NEWLINE_action (Antlr4.Runtime.RuleContext _localctx, System.Int32 actionIndex) [0x00004] in <9d419cb6180b453c97c5942b14e16443>:0 
  at YarnSpinnerLexer.Action (Antlr4.Runtime.RuleContext _localctx, System.Int32 ruleIndex, System.Int32 actionIndex) [0x0002d] in <9d419cb6180b453c97c5942b14e16443>:0 
  at Antlr4.Runtime.Atn.LexerCustomAction.Execute (Antlr4.Runtime.Lexer lexer) [0x00000] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Atn.LexerActionExecutor.Execute (Antlr4.Runtime.Lexer lexer, Antlr4.Runtime.ICharStream input, System.Int32 startIndex) [0x00067] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Atn.LexerATNSimulator.Accept (Antlr4.Runtime.ICharStream input, Antlr4.Runtime.Atn.LexerActionExecutor lexerActionExecutor, System.Int32 startIndex, System.Int32 index, System.Int32 line, System.Int32 charPos) [0x00042] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Atn.LexerATNSimulator.FailOrAccept (Antlr4.Runtime.Atn.SimState prevAccept, Antlr4.Runtime.ICharStream input, Antlr4.Runtime.Atn.ATNConfigSet reach, System.Int32 t) [0x00014] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Atn.LexerATNSimulator.ExecATN (Antlr4.Runtime.ICharStream input, Antlr4.Runtime.Dfa.DFAState ds0) [0x000b7] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Atn.LexerATNSimulator.Match (Antlr4.Runtime.ICharStream input, System.Int32 mode) [0x0004c] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Lexer.NextToken () [0x0008f] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at YarnSpinnerLexer.NextToken () [0x0009c] in <9d419cb6180b453c97c5942b14e16443>:0 
  at Antlr4.Runtime.BufferedTokenStream.Fetch (System.Int32 n) [0x0000e] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.BufferedTokenStream.Sync (System.Int32 i) [0x00014] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.BufferedTokenStream.Consume () [0x00059] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Parser.Consume () [0x00016] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at Antlr4.Runtime.Parser.Match (System.Int32 ttype) [0x0001c] in <a5072916b6fa4dfe9966300b7034554c>:0 
  at YarnSpinnerParser.header () [0x00074] in <9d419cb6180b453c97c5942b14e16443>:0 
  at YarnSpinnerParser.node () [0x0004c] in <9d419cb6180b453c97c5942b14e16443>:0 
  at YarnSpinnerParser.dialogue () [0x000a3] in <9d419cb6180b453c97c5942b14e16443>:0 
  at Yarn.Compiler.Compiler.CompileString (System.String text, System.String fileName, Yarn.Program& program, System.Collections.Generic.IDictionary`2[System.String,Yarn.Compiler.StringInfo]& stringTable) [0x0003a] in <9d419cb6180b453c97c5942b14e16443>:0 
  at YarnEditorImportPlugin.Import (System.String sourceFile, System.String save_path, Godot.Collections.Dictionary options, Godot.Collections.Array r_platform_variants, Godot.Collections.Array r_gen_files) [0x0005f] in <da64331e5c7249b9811ac33885a82f81>:0 

[desplesda]

I'm not able to reproduce this issue in develop; hopefully, once we deploy the 2.0 release to NuGet, this issue will be resolved. I'll close this issue now; if you continue to see it, please let me know!

[TheOneTrueZandra]

I've tracked down the "source" of this crash, and would suggest re-opening it and fixing it.

I checked out the YarnSpinner build that's currently on Nuget using a decompiler, and it looks like the issue lies with this line of code in YarnSpinnerLexer.

Basically, this line compiles out to using System.Tuple, rather than Antlr4's Antlr4.Runtime.Sharpen.Tuple.

On some platforms/.NET versions, it seems like the context of the function itself using Antlr4's Tuple is enough to inform the runtime to use that type rather than System.Tuple. On other platforms, the runtime tries to use System.Tuple, resulting in an error, as there is no CommonToken constructor that accepts a System.Tuple as the first argument.

In theory, a simple fix could just be to explicitly state that the Tuple type used/created there is Antlr4's Tuple, which should inform any runtimes which type is actually being used, and thus eliminate any runtime-specific errors.

I checked develop's source as well, and found the equivalent line. No idea if it still happens there (I don't know how to get that through nuget), but if what I'm guessing here is true, it probably fails there as well.

I'm confused as to how this compiles to that at all, honestly, because I'd expect the C# compiler would complain about passing in a System.Tuple instead of Antlr4's at compile time, but, alas. Here we are.

EDIT : Looks like more recent versions of Antlr4 use System.Tuple, so an even easier fix could be to simply update the Antlr4 dependency for the Nuget package.

[sesopenko]

I was experiencing this problem in YarnSpinner 1.2 and switched to YarnSpinner 2.0.0-beta0006 and the problem went away. The library's changed substantially for 2.0, but I managed to get it to compile a yarn file successfully without the missing method exception noted in the original report.

[TheOneTrueZandra]

Good to know! Glad to see it's fixed on 2.0

[sesopenko]

I spoke too soon, and ran into a Tuple exception once I introduced options to my 2.0 syntax yarnfile. When it only has nodes with simple lines, it compiles and runs properly. I'm refactoring my Godot project in a way which makes this easier to troubleshoot and will share more details soon.

[sesopenko]

On newer target frameworks, such as 4.7.2 for Godot Mono projects, the issue occurs. If I clone this repo, change YarnSpinnerCompiler.csproj to target net472 then the unit tests get the error during build.

I'm forking the repo and will submit a PR with a fix for the develop branch (2.0 beta).

FYI @desplesda, @TheOneTrueZandra

[sesopenko]

Ugh never mind, when the library targets netstandard2.0 then Antlr4.Runtime.Sharpen.Tuple doesn't exist in the Antlr4 library. This isn't going to be easy to fix, because YarnSpinner is forced to upgrade their targeted framework, it would result in lots of problems for other dependant projects.

I don't have the time to dig into a solution for this and will look at other libraries instead for now, such as Ink.

Here's a commit to a fork I made, demonstrating the problem with a fix. I didn't spin up a PR because it would be backwards incompatible, jumping all the way from 2.0 to 4.7.2, and would probably break a number of dependant projects.

This should be reopened though, because dot.net 2.0 is no longer supported and projects targeting the newer frameworks are going to have a bad time. Myself, I'm moving on, and looking into other libs, such as Ink.

Example commit here

[desplesda]

Re-opening this issue; the community efforts to improve Godot support are running into this, and I want to keep this on my radar. Thanks to all who've helped identify so much about this problem!

[dogboydog]

This is where I'm hitting this error in YarnSpinner-Godot

Method not found: void Antlr4.Runtime.CommonToken..ctor(
System.Tuple`2<Antlr4.Runtime.ITokenSource, Antlr4.Runtime.ICharStream>,int,int,int,int)
 @16392
	IndentAwareLexer.InsertToken()
	at [...]\decompiler\2b0ae3b2904644bbb9df20e53100124d3d200\22\4e13dad1\IndentAwareLexer.cs:line 164
	IndentAwareLexer.HandleNewLineToken()at [...]\decompiler\2b0ae3b2904644bbb9df20e53100124d3d200\22\4e13dad1\IndentAwareLexer.cs:line 113
	IndentAwareLexer.CheckNextToken()
	IndentAwareLexer.NextToken()
	BufferedTokenStream.Fetch()
	BufferedTokenStream.Sync()
	BufferedTokenStream.Consume()
	Parser.Consume()
	Parser.Match()
	YarnSpinnerParser.line_formatted_text()
	YarnSpinnerParser.line_statement()
	YarnSpinnerParser.shortcut_option()
	YarnSpinnerParser.shortcut_option_statement()
	YarnSpinnerParser.statement()
	YarnSpinnerParser.body()
	YarnSpinnerParser.node()
	YarnSpinnerParser.dialogue()
	Compiler.ParseSyntaxTree()
	Compiler.ParseSyntaxTree()
	Compiler.Compile()
	YarnProjectUtility.CompileAllScripts()
	YarnProjectUtility.UpdateYarnProject()
	YarnImporter.ImportYarn()
	YarnImporter.Import()

<Thread Pool> @18880