tokenizer.rb:17: [BUG] Segmentation fault

[nocnob]

git repository: https://github.com/ceshiyixiaba/languages

docker enviroment，error message:

docker run --user=1000 --rm -v $(pwd):$(pwd) -w $(pwd) -t linguist | more
/usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/tokenizer.rb:17: [BUG] Segmentation fault at 0x0000004700000057
ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux-musl]

-- Control frame information -----------------------------------------------
c:0026 p:---- s:0154 e:000153 CFUNC  :extract_tokens
c:0025 p:0007 s:0149 e:000148 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/tokenizer.rb:17
c:0024 p:0043 s:0144 e:000143 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/classifier.rb:106
c:0023 p:0031 s:0136 e:000135 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/classifier.rb:84
c:0022 p:0043 s:0129 e:000128 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/classifier.rb:22
c:0021 p:0011 s:0122 e:000121 BLOCK  /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist.rb:32
c:0020 p:0026 s:0119 e:000118 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist.rb:100
c:0019 p:0026 s:0113 e:000112 BLOCK  /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist.rb:31 [FINISH]
c:0018 p:---- s:0108 e:000107 CFUNC  :each
c:0017 p:0020 s:0104 e:000103 BLOCK  /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist.rb:29
c:0016 p:0026 s:0099 e:000098 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist.rb:100
c:0015 p:0042 s:0093 e:000092 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist.rb:24
c:0014 p:0026 s:0086 e:000085 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/blob_helper.rb:368
c:0013 p:0041 s:0082 e:000081 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/lazy_blob.rb:70
c:0012 p:0030 s:0077 e:000076 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/blob_helper.rb:383
c:0011 p:0004 s:0073 e:000072 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/repository.rb:174
c:0010 p:0135 s:0066 e:000065 BLOCK  /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/repository.rb:164 [FINISH]
c:0009 p:---- s:0057 e:000056 CFUNC  :each_delta
c:0008 p:0158 s:0053 e:000052 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/repository.rb:149
c:0007 p:0038 s:0044 e:000043 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/repository.rb:116
c:0006 p:0032 s:0040 E:000ea8 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/lib/linguist/repository.rb:68
c:0005 p:0176 s:0035 E:001bd8 METHOD /usr/local/bundle/gems/github-linguist-7.20.0/bin/github-linguist:52
c:0004 p:0112 s:0016 e:000015 TOP    /usr/local/bundle/gems/github-linguist-7.20.0/bin/github-linguist:142 [FINISH]
c:0003 p:---- s:0013 e:000012 CFUNC  :load
c:0002 p:0112 s:0008 E:002000 EVAL   /usr/local/bundle/bin/github-linguist:23 [FINISH]
c:0001 p:0000 s:0003 E:000810 (none) [FINISH]

I tested different versions:

• github-linguist 7.12.2: OK
• github-linguist 7.13.0: Error

$ /var/lib/gems/2.7.0/gems/github-linguist-7.12.2/bin/git-linguist stats --commit=`git rev-parse HEAD`
{"OpenEdge ABL":116388}
$ /var/lib/gems/2.7.0/gems/github-linguist-7.13.0/bin/git-linguist stats --commit=`git rev-parse HEAD`
free(): invalid next size (normal)
[1]    3231762 abort (core dumped)  /var/lib/gems/2.7.0/gems/github-linguist-7.13.0/bin/git-linguist stats

$ ruby -v
ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux-gnu]
$ uname -a
Linux VM-0-10-ubuntu 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

[lildude]

🤔 7.13 was the first release after @smola made changes to the tokenizer in #5205 and #5186 and #5061

@smola any idea what's going on here?

[smola]

@lildude I can reproduce the issue. It seems to be related to very long tokens. I'm trying to identify the root cause and fix it.

[lildude]

I've created #5938 from this as it's a bug which is now affecting GitHub Enterprise Server customers. Have you had any further luck @smola?