Refactor hash calculation and token emission logic for improved clarity and performance

[SashaXser]

whichlang/inference_short
                        time:   [227.00 ns 228.02 ns 229.51 ns]
                        thrpt:  [120.50 MiB/s 121.29 MiB/s 121.84 MiB/s]
                 change:
                        time:   [-12.746% -12.392% -11.959%] (p = 0.00 < 0.05)
                        thrpt:  [+13.583% +14.144% +14.608%]
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
whichlang/inference_long
                        time:   [4.7891 µs 4.8083 µs 4.8306 µs]
                        thrpt:  [121.02 MiB/s 121.58 MiB/s 122.07 MiB/s]
                 change:
                        time:   [-15.395% -14.725% -14.222%] (p = 0.00 < 0.05)
                        thrpt:  [+16.581% +17.267% +18.196%]
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe
whichlang/inference_jp_short
                        time:   [227.39 ns 228.24 ns 229.36 ns]
                        thrpt:  [224.53 MiB/s 225.63 MiB/s 226.48 MiB/s]
                 change:
                        time:   [-9.2561% -8.7232% -7.9788%] (p = 0.00 < 0.05)
                        thrpt:  [+8.6706% +9.5569% +10.200%]
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  2 (2.00%) high mild
  8 (8.00%) high severe
whichlang/inference_jp_medium
                        time:   [2.4745 µs 2.4780 µs 2.4820 µs]
                        thrpt:  [248.22 MiB/s 248.62 MiB/s 248.97 MiB/s]
                 change:
                        time:   [-7.4750% -7.0078% -6.5422%] (p = 0.00 < 0.05)
                        thrpt:  [+7.0002% +7.5359% +8.0789%]
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe