Use numeric tokens for faster index builds

[platypii]

Summary

• encode default ordinary, shape, and hex n-grams directly as numeric IDs
• avoid allocating token strings while scanning and decode only distinct tokens during output
• preserve the existing string path for custom n-gram configurations
• update hyparquet-writer to 0.16.3 for the INT32 delta fast path and corrected declarations

Benchmark

WildChat 25k-row sample, 250 row groups:

Build	Mean time	Change
Numeric prefix IDs, writer 0.16.1	25.67 s	baseline
Numeric tokens, writer 0.16.2	14.42 s	43.8% faster

Version 0.16.3 contains the declaration fix only, with no runtime change from the benchmarked writer version.

The generated index remained byte-identical: 22,157,787 bytes with SHA-256 ef5ecbcf79f2e7bc5591ac24c186365c9fab4da2a7481d9549acd25eda36a0e0.

CPU profiling reduced flushBlock from 3.05 s to 0.76 s, while ordinary and structural token extraction were roughly halved.

Validation

• npm test: 150 tests passed
• npm run lint
• npx tsc --noEmit
• numeric token equivalence tests cover representative input, deterministic random input, ordering, invalid codes, and the custom-options fallback