Skip to content

Add hex shape n-grams - #10

Merged
platypii merged 1 commit into
masterfrom
hex-shape-ngrams
Jul 31, 2026
Merged

Add hex shape n-grams#10
platypii merged 1 commit into
masterfrom
hex-shape-ngrams

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Summary

  • add a metadata-gated hexadecimal shape projection for UUID and hash-like regexes
  • increase ordinary shape windows from 4 to 8 symbols for better date, SSN, phone, and card pruning
  • extract ordinary and hexadecimal shapes in one source traversal
  • preserve compatibility with indexes that do not contain either shape layer

Performance

On the local 25k-row WildChat sample, the final index is 2.0% larger than the shape-4 index; the hex layer itself adds 0.19% over shape-8. Candidate blocks changed as follows:

  • slash date: 151 to 76
  • SSN: 1,401 to 6
  • phone: 1,401 to 42
  • credit card: 1,401 to 14
  • UUID: 2,500 to 49

Common dash dates and unseparated 40-character hex strings remain unselective on this corpus.

Verification

  • npm test (142 tests)
  • npm run lint

@platypii
platypii merged commit b6cb986 into master Jul 31, 2026
6 checks passed
@platypii
platypii deleted the hex-shape-ngrams branch July 31, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant