Repository navigation

[improvement](be) Move jieba into the Doris tree and cut chinese text over a flat dictionary - #70266

Draft

airborne12 wants to merge 10 commits into

apache:masterfrom

airborne12:snii-analysis-t7-chinese

Draft

[improvement](be) Move jieba into the Doris tree and cut chinese text over a flat dictionary#70266
airborne12 wants to merge 10 commits into
apache:masterfrom
airborne12:snii-analysis-t7-chinese

Commits on Nov 3, 2026
- [test](be) Add a tokenizer benchmark that also digests every row's tokens
  
  airborne12
  and
  claude
  committed
  6100f8e
- [fix](be) Fix three analyzer bugs that keep the analysis output from being frozen
  
  airborne12
  and
  claude
  committed
  3aa8e31
- [test](be) Freeze the output of every text analyzer and ANN index file
  
  airborne12
  and
  claude
  committed
  6e41ceb
- [refactor](be) Add the CLucene-free text analysis interface and its two CLucene adapters
  
  airborne12
  and
  claude
  committed
  5271950
- [refactor](be) Switch every consumer of text analysis to the TextAnalyzer interface
  
  airborne12
  and
  claude
  committed
  25ead53
- [refactor](be) Move every analysis component onto the CLucene-free stream interface
  
  airborne12
  and
  claude
  committed
  d90ab96
Commits on Nov 4, 2026
- [improvement](be) Tokenize english and basic with a vectorized byte classifier
  
  airborne12
  and
  claude
  committed
  23d1e36
- [improvement](be) Tokenize standard and unicode with a dense UAX#29 table
  
  airborne12
  and
  claude
  committed
  50041fb
- [refactor](be) Move jieba into the Doris tree for the chinese parser
  
  airborne12
  and
  claude
  committed
  82aa141
- [improvement](be) Cut chinese text over a flat jieba dictionary
  
  airborne12
  and
  claude
  committed
  058fb5c