Repository navigation
[improvement](be) Rewrite the tokens of a batch in place in the one-to-one filters - #70269
Draft
airborne12 wants to merge 13 commits into
Draft
airborne12 wants to merge 13 commits into
airborne12 wants to merge 13 commits into
Conversation
…kens
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is moving off CLucene one analyzer at a
time, and every step has to show two things against the code before it: that it
is faster, and that every row still gives the same tokens. There was no tool for
either.
benchmark_analyzer.hpp measures every builtin parser and a few custom chains on
the path the SNII writer takes for a row (a new stream per row) and with one
stream across rows, over corpora given as `.rows` files. With
ANALYZER_BENCH_DIGEST set it registers only the benchmark "digest", which writes
one line per analyzer, corpus and row: the number of terms and a digest of their
bytes and positions. Diffing the files of two binaries names every row whose
tokens changed, on real text and with the real kuromoji dictionary, which the
unit tests cannot hold.
analyzer_ab.py runs two binaries in alternating rounds and reports the median of
the per-round throughput ratios, because the host load moves a single
measurement by 10 to 20 percent. analyzer_corpus.py builds the corpora; the
Chinese and Japanese ones are cut from the Wikipedia CirrusSearch dumps of
2025-12-29, so the same date gives the same corpus. analyzer_benchmark.md lists
the steps.
The analysis rewrite is measured against this commit, branch
snii-analysis-bench-baseline.
### Release note
None
### Check List (For Author)
- Test: Manual test
- BUILD_TYPE=RELEASE DORIS_BENCHMARK_HEADER=benchmark_analyzer.hpp
./build.sh --benchmark builds benchmark_test.
- The digest mode over six corpora of 8 MB (four mixed ones, zh_wiki and
ja_wiki) writes 5,240,060 lines in 67 s. A build with the separate
dynamic_cast fix of the tokenizer base class, which keeps every token,
gives a byte-identical file.
- analyzer_ab.py, five rounds: that fix measures 1.84 on basic and 1.64 on
custom standard + lowercase. The A/A run stayed within 0.94 to 1.08 per
analyzer on a host at load 140 to 280, more than the 5% the method
allows, so finer comparisons need a quieter host.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…being frozen
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Freezing the output of every text analyzer, in the next commit,
ran into three bugs that no frozen digest can hold.
1. The word_delimiter filter threw std::out_of_range for a token holding a NUL
byte when catenate_words, catenate_numbers or catenate_all was set.
WordDelimiterConcatenation::append called std::string::append(const char*,
pos, n), which resolves to the string_view overload and measures the text up
to its first NUL, so for "foo\0bar" the second part started past the end.
The SNII and V1/V2 writers catch only CLuceneError and doris::Exception, so
the error left the writer. The part is now appended from its offset.
2. IKAnalyzer::tokenStream created its tokenizer with new and then called
reset(), which throws when a value starts with a character cut short, such as
"\xE4\xB8". The tokenizer and its buffers, about 100 KB, leaked once per such
value, on the query path and on the SNII write path, which build a stream per
value. A unique_ptr now owns it until reset() returns.
3. The IK dictionary was loaded under std::call_once, which is used up even when
loading fails. After one failure, for example a dictionary path that is fixed
later, every IK analyzer kept failing until the BE restarted, and in the unit
tests the result of the IK tests depended on whether a test with a bad path
ran first. Dictionary::initial now publishes the dictionary only after it has
loaded, and the next call tries again. reload() and getSingleton(cfg), which
only existed to recover from that state, are removed.
### Release note
Fix the word_delimiter filter failing on values with a NUL byte when a catenate
option is set, and a memory leak of the IK analyzer on values it cannot decode.
### Check List (For Author)
- Test: Unit Test
- New: WordDelimiterFilterTest.CatenatePartsAcrossNul, and
IKTokenizerTest.TestUndecodableValueThrowsWithoutLeakingTheStream, whose
leak LeakSanitizer reports when the binary exits. Changed:
IKTokenizerTest.TestDictionaryExceptionHandling checks that a failed load
is retried and that the first loaded configuration stays.
- ./run-be-ut.sh --run --filter='WordDelimiterFilterTest*:IKTokenizerTest*'
passes on ASAN, also three times with GTEST_SHUFFLE=1 together with the
golden tests of the next commit.
- Behavior changed: Yes. A value with a NUL byte under word_delimiter with a
catenate option is indexed instead of failing; a failed IK dictionary load is
retried by the next IK analyzer instead of failing until a restart.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is about to be rewritten without
CLucene. A change in what an analyzer emits raises no error anywhere: queries
just stop finding the rows indexed with the old tokens. So the output is frozen
before any implementation changes, and every later step has to keep it.
AnalysisGolden runs 155 analyzer configurations over 13 corpus families. The
configurations are written the way users write them, as index properties and
analyzer, normalizer, tokenizer and filter policies, so the matrix does not move
when the implementation behind it does: every builtin parser with its lower_case
and stopwords values, every custom tokenizer, token filter and char filter with
its settings (pairwise covering arrays for the 16 pinyin and 9 word_delimiter
switches), the index's own char filter, normalizers, and the normalize() of
prefix and wildcard queries. The families aim at known risks: token windows at
255 bytes and at 255 UTF-16 code units, batch sizes of 255 to 513 tokens,
characters across 4096-byte chunks, invalid UTF-8, sampled jieba and IK
dictionary words, unknown Han strings, every pair of word break classes, sampled
BMP code points, NUL bytes and long lines. Each configuration runs with a new
stream per line, as queries and the SNII writer read values, and with one stream
across lines, as the V1/V2 writers do. Both must give the same XXH3 digest, the
one frozen in analysis_golden.txt; on the current code no configuration
disagrees. ANALYSIS_GOLDEN_FULL=1 scans every code point against
analysis_golden_full.txt. A test checks that every component AnalysisFactoryMgr
registers is in some configuration, and that the ICU analyzers give the same
output with Turkish as the default locale. The kuromoji configurations use a
small dictionary the test builds, because the real one is not in the
repository. The 4030 jobs run on up to 16 threads.
AnalysisDifferential compares the production english, standard, unicode,
custom standard and chinese analyzers with their old implementations, which stay
linked into the unit tests, on the same families and on one second of random
input, and shows that an injected difference is caught.
AnalysisWriteGolden freezes what the writers make of the tokens: the bytes of
the SNII file, and, because V1/V2 files carry timestamps, the terms with their
documents and positions read back, for chinese, ik, icu and a custom chain.
AnnWriteGolden freezes every sub-file of HNSW, IVF and IVF_ON_DISK indexes in V2
and in SNII, built on one OpenMP thread from vectors whose distances are exact
in float, and written twice to show the bytes repeat.
Lines of the digest files are only added: ANALYSIS_GOLDEN_REGEN=<commit> appends
the missing ones and never rewrites one; ANALYSIS_GOLDEN_DUMP writes the token
streams to diff two binaries.
check-header-deps.py gets two rules for the code moving off CLucene (snii/,
analysis/, query/, common/ and ann/ under storage/index): no file there, sources
included, and no project header it reaches may include a CLucene header, and
the code there may not name a CLucene symbol once comments and strings are
stripped, because the PCH makes CLucene visible without an include. What still
breaks the rules is listed by the file that breaks it, 31 and 51 entries; a
missing or stale entry fails the gate, so the lists only shrink.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='*AnalysisGolden*:*AnalysisDifferential*:*AnalysisWriteGolden*:*AnnWriteGolden*:SniiWriterGoldenBytes*:SniiGoldenCorpus*'
passes on ASAN. The default set takes 33 s of wall time on 16 threads
for 515 s of work; ANALYSIS_GOLDEN_FULL=1 takes 920 s.
- GTEST_SHUFFLE=1 with '*AnalysisGolden*:*IK*' passes three times, seeds
56420, 13481 and 56363.
- The same tests pass on a RELEASE build, and both builds write
byte-identical digests (ANALYSIS_GOLDEN_WRITE, 2065 lines). The ANN
vectors have small integer coordinates for that: with fractional ones
the two builds wrote different HNSW graphs, because -O0 and -O3 round
faiss's distance sums differently.
- build-support/run-clang-tidy.sh --base origin/master reports no warning
on the changed lines.
- Mutations, each built, run and reverted: SimpleTokenizer<char> cutting
english tokens at 254 bytes instead of 255 fails the 4 english
configurations; basic splitting tokens over 255 bytes instead of
truncating them fails the 6 basic ones; IK lexemes ending one character
later fail the 10 IK ones; jieba without its lowercasing fails the 2
chinese ones with lower_case=true. Removing only the first-byte condition
of that lowercasing changes no digest: no token of the shipped
dictionaries starts with a non-ASCII byte and holds an ASCII letter.
- build-support/tests/test-build-hygiene-header-deps.sh passes, with new
cases for an angle include two hops from a root, a root source file,
stale entries in both tables, and CLucene words in comments and strings.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wo CLucene adapters
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is moving off CLucene. Today every
consumer (the SNII writer, the query side, the V1/V2 writers) reaches an
analyzer through lucene::analysis::Analyzer, TokenStream and Token, one virtual
call per token, so no analyzer can leave CLucene before every consumer does,
and the other way round. This adds the interface both sides move to, with no
production caller yet, and the two adapters that let consumers and analyzers
move one at a time.
The interface is in storage/index/inverted/analysis/ and includes no CLucene
header; the gate adds no exception for it.
- TokenBatch: the tokens of one next_batch() call, each a term view and a
position increment. A term points into the value, into memory the stream
keeps, or into the batch's arena, which never moves a term it handed out.
clear() keeps the token array and the arena's largest block;
release_oversized() frees everything once the batch keeps more than 64 KiB,
so one huge value does not pin its memory until the segment ends.
- TextTokenStream, with TextTokenizer and TextTokenFilter: reset(text) binds a
value, next_batch() hands out up to 256 tokens and returns false, with an
empty batch, only at the end, and reserved_bytes() reports what survives a
value.
- PerTokenTokenizer and PerTokenFilter: bases for components that make one
token at a time; the base copies each term into the batch. A filter reads its
input through a private batch, so it may hold an input token across batches,
as word_delimiter holds a token while it hands out its parts.
- TextAnalyzer: immutable after it is built, shared by threads, makes streams.
The 64 KiB high water mark and release_oversized_scratch() move from
token_stream.h to a new scratch.h, so the batch uses them without CLucene.
The adapters are in storage/index/inverted/clucene/:
- LuceneBackedTextAnalyzer puts a CLucene analyzer behind TextAnalyzer, for the
consumers that switch before their analyzers do. Each value gets a new reader
through the index's char filter and a new CLucene stream, as a query reads a
value today (the SNII writer re-initializes one reader instead); terms are
copied into the batch, and a CLuceneError comes out as a doris::Exception. It
goes once every analyzer is native.
- LuceneAnalyzerBridge puts a TextAnalyzer behind lucene::analysis::Analyzer,
for the V1/V2 writers. reusableTokenStream() rebinds one stream per scalar
value, tokenStream() makes an owned stream per array element, reset() starts
the value as SDocumentsWriter::invertField() expects, and terms go out with
setTextNoCopy(), empty ones included, because those writers give an empty
term a position.
One difference turned up and is pinned by a test: on the V1/V2 array path,
ICUAnalyzer::tokenStream() resets the stream and invertField() resets it
again, and ICUTokenizer, which reads its reader in reset(), finds it consumed:
array elements of an icu index are indexed as nothing. The bridge starts each
element once and indexes them. No table can have such an index, because the FE
accepts no parser and no analyzer on an array column
(InvertedIndexUtil.checkInvertedIndexParser).
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='TokenBatchTest.*:PerTokenStreamTest.*:LuceneBackedTextAnalyzerTest.*:LuceneAnalyzerBridgeTest.*:LuceneBridgeWriterTest.*:AnalysisGolden*:AnalysisDifferential*:AnalysisWriteGolden*:AnnWriteGolden*:SniiWriterGoldenBytes*:SniiGoldenCorpus*:InvertedIndexWriterTest.*'
passes on ASAN, with no leak report.
- TokenBatchTest: 10000 copied terms over many arena blocks keep their
bytes; clear() keeps the token array and the largest block, and a batch
that fits allocates nothing; release_oversized() leaves at most 64 KiB
after 10000 tokens or a 1 MiB term, and keeps an ordinary batch.
- PerTokenStreamTest: batches of at most 256 tokens; a filter making two
tokens out of one; a filter dropping every token returns false, never an
empty batch; a filter holding a 300-character token across two batches;
next_token() is not called after it returned false; reset() in the middle
of a value does not replay it; reserved_bytes() adds up the chain and
reset() releases an oversized input batch.
- expect_text_stream_contract(), a checker later components reuse: false
only with an empty batch and again after, no token for an empty value with
a null or a non-null pointer, a reused stream equal to a new one, and a
reset after the first batch. It passes on the test streams, on a scripted
CLucene analyzer and on the standard analyzer through
LuceneBackedTextAnalyzer.
- AnalysisGolden.LuceneBackedTextAnalyzerMatchesTheFrozenDigests: all 122
analyzer configurations through LuceneBackedTextAnalyzer, with a new
stream per line and with one stream for all lines, match
analysis_golden.txt (3172 jobs).
- AnalysisGolden.EightThreadsShareOneTextAnalyzer: eight threads share each
configuration's analyzer on mixed_zh_en and invalid_utf8, half with a new
stream per line, and all match the frozen digests.
- AnalysisWriteGolden.LuceneAnalyzerBridgeWritesTheSameV2Index: the
production V2 writer with its analyzer behind the bridge writes the
frozen string-column digests of chinese, ik, icu and the custom chain,
and the same array-column index as the analyzer itself, except the icu
case above.
- LuceneBridgeWriterTest: a doris::Exception from reset() or next_batch()
reaches add_values() and add_array_values() as
INVERTED_INDEX_ANALYZER_ERROR through the CLucene frames.
- analysis_golden.txt and ann_write_golden.txt unchanged.
build-support/check-build-hygiene.sh passes with the same 31 + 51
exceptions. build-support/run-clang-tidy.sh reports no warning on the
changed lines.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yzer interface
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is moving off CLucene. The previous
change added the CLucene-free TextAnalyzer interface and two adapters with no
production caller. This one moves every consumer onto the interface, so the
analyzers can follow one at a time. The analyzers themselves still run on
CLucene behind LuceneBackedTextAnalyzer, except the gram family, which moves now
because the SNII writer depends on its stream.
- AnalyzerProvider::get_analyzer(), InvertedIndexAnalyzerCtx::analyzer,
InvertedIndexAnalyzer::create_analyzer() and the IndexPolicyMgr getters hand
out a TextAnalyzerPtr. index_policy_mgr.h and inverted_index_parser.h no longer
name a CLucene type, and no AnalyzerPtr is left outside the analysis layer.
- The index-level char filter (char_filter_type and friends) moves from the
readers each consumer built (create_reader) into the analyzer:
create_analyzer() and create_analyzer_provider() put it in front of whatever
they build, builtin, custom or normalizer, so every consumer reads every value
through it exactly as before. IndexPolicyMgr's unused outer_char_filter_map
parameter goes.
- get_analyse_result() has one loop: a reusable form over a stream and a batch,
and a one-shot form over an analyzer. analyze() and normalize() use it.
- The SNII writer keeps one stream and one batch for all its rows, made in
init() and destroyed by finish() and close_on_error(). The two lanes
(_plain_lane_token_stream, _finish_plain_lane_row) and _char_string_reader go;
the memory report covers the stream and the batch of every analyzer
(_report_analysis_capacity), not only the gram tokenizer's buffers. The solved
gram density is set directly on the stream, which replaces
_pending_density_permille.
- GramTokenizer is a native TextTokenizer that hands out its grams as views, and
GramTextAnalyzer is the analyzer of a gram-family policy, which the FE allows
no filter. The CLucene version stays as LuceneGramTokenizer for a CLucene
chain, which only a policy created before the FE check (a gram tokenizer plus
a filter) can still build. GramExtractor and GramTokenizer release a buffer
above 64 KiB at the next row instead of keeping it until the segment ends.
- MATCH without an index and tokenize() tokenize a block with one stream and
one batch.
- The V1/V2 writer reaches its analyzer through LuceneAnalyzerBridge and hands
it plain readers.
- create_builtin_analyzer() and normalize() turn a CLuceneError, such as a
missing dictionary file, into an INVERTED_INDEX_ANALYZER_ERROR exception, so
the CLuceneError catches of analyze(), the SNII writer, the scoring collector,
the search functions and tokenize() go. function_search.cpp keeps the one
around the whole evaluation, which covers index reads too.
- The analyzer benchmark drives TextAnalyzers built through the production
entry points: "row/" with a stream per row, "reuse/" with one stream.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Regression test, Manual test
- ./run-be-ut.sh --run with the analysis, golden, gram, SNII, MATCH,
tokenize, search, inverted index reader/writer, policy and collector
suites passes on ASAN (3945 tests).
- analysis_golden.txt only gains lines: four frozen write/v2-array digests
of an ARRAY<STRING> column on V2, added at 5271950, whose bridge
path wrote them first; there the icu one already read 282 postings,
digest 118e86a496cf6071, as here. The other digests are unchanged.
- The benchmark's digest mode over all six corpora gives the same tokens
as the T1 baseline binary in all 5,240,060 rows; row/ and reuse/ run.
- New tests: the index char filter is part of create_analyzer() and
create_analyzer_provider(); a missing chinese dictionary fails SNII
init(), analyze() and tokenize() with INVERTED_INDEX_ANALYZER_ERROR;
after a 2 MiB row the SNII batch is under 64 KiB and the writer charges
exactly the stream's and the batch's capacity; finish() and
close_on_error() destroy the stream; the solved density is on the
writer's GramTokenizer; the native GramTokenizer gives the CLucene
tokenizer's grams and releases a long row's scratch at the next row.
- Fixed while widening the run: the golden write tests and the adapter
tests now set up ExecEnv's tmp file directory themselves
(test::ensure_tmp_file_dirs()), since some earlier test may have
cleared it.
- ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster
of this build: 232 of 252 suites pass on the first run. The other 20
need debug points (18) or room under the FE's limit of 100 index
policies, which the earlier suites fill (2); all 20 pass on a fresh
cluster with enable_debug_points = true. The BE logged no sanitizer
error.
- build-support/run-clang-tidy.sh reports no warning on the changed
lines. build-support/check-build-hygiene.sh passes, with the CLucene
exception tables down from 31 + 51 to 29 + 46 entries: the SNII writer,
the compaction eligibility check, analyzer_provider.h and
inverted_index_parser.h no longer reach or name CLucene.
- Behavior changed: No. Two differences are only reachable outside SQL or on
an error path: a V1/V2 ARRAY index with the icu parser now indexes its
elements (before, ICUAnalyzer::tokenStream() and invertField() both reset the
stream and the second reset found the reader consumed), but the FE accepts no
parser on an array column; and a CLuceneError raised while tokenize() or MATCH
without an index analyzes a value now reports INVERTED_INDEX_ANALYZER_ERROR.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ream interface ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The SNII text analysis is moving off CLucene. The two previous changes added the TextAnalyzer interface and moved every consumer onto it, with the analyzers still CLucene ones behind LuceneBackedTextAnalyzer. This change moves the components themselves, all in one step because the two sets of base classes cannot be mixed in one chain: the 11 registered tokenizers (keyword, empty, standard, ngram, edge_ngram, char_group, basic, icu, pinyin, ik_smart, ik_max_word) and kuromoji, the 6 token filters, the 3 char filters, their factories and the analyzers built from them. english, standard, unicode and chinese are CLucene analyzers, not Doris components, and stay behind LuceneBackedTextAnalyzer until their own tasks. - TokenizerFactory::create() const returns a TextTokenizer, TokenFilterFactory::create() const takes and returns a TextTokenStream, and CharFilterFactory::create() const returns a TextCharFilter, a text-to-text function run in reset() instead of a CLucene Reader. - ChainAnalyzer is char filters (CharFilteredStream), a tokenizer and token filters. A custom analyzer and a custom normalizer are ChainAnalyzers, and create_builtin_analyzer() builds the basic, icu, ik and kuromoji parsers as preset chains with the same lowercasing and dictionaries as before. The index char filter is a CharFilteredAnalyzer around whatever the index analyzes with; create_reader() is gone and filter_characters() runs the native char_replace. - Components that make one token at a time derive from PerTokenTokenizer and PerTokenFilter and keep their state machines. Terms that stay valid until the next value, views of it or per-value storage, go into the batch without a copy. - The source byte offsets added with apache#67918 are removed: nothing outside the analysis layer read them. Their only effect on terms, strict handling of invalid UTF-8 below a pinyin filter with ignore_pinyin_offset=false, stays as set_strict_utf8(): icu, ngram and edge_ngram throw, ik folds rune by rune. PositionCapability had no reader either and goes. - Every component reports what it keeps from one value to the next (reserved_bytes()) and releases buffers above 64 KiB at the next value. basic, ik, pinyin and kuromoji kept a long value's buffers until the stream died, which was harmless with a stream per value and is not with the reused stream of the SNII writer. - The IK core reads the value through a TextCursor (the same 4096-byte window and read semantics), throws doris::Exception instead of CLuceneError, and decodes runes with analysis/common/rune.h, jieba's rules byte for byte, instead of the CLucene submodule's Unicode.hpp. StandardTokenizerImpl reads through the same cursor. The ik tokenizer factory builds its Configuration once instead of for every stream. - Deleted: analysis/reader.h (ReaderPtr moves to clucene/lucene_reader.h for the V1/V2 writer), token_stream.h, DorisTokenizer, DorisTokenFilter, DorisCharFilter, LuceneGramTokenizer, and the BasicAnalyzer, ICUAnalyzer, IKAnalyzer and KuromojiAnalyzer classes. match.cpp had CLucene's _CLFINALLY only through the old headers; it uses Defer. ### Release note None ### Check List (For Author) - Test: Unit Test, Regression test, Manual test - ./run-be-ut.sh --run with T3's suites plus the component, query engine, kuromoji, ik and icu suites passes on ASAN: 4121 tests, 6 skipped. - The new component_stream_test runs every tokenizer, filter and char filter through the stream contract (kuromoji in kuromoji_viterbi_test, on the dictionary that test builds); checks that after a 2 MiB value and a short one each keeps at most 1 MiB, exactly what its members hold (read with -fno-access-control); checks strict UTF-8 (icu, ngram and edge_ngram throw, ik folds rune by rune, a pinyin filter with ignore_pinyin_offset false turns it on upstream, the other tokenizers ignore it); and checks that a value NFKC_CF folds to nothing makes no token. - analysis_golden.txt and ann_write_golden.txt are unchanged, and ANALYSIS_GOLDEN_FULL=1 (every code point through every configuration) passes. The benchmark's digest mode gives the same tokens as the T1 baseline binary in all 5,240,060 rows of the six corpora and fourteen analyzers, kuromoji on the real dictionary included. - Throughput with be/benchmark/analyzer_ab.py: five alternating rounds, the median of the per-round ratios on each of six corpora and their geometric mean; an A/A of the T3 binary stays within 0.98 to 1.03. Against T3, reuse/ (one stream for all values) and row/ (a stream per value): basic 2.49 and 2.01, custom basic+lowercase 2.48 and 2.00, keyword+lowercase 1.88 and 1.54, standard+asciifolding+lowercase 1.57 and 1.24, standard+lowercase 1.76 and 1.51, standard+word_delimiter+lowercase 2.10 and 1.70, icu 1.35 and 1.18, ik_max_word 1.17 and 1.02, ik_smart 1.17 and 1.01, kuromoji 1.01 and 1.01. Against the T1 baseline binary: basic 1.82 and 1.73, custom basic 1.83 and 1.72, keyword 1.15 and 1.40, standard+asciifolding 1.23 and 1.09, standard 1.41 and 1.34, standard+word_delimiter 1.79 and 1.57, icu 1.14 and 1.12, ik_max_word and ik_smart 0.99 and 0.99, kuromoji 0.99 and 0.99; the lowest single corpus is 0.96 (ik on agent_input). - ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster of this build with debug points: 249 of 252 suites pass. Two hit the FE's limit of 100 index policies, which the earlier suites fill, and test_drop_index_with_format_v2 lists the index file as soon as the FE alter job finishes, while the BE received the drop-index task 23 and 28 seconds later in two runs. All three pass on a fresh cluster, and the BE logged no sanitizer error. - build-support/check-build-hygiene.sh passes with the CLucene exception tables down from 29 + 46 to 13 + 12 entries; analyzer.cpp is the only file of analysis/ left in them. build-support/run-clang-tidy.sh reports no warning on the changed lines. - The rewritten tests drop only assertions on deleted features: token offsets and source byte offsets (keyword 6, ngram 2, icu tokenizer 8, pinyin tokenizer 50, ik 2, lowercase 4, icu normalizer filter 7, icu normalizer char filter 7 macros, pinyin filter offsets in 27 tests and its provenance accessors in 10), the token type of lowercase (2, TokenRef has none), and char_filter_test.cpp, whose 3 assertions tested the reader methods of the deleted DorisCharFilter. - Behavior changed: No. The terms and positions of every analyzer are unchanged (above). An analyzer error now comes from the component as a doris::Exception instead of through the CLuceneError conversion, with the same INVERTED_INDEX_ANALYZER_ERROR code and a slightly different message. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lassifier
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The english parser still ran CLucene's SimpleTokenizer<char>
behind LuceneBackedTextAnalyzer, and the basic tokenizer walked the value byte
by byte into a token list. Both now classify the value 64 bytes at a time and
cut it at the edges of the letter-and-digit bits.
- tokenizer/byte_classifier.h: classify_block() turns 64 bytes into a bitmap
of ASCII letters and digits, a bitmap of the bytes that can start a
multi-byte character and, when lowercasing, a copy with A-Z folded. It is a
plain loop the compiler vectorizes on every target, the byte masks turned
into bits by simd::bytes32_mask_to_bits32_mask as the Doris filters turn
theirs. classify_tail() copies the last bytes through a stack buffer, so
nothing past the value is read.
- simd::bytes32_mask_to_bits32_mask gets a NEON branch; aarch64 took its
scalar loop. It compiles for aarch64 but has not run: no aarch64 host or
emulator was at hand, so BitsTest still has to pass on an ARM machine.
- EnglishTokenizer replaces SimpleAnalyzer<char> for the english parser and an
unknown one: the runs of ASCII letters and digits, cut every 255 bytes,
lowercased into a copy that keeps the offsets of the value, or views of the
value without lower_case. The chain has no stop words: SimpleAnalyzer<char>
never read the ones set_stopwords() gave it.
- BasicTokenizer runs the same block loop and decodes a character only at a
byte that can start one, the only bytes the scalar loop made a token of
besides the runs; a run is cut to 255 bytes and the rest dropped, as before.
It lowercases its own tokens, so the builtin basic parser drops its
lowercase filter, which changed nothing else in them: the CJK characters it
keeps have no case. With extra_chars it keeps the scalar loop.
- normalize() lowercases the text of a builtin parser with the same ASCII
function the tokenizers use. The std::tolower it replaces changed only A-Z
as well: the BE never calls setlocale.
- rowid_read_ahead.cpp includes storage/iterators.h for StorageReadOptions; it
compiled only through another file of its unity batch, and the new source
moved it to a batch without that file.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis suites with the new ones, 3396 of 3402
tests pass; the 6 skipped are the opt-in code point scan, four gram
report tests and the vector width check, which runs in the build
without AVX2 below. The wider set of analysis, index and search suites:
4152 of 4159 pass, 7 skipped. The code point scan passes with
ANALYSIS_GOLDEN_FULL=1. New tests:
ByteClassifierTest (every byte value at every position, random blocks,
every tail length, a guard page after the value), EnglishTokenizerTest
(CLucene's SimpleAnalyzer<char> run beside it on random bytes, block
edges, the 255-byte cuts, batches around 256 tokens, views without
lowercase, a guard page), BasicTokenizerTest (the same against the
scalar loop), BitsTest for the mask conversion, and ComponentStreamTest
for the buffers both release after a long value. After the last change,
push_chinese_char forced inline, the tokenizer, component and golden
suites ran again: 60 pass, 2 skipped.
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows of
the six corpora and fourteen analyzers.
- -Rpass=loop-vectorize with the RELEASE flags: the classification loop
is vectorized at width 32 with AVX2 and at width 16 with SSE4.2 alone,
in all four instantiations (a block or a tail, with or without the
lowered copy).
- A second ASAN UT build with USE_AVX2=0 in its own CMAKE_BUILD_DIR
(USE_AVX2=0 in its CMakeCache.txt) runs the same 3402 tests: 3397 pass,
ByteClassifierTest.BuiltWithTheExpectedVectorWidth among them, which
asserts that __AVX2__ is not defined there.
- Throughput of the benchmark binary of this commit, with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each of six corpora and their geometric mean.
Against the T1 baseline binary, reuse/ (one stream for all values) and
row/ (a stream per value): english 3.78 and 3.12, no corpus below 1.80
(row/http_request); basic 7.42 and 7.74, no corpus below 4.27
(reuse/ja_wiki); custom basic+lowercase 2.40 and 2.37. Against T4:
english 4.81 and 3.91, basic 4.05 and 4.45, custom basic+lowercase 1.31
and 1.38, and standard, which this change does not touch, 1.00 and
0.99. push_chinese_char is forced inline: out of line, its call for
every lead byte made basic 5-8% slower on ja_wiki and zh_wiki. A binary
with the same machine code for both tokenizers at other alignments
differs by up to 8% on one corpus of row/english (geomean 0.96), so a
difference of that size is code layout.
- check-build-hygiene.sh passes with the CLucene exception tables
unchanged; clang-format 16 (run_clang_format.py) leaves the changed
files as they are; run-clang-tidy.sh reports nothing on the changed
lines.
- Behavior changed: No. Every token of english and basic is the same as
before (above).
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…able
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The builtin standard and unicode parsers still ran CLucene's
standard95 StandardTokenizer behind LuceneBackedTextAnalyzer, and the custom
standard tokenizer ran the JFlex scanner StandardTokenizerImpl one code point
at a time through a buffer it refills and shifts. Both now run one flat table
of the same DFA over the value.
- tokenizer/standard/uax29_table.{h,cpp}: Uax29Table lays the scanner's ZZ_*
tables out at first use: 59 states by 29 classes with one byte per
transition, and a compact copy of the two-level character map, so that a
step is one load and no table is copied by hand. CLucene's standard95
scanner has the same tables entry for entry. longest_match() is the JFlex
longest match, over a window its caller defines.
- tokenizer/utf8_validator.h: the UTF-8 rule of the builtin parsers, ported
from CLucene's StringUtil::validate_utf8, which accepts overlong forms, and
the decoder of CLucene's scanner that goes with it.
- Uax29Tokenizer, the builtin standard and unicode parsers: a value strict
UTF-8 accepts (simdutf) passes, the lenient rule decides the others, and a
value it refuses has no token, as before. The window (a token is scanned
while it is shorter than 255 bytes), the drop of a longer token without a
gap, the ASCII lowercase of WORD tokens and the 33 stop words, also only
for WORD tokens, follow the old tokenizer. Lowercasing folds the whole value
into a copy and scans the copy: A-Z and a-z are one class of the DFA and
every accepting state reached through a letter is WORD, so the tokens are
the same; Uax29TableTest checks both on the table. Stop words are looked up
as packed integers after a length test, not in an unordered_set.
- StandardTokenizer, the custom standard tokenizer: a valid UTF-8 value runs
through the same table with the scanner's window, max_token_length bytes
from each token start: a character that does not fit ends the token, and
one that does not fit an empty window ends the value, as the scanner's
refill did. Its terms are views of the value. Any other value still runs
through StandardTokenizerImpl, whose reaction to invalid bytes depends on
where its buffer refills; those terms are copied, since a refill moves the
buffer.
- analyzer.cpp builds the standard and unicode parsers on Uax29Tokenizer;
lucene_builtin_analyzer stays for chinese.
- storage/utils.cpp no longer opens ErrorCode for whatever follows it in a
unity batch: the new sources moved it into a batch with version_graph.cpp,
and gen_cpp/types.pb.h, first included there, then saw two LIMIT_REACH.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis, index and search suites with the T5 and
T6 ones, 4177 of 4184 tests pass; the 7 skipped are opt-in reports and
generators, the opt-in code point scan and the vector width check of the
build without AVX2. The code point scan passes with
ANALYSIS_GOLDEN_FULL=1. AnalysisDifferential, which runs both standard
parsers beside standard95 and the custom tokenizer beside
StandardTokenizerImpl for max_token_length 1, 2, 3, 4, 5, 16, 255, 256,
1000 and 16383, on the golden families and on random lines, passes.
New tests: Uax29TableTest (the table against ZZ_* entry by entry, the
class of every code point, every first character accepted, the proof
that lowercasing the whole value changes only WORD tokens),
Utf8ValidatorTest (CLucene's validate_utf8 on every value of up to three
bytes and on random bytes, the decoder against CLucene's
string_to_wstring on every accepted sequence), Uax29TokenizerTest
(standard95 beside it on random values for every lower_case and stop
words setting, the 255-byte window and drop, overlong and invalid
values, the 33 stop words with their prefixes, suffixes and case
variants, batches around 256 tokens, a guard page) and
CustomStandardTokenizerTest (the scanner beside it on random values for
the same ten max_token_length values, the cut at the window, the end of
the value at a character longer than the window, the copied terms of an
invalid value with max_token_length=8, the views of a valid one).
LuceneBackedTextAnalyzerTest now wraps CLucene's standard analyzer
itself instead of the standard parser.
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows of
the six corpora and fourteen analyzers, the builtin standard parser and
three custom standard chains among them.
- Throughput of the benchmark binary of this change, with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each of six corpora and their geometric mean.
Against the T1 baseline binary, reuse/ (one stream for all values) and
row/ (a stream per value): the builtin standard parser 1.82 and 1.79, no
corpus below 1.52 (prose_en); unicode is the same analyzer. The custom
standard chains, whose lowercase filter is still per token: standard +
lowercase 1.84 and 1.78, + asciifolding + lowercase 1.51 and 1.33,
+ word_delimiter + lowercase 2.06 and 1.78. Against T5: standard 2.13
and 2.06 (it no longer goes through the CLucene adapter), the custom
chains 1.13 to 1.32.
- check-build-hygiene.sh passes with the CLucene exception tables
unchanged (analyzer.cpp keeps CLucene for chinese); clang-format 16
(run_clang_format.py) leaves the changed files as they are;
run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. Every token of standard, unicode and the custom
standard tokenizer is the same as before (above).
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The chinese parser was the last builtin parser still on a
CLucene analyzer: LanguageBasedAnalyzer behind LuceneBackedTextAnalyzer, over
the copy of cppjieba in CLucene's contribs-lib. jieba's segmenters now live in
the BE as analysis/tokenizer/jieba/, ported without a change in how they cut,
and the parser runs them through a tokenizer of the new stream interface.
This commit only moves the code; a later one changes its data layout.
- analysis/tokenizer/jieba/: trie.h, dict_trie.h, mp_segment.h, hmm_model.h,
hmm_segment.h, mix_segment.h, full_segment.h, pre_filter.h, segment_base.h,
unicode.h and jieba.h from cppjieba, string_util.h from limonp, with
cppjieba's and limonp's namespaces replaced by
doris::segment_v2::inverted_index::jieba, so that doris_be_test links both
copies without merging them. No namespace std, XLOG or XCHECK: a dictionary,
user dictionary or model file that cannot be read, or has an illegal line,
throws INVERTED_INDEX_ANALYZER_ERROR instead of aborting the process.
Decoding comes from analysis/common/rune.h, which ik already uses. The
keyword extractor, POS tagger, query segment, logging and std extensions are
not brought over, so idf.utf8 is no longer read; it stays in the package.
The asserts on dictionary data are DCHECKs, active in the same builds.
- JiebaTokenizer: copies the value, cuts it with MixSegment and the HMM in the
coarse mode or with FullSegment in the fine mode, and hands the words out in
batches, cut to 255 bytes. With lower_case, a word whose first byte is ASCII
is lowercased where it lies when it is handed out, as the old tokenizer did
to its buffer, so a word sees the lowercasing of the words before it. The
fine mode's words overlap: it hands out a run of ASCII letters and digits
whole and then the runs of letters and of digits in it, and jieba reads a
stray continuation byte and the byte after it as one character, so a run
can start with a byte that is not ASCII, keep its capitals, and be lowered
by the next word. A word that still has a capital after its turn is
therefore copied into the batch; the other terms are views of the copy of
the value. The dictionaries are loaded once per directory, and a failed
load is tried again by the next analyzer.
- analyzer.cpp builds the chinese parser on JiebaTokenizer, and with
lucene_builtin_analyzer and the CLuceneError catch gone it includes no
CLucene: it leaves the CLucene exception tables of check-header-deps.py.
- LICENSE.txt, dist/LICENSE-dist.txt, dist/licenses/LICENSE-cppjieba.txt,
dist/licenses/LICENSE-limonp.txt and .licenserc.yaml carry the MIT terms of
the derived files.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T7 ones, 4191 of 4198 tests pass; the 7 skipped are opt-in reports and
generators, the opt-in code point scan and the vector width check of the
build without AVX2. The code point scan passes with
ANALYSIS_GOLDEN_FULL=1. JiebaTokenizerTest runs the parser beside
LanguageBasedAnalyzer in both modes, with and without lowercase, on
random values with invalid bytes, on every word of jieba.dict.utf8 and
user.dict.utf8 (64 to a value, half with raised letters) and on
mixed-case, unknown and undecodable words. A death test does the same on
a dictionary of its own, with words such as 卡拉OK, Q版本 and words of a
supplementary plane, in a process of its own because CLucene's jieba
keeps the first dictionary it loads. Other new cases: 你好\xE4\xB8 makes
no token, the first byte decides the lowercase, supplementary-plane
characters and user words, loading without idf.utf8, a failed load tried
again, eight threads on one analyzer, batches around 256 tokens, the
stream contract, and the buffers released after a long value.
- Mutation: lowercasing the words of a batch in reverse order fails
AnalysisDifferential and JiebaTokenizerTest's random values, both on
the shipped dictionaries, where a stray continuation byte starts a run
of letters that keeps its capitals, and the batching test. The source
was restored before the final runs.
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
chinese_coarse and chinese_fine among them.
- The port was also compared with the original headers directly, outside
the BE build: 3,408,272 calls of cut and cut_all with 0 differences, on
the shipped dictionaries (every byte and pairs and triples of chosen
bytes, 300k random sentences with invalid bytes, every dictionary word
alone and joined to the next, about 70 MB of corpora) and on edited ones
(an undecodable word, user dictionaries of 1 to 4 columns, CRLF); 15
broken files fail on both sides.
- check-build-hygiene.sh passes, with analyzer.cpp out of both CLucene
exception tables (12+11 now); clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. Every token of the chinese parser is the same as
before (above). A missing dictionary file still fails the analyzer, now
with INVERTED_INDEX_ANALYZER_ERROR from the BE.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: jieba, moved into the BE unchanged in the previous commit,
still walked a trie of hash maps behind pointers, kept a 20-byte RuneStr and a
DAG entry with an inline vector per character, and looked separators, stop
words and HMM emissions up in hash sets and maps. The chinese parser now cuts
over flat arrays built once from the same dictionaries, with every decision
the old segmenters made kept, so the tokens do not change.
- flat_dict.h: FlatDict, built from the loaded DictTrie, HMMModel and stop
word file, which are dropped afterwards. The trie is breadth-first arrays
(child_begin, key, weight, has_value), so the children of a node are
consecutive and a node's count is the difference of two child_begin
entries; the first rune has a table of 0x10000 entries, supplementary first
runes are searched for among the root's children, up to 8 children are
scanned and more are bisected. User single-rune words are a bitmap with a
set for supplementary runes, the emission probabilities a rune-major table
of 0x10000 x 4 with a map for supplementary runes, and stop words of up to
8 bytes an open-addressing table keyed by their bytes, longer ones a sorted
vector. The pointer trie is never built: a stable sort keeps the last unit
of each word, which is what inserting the units in order kept.
- segmenter.h: the mix and full cuts of MPSegment, HMMSegment, MixSegment,
FullSegment and PreFilter over a JiebaScratch (jieba_scratch.h): runes as
code point and byte offset arrays, the DAG walked straight into the DP or
the full cut in its old order instead of being stored, one column of
Viterbi weights. Kept as they decide tokens: the lenient decoding of
rune.h, no token for a value that does not decode, the DP adding weights in
the old order and keeping the first maximum (strict >), MAX_WORD_LENGTH
512, FullSegment's word_len carried from one rune to the next, the whole
alphanumeric run before its runs of letters and digits, user single-rune
words, stop words dropped before lowercasing. The separators are compared
directly: space, \t, \n, U+FF0C and U+3002, all SPECIAL_SEPARATORS held.
- jieba.h loads the files and keeps only the FlatDict; JiebaTokenizer owns a
JiebaScratch, releases it above the high water mark at reset() like its
other buffers and counts it in reserved_bytes(). next_batch() is unchanged.
- trie.h, mp_segment.h, hmm_segment.h, mix_segment.h, full_segment.h,
pre_filter.h and segment_base.h are gone; LICENSE.txt and .licenserc.yaml
list the derived files that remain.
### Release note
The chinese parser tokenizes about 3.2 times as fast in the coarse mode and
2.3 times as fast in the fine mode, and its dictionaries take less memory.
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T7 ones, 4191 of 4198 tests pass and the 7 skipped are the opt-in ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1. JiebaTokenizerTest
passes unchanged: the live differential with LanguageBasedAnalyzer on
random values, every dictionary word, mixed-case, unknown and
undecodable words, and on a dictionary of its own in a death test.
ComponentStreamTest counts the scratch among the bytes a long value
leaves. IndexFileWriterTest.CacheEvictionTest failed once in a run of the
wider suite list and passes on its own: it depends on what earlier tests
leave in a shared searcher cache, not on this change.
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
chinese_coarse and chinese_fine among them.
- The new code was also compared with the original CLucene headers
directly, outside the BE build and with the real jieba_tokenizer.cpp:
4,208,058 cuts and 7,364,072 tokenizer runs (both modes, lowercase on
and off, the scratch reused and fresh) on every byte and pairs and
triples of chosen bytes, 300k random sentences with invalid bytes, every
dictionary word alone and joined to the next, long values, all corpora,
and edited dictionaries (an undecodable main word, duplicate main words,
supplementary-plane user words including words of 512 and 513 runes,
supplementary-plane emissions, CRLF files, 12k stop words): 0
mismatches, in a release and in an ASAN+UBSan build. 9 of 10 deliberate
mutations of the new code were caught; the tenth swaps E and S on a tie
at the last rune, which ends a word either way.
- Throughput of the benchmark binary of this change with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each corpus and their geometric mean, reuse/ and
row/. Against the T1 baseline: coarse 3.62 and 3.23 (zh_wiki 2.85 and
2.70, ja_wiki 3.18 and 2.93, the four mixed corpora 3.97 and 3.47), fine
2.44 and 2.28 (zh_wiki 2.48 and 2.41, ja_wiki 2.66 and 2.54, mixed 2.37
and 2.18). Against the previous commit: coarse 3.54 and 3.10, fine 2.43
and 2.19, so the move itself kept the old speed.
- Memory of loading the shipped dictionaries, each in a process of its own
with glibc malloc: the old cppjieba::Jieba with its five files +132.3 MB
of RSS after loading (+145 MB at the peak); the new holder +40.5 MB
right after loading and +11.2 MB after malloc_trim (+51 MB at the peak),
the difference being freed loader memory that glibc keeps and the BE's
jemalloc returns on decay. Loading takes 0.19 s instead of 0.51 s.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. Every token of the chinese parser is the same as
before (above).
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tokenizer
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The icu tokenizer asked ICU for the script of every character
and whether it is a combining mark or an emoji, allocated a UText per script
run, converted every value through UnicodeString::fromUTF8 and every term back
through UnicodeString and std::string, and the builtin icu parser lowercased
every token again in LowerCaseFilter. RuleBasedBreakIterator, which holds the
rules, does not change; the cost around it does.
- icu_char_table.{h,cpp}: tables built once per process for the BMP: an int16
script per code point and combining-mark, emoji and emoji-RK bitmaps, about
152 KB; supplementary code points still ask ICU.
- script_iterator: reads the table, runs its loop on locals with a test of
whether a character joins the run that does not branch on each condition,
and skips ASCII eight units at a time inside a Latin run, which is exact
because every ASCII character is Latin or Common (a test checks it).
- break_iterator_wrapper and composite_break_iterator: the emoji test from the
table, the previous boundary tracked instead of asked of ICU, and one UText
owned by the composite iterator and reopened for each run.
- icu_tokenizer: a native batch tokenizer. An ASCII value is widened to UTF-16
directly, a valid non-ASCII value converted by simdutf's validating
converter, an invalid one by u_strFromUTF8WithSub with U+FFFD, as
UnicodeString::fromUTF8 did; strict mode throws where it threw. The terms of
an ASCII value are views of it, the others are encoded straight into the
batch, still cut at 255 UTF-16 units without splitting a surrogate pair.
The builtin parser lowercases in the tokenizer: an ASCII term byte by byte,
a term of U+3000 to U+9FFF only (no case there) as it is, any other with
ICU's root-locale case mapping, which is what LowerCaseFilter did; the
custom icu_tokenizer does not lowercase. The combine_cj option, always on,
is gone: the table builds the Han, Hiragana and Katakana grouping in.
- analyzer.cpp: the icu preset uses the lowercasing tokenizer and no
LowerCaseFilter, as the basic preset does; tokenizer_chain() has no user
left and is removed.
ICU's break iterator keeps some state from one value to the next: after a
Common-script kana mark with no dictionary, such as U+309B, a later U+30FC
U+0301 in the same stream comes out as one token instead of two. The old code
does the same, and this change keeps it, so a reused stream still matches the
old reused stream token for token.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T8 ones, 4203 of 4210 tests pass and the 7 skipped are the opt-in ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1. The golden files
freeze the icu outputs, supplementary characters around the 255th UTF-16
unit and invalid UTF-8 included. New: ICUCharTableTest (every BMP entry
and every code point's lookup against the old ICU calls, and that every
ASCII character is Latin or Common) and ICUTokenizerStreamTest (the
builtin parser against the custom tokenizer followed by LowerCaseFilter
on 6,000 random values, lenient and strict; each lowercase path; no case
mapping in U+3000 to U+9FFF; a Turkish default locale gives the root
output; one stream reused across scripts gives what new streams give;
strict mode throws; 255-unit cuts with surrogate pairs; the buffers
released after a long value).
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
icu among them.
- The new code was also compared with the old tokenizer and LowerCaseFilter
directly, outside the BE build: 7,285,055 values per mode (builtin and
custom, lenient and strict): every code point alone and in context,
30,132 values around the 255th unit, 300k random values with invalid
bytes and every row of the six corpora, through kept streams and with
242,287 new-stream checks per mode. 0 differences, the same exceptions
in strict mode, and none under a Turkish default locale, in a release
and in an ASAN/UBSan build.
- Throughput with be/benchmark/analyzer_ab.py, on a host loaded far above
its usual level (load 118 to 246), so the ranges are wide. Against the
T1 baseline, reuse/ and row/: a first run of 5 rounds gave icu 2.06 and
2.01, the four mixed corpora 2.07 to 2.77, zh_wiki 1.48 and 1.51 and
ja_wiki 1.68 and 1.48; a second run of 9 rounds at load 220 gave zh_wiki
1.41 and 1.32 and ja_wiki 1.36 and 1.46. Against T7: zh_wiki 1.32 and
1.34, ja_wiki 1.30 and 1.31, which the standalone harness confirms at a
lower load (zh_wiki 1.34, ja_wiki 1.30, the four mixed corpora 2.11 in
geometric mean). With T4's 1.14 and 1.12 over the T1 baseline, that puts
zh_wiki at about 1.5 and ja_wiki at about 1.47 against it. ICU's break
iterator and its CJK dictionary now take about 88% of the time on
Chinese and Japanese text, so 1.5 on each is out of reach without
changing ICU's dictionary breaking, which this change keeps.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. Every token of the icu parser and the custom
icu_tokenizer is the same as before (above).
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The ik tokenizer walked dictionaries of DictSegment nodes,
each a map or array of child pointers, reached them through
Dictionary::getSingleton() for every character, asked a chain of range tests
for each character's type, regularized every word, kept its candidate hits in
a std::list and its results in a std::queue rebuilt for each value, and built
a std::string for every word it handed out. The tokens do not change.
- dic/FlatDict.h: the dictionaries as breadth-first flat tries of 12-byte
nodes, children sorted (scanned up to 8, bisected above), with a table of
0x10000 entries for the root's children. match() and isWord() keep the hit
semantics of the old walk. DictSegment is now a load-time builder: each
loader fills one, turns it into a FlatDict and frees it, and a load is still
published only after it succeeds, so a failed load can be tried again. The
FlatDict constructor lives in DictSegment.cpp, since a new source file would
move the Storage unity batches. dic/Hit.h holds (const FlatDict*, node).
- core/CharacterUtil: identifyCharType reads a table of 0x10000 entries built
once for the BMP, the block rules above it; a character's type is taken
once, on the regularized character; regularizeInPlace returns a word with no
0xE3 or 0xEF byte and no ASCII capital untouched, and regularizeString wraps
it with its old signature.
- core/IKSegmenter, AnalyzeContext, CJKSegmenter, CN_QuantifierSegmenter: the
Dictionary is taken once by reference instead of through getSingleton() on
every character; the four segmenters are members called in the old order;
an idle quantifier segmenter only releases its lock on a character that is
not Chinese, and the surrogate pair segmenter only runs on surrogate pairs;
CJKSegmenter's hits are a vector compacted in place in the same order; the
results are one vector reused across windows and values; words are handed
out as views, and Lexeme keeps no string.
- IKTokenizer: the words of a value lie back to back in one buffer reused for
every value and are lowercased there, in lenient and strict mode alike; a
value that does not decode still throws at reset().
Kept as they decide tokens: charEnd is the current index after matchWithHit
and the index plus one after matchInMainDict, which the quantifier path reads
as its next index; the 4096-byte windows with BUFF_EXHAUST_CRITICAL; the
255-byte cut at a character boundary; lowercase always on for dictionary
matching, so "Hello ＡＢＣ" with lower_case=false still gives hello and ABC.
An IKTokenizer or IKSegmenter now needs a loaded dictionary when it is made,
not at its first Chinese character; the factory and the builtin preset load
it first.
### Release note
The ik parser tokenizes about 1.8 times as fast, and its dictionaries take
about a tenth of the memory.
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T9 ones, 4213 of 4220 tests pass and the 7 skipped are the opt-in ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1. New:
IKFlatDictTest (FlatDict against DictSegment for every word, prefix and
suffix of main.dic, quantifier.dic, stopword.dic and extra_stopword.dic,
every step and length compared; 20k random strings against a trie with
supplementary-plane words and nodes of more than 8 children; no
DictSegment left after Dictionary::initial; a tokenizer keeps working
with the singleton gone, so the per-character path does not reach it)
and IKCharacterUtilTest (the type table against the block rules on every
rune up to 0x1FFFFF; regularizeString against the old implementation on
300k random byte strings). ik_anayzer_test gains "Hello ＡＢＣ" (hello,
ABC without lowercase; hello, abc with it), a character on the 4096-byte
boundary at offsets 4088 to 4100 for é, 中 and 😀, and 你好\xE4\xB8
throwing. IndexFileWriterTest.CacheEvictionTest failed once in a run of
the wider suite list and passes on its own and in the later runs: it
depends on what earlier tests leave in a shared searcher cache.
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
ik_smart and ik_max_word among them.
- The new code was also compared with the old ik code directly, outside
the BE build: 28,659,978 values in 8 modes (ik_smart and ik_max_word,
lower_case on and off, strict and lenient), through kept and new
streams: every dictionary line alone and joined, random strings with
invalid bytes, every offset around the 4096-byte boundary and all six
corpora in full. 523M tokens and 211,200 identical exceptions, 0
differences; an ASAN/UBSan run of 7.5M values found none, and no
sanitizer report. In that harness, measured as thread CPU time with old
and new taking turns on each 128 KiB chunk (the host load reached about
410, which made wall time useless; old against old gave 0.97 to 1.00),
the new code is 1.71 to 1.92 times as fast on every corpus in both
modes, and the loaded dictionaries take 6.0 MB instead of 64.2 MB.
- Throughput of the benchmark binary of this change with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each corpus and their geometric mean, reuse/ and
row/, at a host load of 50 to 135. Against the T1 baseline: ik_max_word
1.84 and 1.74 (zh_wiki 1.58 and 1.54), ik_smart 1.80 and 1.70 (zh_wiki
1.50 and 1.42), no corpus below 1.42. Against T8: ik_max_word 1.93 and
1.76, ik_smart 1.89 and 1.75.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. Every token of ik_smart and ik_max_word is the same
as before (above).
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o-one filters
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: lowercase, icu_normalizer and asciifolding were per-token
filters: each pulled the tokens of its input one at a time into a batch of its
own and copied every term it handed on, changed or not. They now rewrite the
tokens of their input's batch where they stand, writing only a changed term
into the batch's memory. word_delimiter, pinyin and asciifolding with
preserve_original, which can make two tokens out of one, stay per token. The
tokens and the exceptions do not change.
- lower_case_filter.h: LowerCaseFilter is a TextTokenFilter with a native
next_batch(). Which terms it rewrites decides where it throws, so a term is
left as it is only when it is certainly valid: ASCII without a capital, or
exactly one 3-byte character with a lead byte of 0xE3 to 0xE9 and two
continuation bytes, which lies in U+3000 to U+9FFF and has no case. An ASCII
term with a capital is lowercased byte by byte into the batch; any other term
takes the old path, validation (the same exception) and ICU's root-locale
case mapping into the batch, retried once at the length ICU asks for when the
term grows (as İ does).
- icu_normalizer_filter: native and one to one; a term the quick check passes
keeps its view, any other is normalized as before and copied into the batch;
a failed normalization still logs and leaves the token as it is.
- ascii_folding_filter: ASCIIFoldingFilter is the native one-to-one filter,
keeping the view of a term with nothing to fold; the per-token code stays as
ASCIIFoldingPreserveOriginalFilter, which the factory makes for
preserve_original=true; both share may_fold() and fold().
The filters hand out the batches their input hands out instead of refilling a
batch of 256 tokens, so an exception now comes out of the call that reaches the
bad token, after the batches before it, where a filter used to throw before
handing out the tokens ahead of it. The same values throw, with the same code
and message.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T10 ones, 4222 of 4229 tests pass and the 7 skipped are the opt-in ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1. New:
OneToOneFilterTest (each batch filter against a per-token oracle of the
replaced code on random streams of 1 to 4 batches of 1 to 300 tokens,
with all four classes, empty terms, invalid UTF-8 and terms held as views
or as batch copies: the same terms, increments, codes and messages; for
icu_normalizer in four settings; terms left as they are keep their
views), LowerCaseFilterTest.HandlesEachClassOfTerm (the four classes,
which terms keep their views, and that only five go through ICU),
LowerCaseFilterTest.RejectsALeadWithoutContinuationBytes (keyword +
lowercase on E4 41 41 and four other malformed terms throws
INVERTED_INDEX_ANALYZER_ERROR) and, in ASCIIFoldingFilterFactoryTest,
standard + asciifolding on "café" with preserve_original ("cafe", then
"café" with an increment of 0) and without it.
RejectsInvalidUtf8WithAnalyzerError now reads the valid batch before the
call that throws.
- The analysis golden files are unchanged, and the benchmark's digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
the custom chains with lowercase and asciifolding among them.
- The new filters were also compared with the old ones directly, outside
the BE build, behind the real standard, basic, keyword, icu and empty
tokenizers in 32 chains (the benchmark's custom chains, each tokenizer
before each filter, every icu_normalizer setting, both asciifolding
settings, filters in a row): all corpora, every BMP code point in 15
contexts, supplementary code points, every 3-byte sequence led by 0xE0
to 0xEF and 60k random values with invalid bytes per chain. 137.1M
values, 773M tokens and 923K values that threw: 0 differences in tokens
and exceptions; 57.1M values under ASAN/UBSan, 0 differences and no
sanitizer report. 10 mutants of the new code were all killed, and no
character in U+3000 to U+9FFF has a lowercase mapping (checked for each
one).
- Throughput of the benchmark binary of this change with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each corpus and their geometric mean, reuse/ and
row/, at a host load of 37 to 206, so the ranges are wide. Against the T1
baseline: custom standard+lowercase 2.97 and 3.11, basic+lowercase 4.68
and 4.94, standard+asciifolding+lowercase 2.49 and 2.62 (the three
together 3.26 and 3.43); keyword+lowercase 2.31 and 2.57 (ja_wiki 1.09
and 1.23, zh_wiki 1.12 and 1.27, where ICU's case mapping dominates long
values), standard+word_delimiter+lowercase 2.62 and 2.45. Against T9: the
three 1.67 to 2.04 and 1.80 to 2.17, keyword+lowercase 2.03 and 1.88,
standard+word_delimiter+lowercase 1.33 and 1.41. pinyin is not in the
benchmark and is not touched.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. Every token and every exception is the same as before;
only the call an exception comes out of can be a later one (above).
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 5, 2026
Merged
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #70268 (depends on)
Problem Summary: Let lowercase, icu_normalizer and asciifolding rewrite a token batch in place instead of copying every term: custom analyzers with these filters get 1.7x to 2.2x faster, with the same tokens.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code