Repository navigation
[refactor](be) Keep CLucene out of the shared index file headers - #70272
Draft
airborne12 wants to merge 11 commits into
Draft
airborne12 wants to merge 11 commits into
airborne12 wants to merge 11 commits into
Conversation
…kens
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is moving off CLucene one analyzer at a
time, and every step has to show two things against the code before it: that it
is faster, and that every row still gives the same tokens. There was no tool for
either.
benchmark_analyzer.hpp measures every builtin parser and a few custom chains on
the path the SNII writer takes for a row (a new stream per row) and with one
stream across rows, over corpora given as `.rows` files. With
ANALYZER_BENCH_DIGEST set it registers only the benchmark "digest", which writes
one line per analyzer, corpus and row: the number of terms and a digest of their
bytes and positions. Diffing the files of two binaries names every row whose
tokens changed, on real text and with the real kuromoji dictionary, which the
unit tests cannot hold.
analyzer_ab.py runs two binaries in alternating rounds and reports the median of
the per-round throughput ratios, because the host load moves a single
measurement by 10 to 20 percent. analyzer_corpus.py builds the corpora; the
Chinese and Japanese ones are cut from the Wikipedia CirrusSearch dumps of
2025-12-29, so the same date gives the same corpus. analyzer_benchmark.md lists
the steps.
The analysis rewrite is measured against this commit, branch
snii-analysis-bench-baseline.
### Release note
None
### Check List (For Author)
- Test: Manual test
- BUILD_TYPE=RELEASE DORIS_BENCHMARK_HEADER=benchmark_analyzer.hpp
./build.sh --benchmark builds benchmark_test.
- The digest mode over six corpora of 8 MB (four mixed ones, zh_wiki and
ja_wiki) writes 5,240,060 lines in 67 s. A build with the separate
dynamic_cast fix of the tokenizer base class, which keeps every token,
gives a byte-identical file.
- analyzer_ab.py, five rounds: that fix measures 1.84 on basic and 1.64 on
custom standard + lowercase. The A/A run stayed within 0.94 to 1.08 per
analyzer on a host at load 140 to 280, more than the 5% the method
allows, so finer comparisons need a quieter host.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…being frozen
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Freezing the output of every text analyzer, in the next commit,
ran into three bugs that no frozen digest can hold.
1. The word_delimiter filter threw std::out_of_range for a token holding a NUL
byte when catenate_words, catenate_numbers or catenate_all was set.
WordDelimiterConcatenation::append called std::string::append(const char*,
pos, n), which resolves to the string_view overload and measures the text up
to its first NUL, so for "foo\0bar" the second part started past the end.
The SNII and V1/V2 writers catch only CLuceneError and doris::Exception, so
the error left the writer. The part is now appended from its offset.
2. IKAnalyzer::tokenStream created its tokenizer with new and then called
reset(), which throws when a value starts with a character cut short, such as
"\xE4\xB8". The tokenizer and its buffers, about 100 KB, leaked once per such
value, on the query path and on the SNII write path, which build a stream per
value. A unique_ptr now owns it until reset() returns.
3. The IK dictionary was loaded under std::call_once, which is used up even when
loading fails. After one failure, for example a dictionary path that is fixed
later, every IK analyzer kept failing until the BE restarted, and in the unit
tests the result of the IK tests depended on whether a test with a bad path
ran first. Dictionary::initial now publishes the dictionary only after it has
loaded, and the next call tries again. reload() and getSingleton(cfg), which
only existed to recover from that state, are removed.
### Release note
Fix the word_delimiter filter failing on values with a NUL byte when a catenate
option is set, and a memory leak of the IK analyzer on values it cannot decode.
### Check List (For Author)
- Test: Unit Test
- New: WordDelimiterFilterTest.CatenatePartsAcrossNul, and
IKTokenizerTest.TestUndecodableValueThrowsWithoutLeakingTheStream, whose
leak LeakSanitizer reports when the binary exits. Changed:
IKTokenizerTest.TestDictionaryExceptionHandling checks that a failed load
is retried and that the first loaded configuration stays.
- ./run-be-ut.sh --run --filter='WordDelimiterFilterTest*:IKTokenizerTest*'
passes on ASAN, also three times with GTEST_SHUFFLE=1 together with the
golden tests of the next commit.
- Behavior changed: Yes. A value with a NUL byte under word_delimiter with a
catenate option is indexed instead of failing; a failed IK dictionary load is
retried by the next IK analyzer instead of failing until a restart.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is about to be rewritten without
CLucene. A change in what an analyzer emits raises no error anywhere: queries
just stop finding the rows indexed with the old tokens. So the output is frozen
before any implementation changes, and every later step has to keep it.
AnalysisGolden runs 155 analyzer configurations over 13 corpus families. The
configurations are written the way users write them, as index properties and
analyzer, normalizer, tokenizer and filter policies, so the matrix does not move
when the implementation behind it does: every builtin parser with its lower_case
and stopwords values, every custom tokenizer, token filter and char filter with
its settings (pairwise covering arrays for the 16 pinyin and 9 word_delimiter
switches), the index's own char filter, normalizers, and the normalize() of
prefix and wildcard queries. The families aim at known risks: token windows at
255 bytes and at 255 UTF-16 code units, batch sizes of 255 to 513 tokens,
characters across 4096-byte chunks, invalid UTF-8, sampled jieba and IK
dictionary words, unknown Han strings, every pair of word break classes, sampled
BMP code points, NUL bytes and long lines. Each configuration runs with a new
stream per line, as queries and the SNII writer read values, and with one stream
across lines, as the V1/V2 writers do. Both must give the same XXH3 digest, the
one frozen in analysis_golden.txt; on the current code no configuration
disagrees. ANALYSIS_GOLDEN_FULL=1 scans every code point against
analysis_golden_full.txt. A test checks that every component AnalysisFactoryMgr
registers is in some configuration, and that the ICU analyzers give the same
output with Turkish as the default locale. The kuromoji configurations use a
small dictionary the test builds, because the real one is not in the
repository. The 4030 jobs run on up to 16 threads.
AnalysisDifferential compares the production english, standard, unicode,
custom standard and chinese analyzers with their old implementations, which stay
linked into the unit tests, on the same families and on one second of random
input, and shows that an injected difference is caught.
AnalysisWriteGolden freezes what the writers make of the tokens: the bytes of
the SNII file, and, because V1/V2 files carry timestamps, the terms with their
documents and positions read back, for chinese, ik, icu and a custom chain.
AnnWriteGolden freezes every sub-file of HNSW, IVF and IVF_ON_DISK indexes in V2
and in SNII, built on one OpenMP thread from vectors whose distances are exact
in float, and written twice to show the bytes repeat.
Lines of the digest files are only added: ANALYSIS_GOLDEN_REGEN=<commit> appends
the missing ones and never rewrites one; ANALYSIS_GOLDEN_DUMP writes the token
streams to diff two binaries.
check-header-deps.py gets two rules for the code moving off CLucene (snii/,
analysis/, query/, common/ and ann/ under storage/index): no file there, sources
included, and no project header it reaches may include a CLucene header, and
the code there may not name a CLucene symbol once comments and strings are
stripped, because the PCH makes CLucene visible without an include. What still
breaks the rules is listed by the file that breaks it, 31 and 51 entries; a
missing or stale entry fails the gate, so the lists only shrink.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='*AnalysisGolden*:*AnalysisDifferential*:*AnalysisWriteGolden*:*AnnWriteGolden*:SniiWriterGoldenBytes*:SniiGoldenCorpus*'
passes on ASAN. The default set takes 33 s of wall time on 16 threads
for 515 s of work; ANALYSIS_GOLDEN_FULL=1 takes 920 s.
- GTEST_SHUFFLE=1 with '*AnalysisGolden*:*IK*' passes three times, seeds
56420, 13481 and 56363.
- The same tests pass on a RELEASE build, and both builds write
byte-identical digests (ANALYSIS_GOLDEN_WRITE, 2065 lines). The ANN
vectors have small integer coordinates for that: with fractional ones
the two builds wrote different HNSW graphs, because -O0 and -O3 round
faiss's distance sums differently.
- build-support/run-clang-tidy.sh --base origin/master reports no warning
on the changed lines.
- Mutations, each built, run and reverted: SimpleTokenizer<char> cutting
english tokens at 254 bytes instead of 255 fails the 4 english
configurations; basic splitting tokens over 255 bytes instead of
truncating them fails the 6 basic ones; IK lexemes ending one character
later fail the 10 IK ones; jieba without its lowercasing fails the 2
chinese ones with lower_case=true. Removing only the first-byte condition
of that lowercasing changes no digest: no token of the shipped
dictionaries starts with a non-ASCII byte and holds an ASCII letter.
- build-support/tests/test-build-hygiene-header-deps.sh passes, with new
cases for an angle include two hops from a root, a root source file,
stale entries in both tables, and CLucene words in comments and strings.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wo CLucene adapters
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is moving off CLucene. Today every
consumer (the SNII writer, the query side, the V1/V2 writers) reaches an
analyzer through lucene::analysis::Analyzer, TokenStream and Token, one virtual
call per token, so no analyzer can leave CLucene before every consumer does,
and the other way round. This adds the interface both sides move to, with no
production caller yet, and the two adapters that let consumers and analyzers
move one at a time.
The interface is in storage/index/inverted/analysis/ and includes no CLucene
header; the gate adds no exception for it.
- TokenBatch: the tokens of one next_batch() call, each a term view and a
position increment. A term points into the value, into memory the stream
keeps, or into the batch's arena, which never moves a term it handed out.
clear() keeps the token array and the arena's largest block;
release_oversized() frees everything once the batch keeps more than 64 KiB,
so one huge value does not pin its memory until the segment ends.
- TextTokenStream, with TextTokenizer and TextTokenFilter: reset(text) binds a
value, next_batch() hands out up to 256 tokens and returns false, with an
empty batch, only at the end, and reserved_bytes() reports what survives a
value.
- PerTokenTokenizer and PerTokenFilter: bases for components that make one
token at a time; the base copies each term into the batch. A filter reads its
input through a private batch, so it may hold an input token across batches,
as word_delimiter holds a token while it hands out its parts.
- TextAnalyzer: immutable after it is built, shared by threads, makes streams.
The 64 KiB high water mark and release_oversized_scratch() move from
token_stream.h to a new scratch.h, so the batch uses them without CLucene.
The adapters are in storage/index/inverted/clucene/:
- LuceneBackedTextAnalyzer puts a CLucene analyzer behind TextAnalyzer, for the
consumers that switch before their analyzers do. Each value gets a new reader
through the index's char filter and a new CLucene stream, as a query reads a
value today (the SNII writer re-initializes one reader instead); terms are
copied into the batch, and a CLuceneError comes out as a doris::Exception. It
goes once every analyzer is native.
- LuceneAnalyzerBridge puts a TextAnalyzer behind lucene::analysis::Analyzer,
for the V1/V2 writers. reusableTokenStream() rebinds one stream per scalar
value, tokenStream() makes an owned stream per array element, reset() starts
the value as SDocumentsWriter::invertField() expects, and terms go out with
setTextNoCopy(), empty ones included, because those writers give an empty
term a position.
One difference turned up and is pinned by a test: on the V1/V2 array path,
ICUAnalyzer::tokenStream() resets the stream and invertField() resets it
again, and ICUTokenizer, which reads its reader in reset(), finds it consumed:
array elements of an icu index are indexed as nothing. The bridge starts each
element once and indexes them. No table can have such an index, because the FE
accepts no parser and no analyzer on an array column
(InvertedIndexUtil.checkInvertedIndexParser).
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='TokenBatchTest.*:PerTokenStreamTest.*:LuceneBackedTextAnalyzerTest.*:LuceneAnalyzerBridgeTest.*:LuceneBridgeWriterTest.*:AnalysisGolden*:AnalysisDifferential*:AnalysisWriteGolden*:AnnWriteGolden*:SniiWriterGoldenBytes*:SniiGoldenCorpus*:InvertedIndexWriterTest.*'
passes on ASAN, with no leak report.
- TokenBatchTest: 10000 copied terms over many arena blocks keep their
bytes; clear() keeps the token array and the largest block, and a batch
that fits allocates nothing; release_oversized() leaves at most 64 KiB
after 10000 tokens or a 1 MiB term, and keeps an ordinary batch.
- PerTokenStreamTest: batches of at most 256 tokens; a filter making two
tokens out of one; a filter dropping every token returns false, never an
empty batch; a filter holding a 300-character token across two batches;
next_token() is not called after it returned false; reset() in the middle
of a value does not replay it; reserved_bytes() adds up the chain and
reset() releases an oversized input batch.
- expect_text_stream_contract(), a checker later components reuse: false
only with an empty batch and again after, no token for an empty value with
a null or a non-null pointer, a reused stream equal to a new one, and a
reset after the first batch. It passes on the test streams, on a scripted
CLucene analyzer and on the standard analyzer through
LuceneBackedTextAnalyzer.
- AnalysisGolden.LuceneBackedTextAnalyzerMatchesTheFrozenDigests: all 122
analyzer configurations through LuceneBackedTextAnalyzer, with a new
stream per line and with one stream for all lines, match
analysis_golden.txt (3172 jobs).
- AnalysisGolden.EightThreadsShareOneTextAnalyzer: eight threads share each
configuration's analyzer on mixed_zh_en and invalid_utf8, half with a new
stream per line, and all match the frozen digests.
- AnalysisWriteGolden.LuceneAnalyzerBridgeWritesTheSameV2Index: the
production V2 writer with its analyzer behind the bridge writes the
frozen string-column digests of chinese, ik, icu and the custom chain,
and the same array-column index as the analyzer itself, except the icu
case above.
- LuceneBridgeWriterTest: a doris::Exception from reset() or next_batch()
reaches add_values() and add_array_values() as
INVERTED_INDEX_ANALYZER_ERROR through the CLucene frames.
- analysis_golden.txt and ann_write_golden.txt unchanged.
build-support/check-build-hygiene.sh passes with the same 31 + 51
exceptions. build-support/run-clang-tidy.sh reports no warning on the
changed lines.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yzer interface
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The SNII text analysis is moving off CLucene. The previous
change added the CLucene-free TextAnalyzer interface and two adapters with no
production caller. This one moves every consumer onto the interface, so the
analyzers can follow one at a time. The analyzers themselves still run on
CLucene behind LuceneBackedTextAnalyzer, except the gram family, which moves now
because the SNII writer depends on its stream.
- AnalyzerProvider::get_analyzer(), InvertedIndexAnalyzerCtx::analyzer,
InvertedIndexAnalyzer::create_analyzer() and the IndexPolicyMgr getters hand
out a TextAnalyzerPtr. index_policy_mgr.h and inverted_index_parser.h no longer
name a CLucene type, and no AnalyzerPtr is left outside the analysis layer.
- The index-level char filter (char_filter_type and friends) moves from the
readers each consumer built (create_reader) into the analyzer:
create_analyzer() and create_analyzer_provider() put it in front of whatever
they build, builtin, custom or normalizer, so every consumer reads every value
through it exactly as before. IndexPolicyMgr's unused outer_char_filter_map
parameter goes.
- get_analyse_result() has one loop: a reusable form over a stream and a batch,
and a one-shot form over an analyzer. analyze() and normalize() use it.
- The SNII writer keeps one stream and one batch for all its rows, made in
init() and destroyed by finish() and close_on_error(). The two lanes
(_plain_lane_token_stream, _finish_plain_lane_row) and _char_string_reader go;
the memory report covers the stream and the batch of every analyzer
(_report_analysis_capacity), not only the gram tokenizer's buffers. The solved
gram density is set directly on the stream, which replaces
_pending_density_permille.
- GramTokenizer is a native TextTokenizer that hands out its grams as views, and
GramTextAnalyzer is the analyzer of a gram-family policy, which the FE allows
no filter. The CLucene version stays as LuceneGramTokenizer for a CLucene
chain, which only a policy created before the FE check (a gram tokenizer plus
a filter) can still build. GramExtractor and GramTokenizer release a buffer
above 64 KiB at the next row instead of keeping it until the segment ends.
- MATCH without an index and tokenize() tokenize a block with one stream and
one batch.
- The V1/V2 writer reaches its analyzer through LuceneAnalyzerBridge and hands
it plain readers.
- create_builtin_analyzer() and normalize() turn a CLuceneError, such as a
missing dictionary file, into an INVERTED_INDEX_ANALYZER_ERROR exception, so
the CLuceneError catches of analyze(), the SNII writer, the scoring collector,
the search functions and tokenize() go. function_search.cpp keeps the one
around the whole evaluation, which covers index reads too.
- The analyzer benchmark drives TextAnalyzers built through the production
entry points: "row/" with a stream per row, "reuse/" with one stream.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Regression test, Manual test
- ./run-be-ut.sh --run with the analysis, golden, gram, SNII, MATCH,
tokenize, search, inverted index reader/writer, policy and collector
suites passes on ASAN (3945 tests).
- analysis_golden.txt only gains lines: four frozen write/v2-array digests
of an ARRAY<STRING> column on V2, added at 5271950, whose bridge
path wrote them first; there the icu one already read 282 postings,
digest 118e86a496cf6071, as here. The other digests are unchanged.
- The benchmark's digest mode over all six corpora gives the same tokens
as the T1 baseline binary in all 5,240,060 rows; row/ and reuse/ run.
- New tests: the index char filter is part of create_analyzer() and
create_analyzer_provider(); a missing chinese dictionary fails SNII
init(), analyze() and tokenize() with INVERTED_INDEX_ANALYZER_ERROR;
after a 2 MiB row the SNII batch is under 64 KiB and the writer charges
exactly the stream's and the batch's capacity; finish() and
close_on_error() destroy the stream; the solved density is on the
writer's GramTokenizer; the native GramTokenizer gives the CLucene
tokenizer's grams and releases a long row's scratch at the next row.
- Fixed while widening the run: the golden write tests and the adapter
tests now set up ExecEnv's tmp file directory themselves
(test::ensure_tmp_file_dirs()), since some earlier test may have
cleared it.
- ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster
of this build: 232 of 252 suites pass on the first run. The other 20
need debug points (18) or room under the FE's limit of 100 index
policies, which the earlier suites fill (2); all 20 pass on a fresh
cluster with enable_debug_points = true. The BE logged no sanitizer
error.
- build-support/run-clang-tidy.sh reports no warning on the changed
lines. build-support/check-build-hygiene.sh passes, with the CLucene
exception tables down from 31 + 51 to 29 + 46 entries: the SNII writer,
the compaction eligibility check, analyzer_provider.h and
inverted_index_parser.h no longer reach or name CLucene.
- Behavior changed: No. Two differences are only reachable outside SQL or on
an error path: a V1/V2 ARRAY index with the icu parser now indexes its
elements (before, ICUAnalyzer::tokenStream() and invertField() both reset the
stream and the second reset found the reader consumed), but the FE accepts no
parser on an array column; and a CLuceneError raised while tokenize() or MATCH
without an index analyzes a value now reports INVERTED_INDEX_ANALYZER_ERROR.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ream interface ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The SNII text analysis is moving off CLucene. The two previous changes added the TextAnalyzer interface and moved every consumer onto it, with the analyzers still CLucene ones behind LuceneBackedTextAnalyzer. This change moves the components themselves, all in one step because the two sets of base classes cannot be mixed in one chain: the 11 registered tokenizers (keyword, empty, standard, ngram, edge_ngram, char_group, basic, icu, pinyin, ik_smart, ik_max_word) and kuromoji, the 6 token filters, the 3 char filters, their factories and the analyzers built from them. english, standard, unicode and chinese are CLucene analyzers, not Doris components, and stay behind LuceneBackedTextAnalyzer until their own tasks. - TokenizerFactory::create() const returns a TextTokenizer, TokenFilterFactory::create() const takes and returns a TextTokenStream, and CharFilterFactory::create() const returns a TextCharFilter, a text-to-text function run in reset() instead of a CLucene Reader. - ChainAnalyzer is char filters (CharFilteredStream), a tokenizer and token filters. A custom analyzer and a custom normalizer are ChainAnalyzers, and create_builtin_analyzer() builds the basic, icu, ik and kuromoji parsers as preset chains with the same lowercasing and dictionaries as before. The index char filter is a CharFilteredAnalyzer around whatever the index analyzes with; create_reader() is gone and filter_characters() runs the native char_replace. - Components that make one token at a time derive from PerTokenTokenizer and PerTokenFilter and keep their state machines. Terms that stay valid until the next value, views of it or per-value storage, go into the batch without a copy. - The source byte offsets added with apache#67918 are removed: nothing outside the analysis layer read them. Their only effect on terms, strict handling of invalid UTF-8 below a pinyin filter with ignore_pinyin_offset=false, stays as set_strict_utf8(): icu, ngram and edge_ngram throw, ik folds rune by rune. PositionCapability had no reader either and goes. - Every component reports what it keeps from one value to the next (reserved_bytes()) and releases buffers above 64 KiB at the next value. basic, ik, pinyin and kuromoji kept a long value's buffers until the stream died, which was harmless with a stream per value and is not with the reused stream of the SNII writer. - The IK core reads the value through a TextCursor (the same 4096-byte window and read semantics), throws doris::Exception instead of CLuceneError, and decodes runes with analysis/common/rune.h, jieba's rules byte for byte, instead of the CLucene submodule's Unicode.hpp. StandardTokenizerImpl reads through the same cursor. The ik tokenizer factory builds its Configuration once instead of for every stream. - Deleted: analysis/reader.h (ReaderPtr moves to clucene/lucene_reader.h for the V1/V2 writer), token_stream.h, DorisTokenizer, DorisTokenFilter, DorisCharFilter, LuceneGramTokenizer, and the BasicAnalyzer, ICUAnalyzer, IKAnalyzer and KuromojiAnalyzer classes. match.cpp had CLucene's _CLFINALLY only through the old headers; it uses Defer. ### Release note None ### Check List (For Author) - Test: Unit Test, Regression test, Manual test - ./run-be-ut.sh --run with T3's suites plus the component, query engine, kuromoji, ik and icu suites passes on ASAN: 4121 tests, 6 skipped. - The new component_stream_test runs every tokenizer, filter and char filter through the stream contract (kuromoji in kuromoji_viterbi_test, on the dictionary that test builds); checks that after a 2 MiB value and a short one each keeps at most 1 MiB, exactly what its members hold (read with -fno-access-control); checks strict UTF-8 (icu, ngram and edge_ngram throw, ik folds rune by rune, a pinyin filter with ignore_pinyin_offset false turns it on upstream, the other tokenizers ignore it); and checks that a value NFKC_CF folds to nothing makes no token. - analysis_golden.txt and ann_write_golden.txt are unchanged, and ANALYSIS_GOLDEN_FULL=1 (every code point through every configuration) passes. The benchmark's digest mode gives the same tokens as the T1 baseline binary in all 5,240,060 rows of the six corpora and fourteen analyzers, kuromoji on the real dictionary included. - Throughput with be/benchmark/analyzer_ab.py: five alternating rounds, the median of the per-round ratios on each of six corpora and their geometric mean; an A/A of the T3 binary stays within 0.98 to 1.03. Against T3, reuse/ (one stream for all values) and row/ (a stream per value): basic 2.49 and 2.01, custom basic+lowercase 2.48 and 2.00, keyword+lowercase 1.88 and 1.54, standard+asciifolding+lowercase 1.57 and 1.24, standard+lowercase 1.76 and 1.51, standard+word_delimiter+lowercase 2.10 and 1.70, icu 1.35 and 1.18, ik_max_word 1.17 and 1.02, ik_smart 1.17 and 1.01, kuromoji 1.01 and 1.01. Against the T1 baseline binary: basic 1.82 and 1.73, custom basic 1.83 and 1.72, keyword 1.15 and 1.40, standard+asciifolding 1.23 and 1.09, standard 1.41 and 1.34, standard+word_delimiter 1.79 and 1.57, icu 1.14 and 1.12, ik_max_word and ik_smart 0.99 and 0.99, kuromoji 0.99 and 0.99; the lowest single corpus is 0.96 (ik on agent_input). - ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster of this build with debug points: 249 of 252 suites pass. Two hit the FE's limit of 100 index policies, which the earlier suites fill, and test_drop_index_with_format_v2 lists the index file as soon as the FE alter job finishes, while the BE received the drop-index task 23 and 28 seconds later in two runs. All three pass on a fresh cluster, and the BE logged no sanitizer error. - build-support/check-build-hygiene.sh passes with the CLucene exception tables down from 29 + 46 to 13 + 12 entries; analyzer.cpp is the only file of analysis/ left in them. build-support/run-clang-tidy.sh reports no warning on the changed lines. - The rewritten tests drop only assertions on deleted features: token offsets and source byte offsets (keyword 6, ngram 2, icu tokenizer 8, pinyin tokenizer 50, ik 2, lowercase 4, icu normalizer filter 7, icu normalizer char filter 7 macros, pinyin filter offsets in 27 tests and its provenance accessors in 10), the token type of lowercase (2, TokenRef has none), and char_filter_test.cpp, whose 3 assertions tested the reader methods of the deleted DorisCharFilter. - Behavior changed: No. The terms and positions of every analyzer are unchanged (above). An analyzer error now comes from the component as a doris::Exception instead of through the CLuceneError conversion, with the same INVERTED_INDEX_ANALYZER_ERROR code and a slightly different message. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: the ANN index read and wrote its files through CLucene itself.
VectorIndex::save and load took a lucene::store::Directory, faiss_ann_index.cpp
drove IndexInput and IndexOutput and caught CLuceneError in six places,
AnnIndexReader kept a DorisCompoundReader, and ann_index_writer.h included
CLucene.h and three more CLucene headers it did not use. That ties every
format that stores ANN, SNII included, to CLucene directories. This change
gives ANN a narrow storage interface and puts CLucene behind an adapter; the
bytes on disk, the error codes and the reads V1 and V2 send to the index file
stay the same.
- ann/ann_index_storage.h: AnnIndexInput (length, and read_at with the IO
context of the call, coalescing small reads), AnnIndexOutput (append, and a
close that may be called again), AnnIndexSource (open_input) and
AnnIndexSink (create_output). No CLucene.
- inverted/clucene/lucene_ann_store.{h,cpp}: LuceneAnnSource and LuceneAnnSink
over a lucene::store::Directory, the base class, so the tests keep their
RAMDirectory. read_at binds the IO context to the CLucene input for the one
call and reads through its BufferedIndexInput as before; reads and writes
over 2 GiB go in Int32 pieces; every CLuceneError becomes
INVERTED_INDEX_CLUCENE_ERROR with the file's name.
- ann/: VectorIndex::save(AnnIndexSink*) and load(AnnIndexSource*,
const io::IOContext*). FaissIndexWriter, FaissIndexReader and
CachedRandomAccessReader go through the interface; CachedRandomAccessReader
keeps its _io_mutex and passes each search's g_current_io_ctx to read_at. A
failed read or write inside faiss is still IO_ERROR, a failed open, create or
close still INVERTED_INDEX_CLUCENE_ERROR. AnnIndexReader keeps an
AnnIndexSource, declared before the index as the compound reader was,
because an IVF_ON_DISK index reads through it on every search; load_index no
longer catches CLuceneError. ann_index_writer.h drops CLucene.h,
LanguageBasedAnalyzer.h, bkd_writer.h, index_fs_directory.h and
index_file_writer.h.
- IndexFileWriter::open_ann_sink and discard_ann_staging replace
open_ann_directory and discard_ann_staging_directory, and
IndexFileReader::open_ann_source opens the read side. For now both wrap the
directories of today in the adapters: V1's and V2's directory and compound
reader, SNII's staging directory and its compound reader over the blob.
- check-header-deps.py: the ANN files and index_fs_directory.h leave the lists
of files allowed to reach CLucene.
- index_writer.cpp and segment_read_ahead.cpp include what they use
(index_file_writer.h, cloud/config.h): the first got it through
ann_index_writer.h, the second from a unity batch the new file moves.
Error messages change in three places: save reports the adapter's create,
write or close failure instead of "Failed to save vector index", a missing
ivfdata file is "Failed to open index file: ann.ivfdata, error: ...", and an
exception other than CLuceneError thrown inside a CLucene call keeps its own
code instead of becoming IO_ERROR. An IVF list over 2 GiB is now written in
pieces where it used to fail.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- BE UT under ASAN: be/test/storage/index/ann/ in full with the other index
suites (index file writer and reader, SNII, inverted index, faiss,
vector), 2187 of 2187 tests in 281 suites; segment, rowset and compaction
suites 945 of 948 with 3 skipped; the frozen goldens pass unchanged,
AnnWriteGolden among them, so the index files are byte for byte the same.
The goldens run apart from CloudFileCacheWriteIndexOnlyTest: its TearDown
clears ExecEnv's tmp file dirs, and a test writing index files after it in
one process crashes, SniiWriterGoldenBytes as well (not new here).
New: LuceneAnnStoreTest (a CLuceneError from open, create, append, close,
a read past the end and a failed read comes out as
INVERTED_INDEX_CLUCENE_ERROR; small adjacent reads share one buffered
read; a list's ids come out of the refill of its codes; a read larger
than the buffer goes straight to the file; the IO context is bound for
one call; an output closes once), AnnIndexStorageTest (the codes save and
load report through FaissVectorIndex) and
AnnIndexReaderTest.IvfOnDiskIndexIsDestroyedBeforeItsSource (the inputs
of a searched IVF_ON_DISK index are destroyed before their source, which
would be a heap-use-after-free under ASAN otherwise).
- AnnIndexRangeReadTest writes V1 and V2 indexes of HNSW, IVF and
IVF_ON_DISK to local files, loads them through AnnIndexReader and
searches them, and asserts the range reads and bytes the old code made,
recorded before this change: loads of 9 reads (HNSW), 34 (IVF) and 1
(IVF_ON_DISK), and for IVF_ON_DISK 8 reads in a narrow top-N and 24 in a
range search. The new code makes exactly the same.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines (the
three files that include omp.h through faiss were checked with the
toolchain's include directory added, which clang-tidy's command lacks).
- Behavior changed: No. The index files, the error codes and the reads of V1
and V2 are the same; only the message texts above change.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ories
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: since the ANN index got a storage interface of its own,
V1 and V2 implement it through a CLucene adapter, and SNII still went through
the same adapter over two CLucene directory shims: SniiBlobStagingDirectory
staged the faiss output, and the SNII branch of IndexFileReader::_open opened
the container a second time to read a blob through FSIndexInput and
DorisCompoundReader. This change implements the interface for SNII directly
and deletes the shims. The bytes on disk do not change.
- inverted/snii/snii_ann_store.{h,cpp}:
- SniiAnnSource reads the sub-files of an ANN blob through the container
reader IndexFileReader already opened, so a load opens the file once
instead of twice.
- SniiAnnInput buffers like CLucene's BufferedIndexInput, which V1 and V2
read through: a miss refills inverted_index_read_buffer_size bytes, cut
at the end of the sub-file, and the rest of a read at least that large
goes straight to the caller. read_at refuses offset + size past the
sub-file's length before any I/O, because the container reader bounds a
read by the container only and would serve the next blob.
- Each read goes out with the caller's IO context and is_index_data false,
as FSIndexInput sent it. So inverted_index_snii_read_no_write_file_cache
now covers ANN reads like every other SNII read, and their time goes to
inverted_index_io_timer.
- A NOT_FOUND from the file is IO_ERROR, since an ANN read has no fallback.
- SniiAnnStaging replaces SniiBlobStagingDirectory. Its output buffers
64 KiB, as CLucene's output did, because a staged file's append is a
write(2) of what it is given. close() does nothing when called again, and
an empty append, which faiss makes with a null pointer, writes nothing.
- IndexFileWriter keeps the SNII staging in its own map, with the metadata
the seal needs, so _indices_dirs holds only V1 and V2 directories and the
type assertion on getObjectName() goes away. delete_index, discard and
abandon of the staging, the seal, the release and debug_string use the new
map, and both maps refuse a duplicate key through one helper.
- IndexFileReader::open_ann_source reads SNII through SniiAnnSource. The SNII
branch of _open, which only ANN used, is gone, so open() now returns
INVERTED_INDEX_NOT_SUPPORTED for SNII. The refusal of a blob that is not an
ANN index moves to open_ann_source, without its "it has no CLucene
directory" tail.
- snii_blob_directory.{h,cpp}, which nothing in be/src used, move to the test
directory for the round-trip test that does; check-header-deps.py drops the
deleted files.
A blob that lacks a sub-file its index type writes, or a read past a
sub-file, is now INVERTED_INDEX_FILE_CORRUPTED at the input; through faiss it
is still IO_ERROR, and the ANN reader falls back to brute force on any load
failure as before.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- BE UT under ASAN: the index suites (ANN, index file writer and reader,
SNII, inverted index, faiss, vector), 2201 of 2201 tests in 282 suites;
segment, rowset and compaction suites 945 of 948 with 3 skipped; the
frozen goldens pass unchanged, among them AnnWriteGolden, which now reads
SNII's sub-files through the new input, and SniiWriterGoldenBytes.
New: SniiAnnStoreTest (10 tests: a read past the end fails before any
I/O and reads nothing; NOT_FOUND is IO_ERROR and is_index_data is false
even when the caller sets it; reads and their time are charged to the
caller's context; the no-write-file-cache switch sends ANN reads as
REMOTE_ONLY_ON_MISS when on and READ_THROUGH_AND_WRITE_BACK when off; a
staging output closes once, also after a failed finalize; buffering
against BufferedIndexInput's rules) and four SniiAnnContainerTest cases
(open() refuses every SNII index; the staging is kept apart from the
CLucene directories and refuses a duplicate key; an ANN index with no
rows leaves no staging; the staging calls leave V2's directories alone).
SniiAnnContainerTest passes in full, 24 tests, with the texts of
IvfDataWriteFailuresAreReturnedFromFinish and
ANonAnnBlobIsRefusedByTheDirectoryPath.
- AnnIndexRangeReadTest now covers SNII too, cold and with the file cache
off, recorded on the old code first. The read counts are the same in
every case: loads of 10 (HNSW), 35 (IVF) and 3 (IVF_ON_DISK) reads, and
for IVF_ON_DISK 8 in a narrow top-N, 24 in a range search, 0 once the
lists are cached. The bytes are the same or fewer, fewer where the old 4
KiB refill ran past the end of a sub-file (IVF_ON_DISK load 4191 -> 2697
bytes, top-N 81495 -> 81400). The V1 and V2 numbers are unchanged.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No. The index files and the reads are the same. With
inverted_index_snii_read_no_write_file_cache on, a query's ANN reads no
longer fill the file cache, as the variable already did for SNII's other
reads.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
T13 cuts the CLucene edges of the shared index file headers and clears what is left of CLucene in SNII. It needs both lines: T12 (SNII's ANN storage without CLucene directories) and T4 (every analysis component off CLucene, which removed analysis/reader.h and SNII's own CLucene includes and catches). The two lines change two files in common: - build-support/check-header-deps.py: the CLucene exception tables keep what still breaks the rules with both lines in, which the gate checks itself (an entry that is missing or no longer breaks the rule fails it): the shared index file headers, analyzer.cpp until T7, and snii_index_reader.cpp for T13. The analysis entries T2 to T4 removed, eligibility.cpp (T3) and the two SNII directory shims (T12) are gone. - be/test/storage/index/ann_write_golden_test.cpp: merged as is, T3's ensure_tmp_file_dirs() with T12's reads through open_ann_source(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tches
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: the BE does not build with ENABLE_UNITY_BUILD=OFF and
ENABLE_PCH=OFF. Four sources use a declaration they never include, which a
unity batch hides behind a sibling that includes it, and file_scanner_v2.cpp
defines two constants nothing reads, which clang reports (an error under
-Werror) only for the main file of a translation unit, never inside a batch:
- format_v2/table_reader.cpp uses ShardedKVCache: include
format/format_common.h.
- load/group_commit/wal/wal_table.cpp uses WalFileReader: include
wal_file_reader.h.
- service/http/utils.cpp uses DBUG_EXECUTE_IF: include util/debug_points.h.
- storage/segment/rowid_read_ahead.cpp uses StorageReadOptions: include
storage/iterators.h.
- exec/scan/file_scanner_v2.cpp: drop kIcebergPositionDeleteContent and
kIcebergDeletionVectorContent, which no code reads.
The next change has to be built that way once, because only then does a file
that got CLucene through a shared index header show the include it lacks.
### Release note
None
### Check List (For Author)
- Test: Manual test
- ./build.sh --be --index-tool and the BE UT build pass with
ENABLE_PCH=OFF ENABLE_UNITY_BUILD=OFF; each changed file compiles with
its own command from that build under -Werror.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: SNII and ANN no longer use CLucene, but eight of their eighty
sources (snii_index_reader, snii_bkd_index_reader, snii_index_writer,
snii_bkd_index_writer, ann_index_reader, ann_index_writer, ann_index_iterator,
ann_topn_runtime) still opened 67 CLucene headers, through the index file
headers they share with V1 and V2: index_file_writer.h and
index_file_reader.h included CLucene.h, the writer also index_compound_reader.h
and inverted_index_searcher.h, and index_common.h ended with
index_common_impl.h, which includes CLucene. This change cuts those edges.
- index_file_writer.h forward-declares lucene::store::Directory,
DorisCompoundReader and IndexSearcherBuilder, which it names only through
pointers.
- index_file_reader.h forward-declares lucene::store::IndexInput; its
constructor, destructor and get_inverted_file_size move to the .cpp, because
they destroy or read the stream and the entries, incomplete in the header.
- index_common.h no longer includes index_common_impl.h. The V1 and V2 sources
and the tests that instantiate its templates include it themselves, and
index_storage_format.h forward-declares lucene::store::IndexOutput.
- The -Wconversion pragmas around the removed CLucene includes go with them:
every BE source that still reaches CLucene compiles under the production
flags with CLucene.h included first and nothing suppressed.
- The V1 and V2 sources and tests that got CLucene types through
index_file_writer.h include what they use (index_compound_reader.h,
inverted_index_desc.h).
- snii_index_reader.cpp keys a field's statistics with
StringHelper::to_wstring of the column name, as predicate_collector computes
the same key, instead of CLucene's StringUtil::string_to_wstring. The two
agree on valid UTF-8, which column names are (checked on ASCII, CJK, 2- and
4-byte characters and U+10FFFF).
- check-header-deps.py: the shared index file headers and
snii_index_reader.cpp leave the CLucene exception tables, which now list
analyzer.cpp alone, for the analysis work. The self-test's two stale-entry
cases probed entries earlier changes had removed, and failed; they now list
a clean file of a root, query_info.h, so they hold however far the tables
empty.
- snii_clucene_guard_test checks compaction/eligibility.cpp too: its exemption
waited for the analyzer to move behind a factory, which happened earlier.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT under ASAN, unity build: the frozen goldens 19 of 20 with
EveryCodePoint skipped (it runs in FULL mode only), the index suites
2204 of 2204, the analysis suites 3368 of 3373 with 5 skipped, the same
as before this change; snii_clucene_guard_test passes with
eligibility.cpp checked.
- Built with ENABLE_PCH=OFF ENABLE_UNITY_BUILD=OFF, the only way the
missing includes show: ./build.sh --be --index-tool and the BE UT build
pass.
- Every SNII and ANN source, compiled with -H -fsyntax-only and its own
command from the non-unity build: 80 of 80 open no CLucene header (70
SNII, 10 ANN); 8 of them did before. Over the other CLucene-free roots
(analysis, query, common) only analyzer.cpp does, the one entry the gate
still lists.
- check-build-hygiene.sh passes, and so do its self-tests
(build-support/tests/run.sh): with the gate's stale-entry check disabled,
both rewritten cases fail. clang-format 16 leaves the changed files as
they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 of 16 tasks
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This was referenced Nov 5, 2026
Merged
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #70263, #70271 (depends on)
Problem Summary: Stop the index file headers shared with V1/V2 from pulling CLucene into SNII and ANN code. Also fix four sources that did not build without unity batches.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code