Skip to content

[refactor](be) Move every analysis component onto the CLucene-free stream interface - #70263

Open
airborne12 wants to merge 1 commit into
apache:masterfrom
airborne12:snii-analysis-t4-components
Open

[refactor](be) Move every analysis component onto the CLucene-free stream interface#70263
airborne12 wants to merge 1 commit into
apache:masterfrom
airborne12:snii-analysis-t4-components

Conversation

@airborne12

@airborne12 airborne12 commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #70262

Problem Summary: Move every analysis component (tokenizers, token filters, char filters and their factories) onto the CLucene-free stream interface. Tokens and index files are unchanged.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

🤖 Generated with Claude Code

@airborne12 airborne12 mentioned this pull request Nov 5, 2026
Draft
4 of 16 tasks
@hello-stephen

hello-stephen commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

This was referenced Nov 5, 2026
Draft
Merged
Merged
Merged
Merged
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
@airborne12 @claude
[refactor](be) Move every analysis component onto the CLucene-free st…
045c743 
…ream interface

Issue Number: None

Related PR: None

Problem Summary: The SNII text analysis is moving off CLucene. The two previous
changes added the TextAnalyzer interface and moved every consumer onto it, with
the analyzers still CLucene ones behind LuceneBackedTextAnalyzer. This change
moves the components themselves, all in one step because the two sets of base
classes cannot be mixed in one chain: the 11 registered tokenizers (keyword,
empty, standard, ngram, edge_ngram, char_group, basic, icu, pinyin, ik_smart,
ik_max_word) and kuromoji, the 6 token filters, the 3 char filters, their
factories and the analyzers built from them. english, standard, unicode and
chinese are CLucene analyzers, not Doris components, and stay behind
LuceneBackedTextAnalyzer until their own tasks.

- TokenizerFactory::create() const returns a TextTokenizer,
  TokenFilterFactory::create() const takes and returns a TextTokenStream, and
  CharFilterFactory::create() const returns a TextCharFilter, a text-to-text
  function run in reset() instead of a CLucene Reader.
- ChainAnalyzer is char filters (CharFilteredStream), a tokenizer and token
  filters. A custom analyzer and a custom normalizer are ChainAnalyzers, and
  create_builtin_analyzer() builds the basic, icu, ik and kuromoji parsers as
  preset chains with the same lowercasing and dictionaries as before. The index
  char filter is a CharFilteredAnalyzer around whatever the index analyzes with;
  create_reader() is gone and filter_characters() runs the native char_replace.
- Components that make one token at a time derive from PerTokenTokenizer and
  PerTokenFilter and keep their state machines. Terms that stay valid until the
  next value, views of it or per-value storage, go into the batch without a copy.
- The source byte offsets added with apache#67918 are removed: nothing outside the
  analysis layer read them. Their only effect on terms, strict handling of
  invalid UTF-8 below a pinyin filter with ignore_pinyin_offset=false, stays as
  set_strict_utf8(): icu, ngram and edge_ngram throw, ik folds rune by rune.
  PositionCapability had no reader either and goes.
- Every component reports what it keeps from one value to the next
  (reserved_bytes()) and releases buffers above 64 KiB at the next value. basic,
  ik, pinyin and kuromoji kept a long value's buffers until the stream died,
  which was harmless with a stream per value and is not with the reused stream
  of the SNII writer.
- The IK core reads the value through a TextCursor (the same 4096-byte window and
  read semantics), throws doris::Exception instead of CLuceneError, and decodes
  runes with analysis/common/rune.h, jieba's rules byte for byte, instead of the
  CLucene submodule's Unicode.hpp. StandardTokenizerImpl reads through the same
  cursor. The ik tokenizer factory builds its Configuration once instead of for
  every stream.
- Deleted: analysis/reader.h (ReaderPtr moves to clucene/lucene_reader.h for the
  V1/V2 writer), token_stream.h, DorisTokenizer, DorisTokenFilter,
  DorisCharFilter, LuceneGramTokenizer, and the BasicAnalyzer, ICUAnalyzer,
  IKAnalyzer and KuromojiAnalyzer classes. match.cpp had CLucene's _CLFINALLY
  only through the old headers; it uses Defer.

None

- Test: Unit Test, Regression test, Manual test
    - ./run-be-ut.sh --run with T3's suites plus the component, query engine,
      kuromoji, ik and icu suites passes on ASAN: 4121 tests, 6 skipped.
    - The new component_stream_test runs every tokenizer, filter and char
      filter through the stream contract (kuromoji in kuromoji_viterbi_test,
      on the dictionary that test builds); checks that after a 2 MiB value and
      a short one each keeps at most 1 MiB, exactly what its members hold (read
      with -fno-access-control); checks strict UTF-8 (icu, ngram and edge_ngram
      throw, ik folds rune by rune, a pinyin filter with ignore_pinyin_offset
      false turns it on upstream, the other tokenizers ignore it); and checks
      that a value NFKC_CF folds to nothing makes no token.
    - analysis_golden.txt and ann_write_golden.txt are unchanged, and
      ANALYSIS_GOLDEN_FULL=1 (every code point through every configuration)
      passes. The benchmark's digest mode gives the same tokens as the T1
      baseline binary in all 5,240,060 rows of the six corpora and fourteen
      analyzers, kuromoji on the real dictionary included.
    - Throughput with be/benchmark/analyzer_ab.py: five alternating rounds,
      the median of the per-round ratios on each of six corpora and their
      geometric mean; an A/A of the T3 binary stays within 0.98 to 1.03.
      Against T3, reuse/ (one stream for all values) and row/ (a stream per
      value): basic 2.49 and 2.01, custom basic+lowercase 2.48 and 2.00,
      keyword+lowercase 1.88 and 1.54, standard+asciifolding+lowercase 1.57
      and 1.24, standard+lowercase 1.76 and 1.51,
      standard+word_delimiter+lowercase 2.10 and 1.70, icu 1.35 and 1.18,
      ik_max_word 1.17 and 1.02, ik_smart 1.17 and 1.01, kuromoji 1.01 and
      1.01. Against the T1 baseline binary: basic 1.82 and 1.73, custom basic
      1.83 and 1.72, keyword 1.15 and 1.40, standard+asciifolding 1.23 and
      1.09, standard 1.41 and 1.34, standard+word_delimiter 1.79 and 1.57,
      icu 1.14 and 1.12, ik_max_word and ik_smart 0.99 and 0.99, kuromoji 0.99
      and 0.99; the lowest single corpus is 0.96 (ik on agent_input).
    - ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster
      of this build with debug points: 249 of 252 suites pass. Two hit the
      FE's limit of 100 index policies, which the earlier suites fill, and
      test_drop_index_with_format_v2 lists the index file as soon as the FE
      alter job finishes, while the BE received the drop-index task 23 and 28
      seconds later in two runs. All three pass on a fresh cluster, and the BE
      logged no sanitizer error.
    - build-support/check-build-hygiene.sh passes with the CLucene exception
      tables down from 29 + 46 to 13 + 12 entries; analyzer.cpp is the only
      file of analysis/ left in them. build-support/run-clang-tidy.sh reports
      no warning on the changed lines.
    - The rewritten tests drop only assertions on deleted features: token
      offsets and source byte offsets (keyword 6, ngram 2, icu tokenizer 8,
      pinyin tokenizer 50, ik 2, lowercase 4, icu normalizer filter 7, icu
      normalizer char filter 7 macros, pinyin filter offsets in 27 tests and
      its provenance accessors in 10), the token type of lowercase (2,
      TokenRef has none), and char_filter_test.cpp, whose 3 assertions tested
      the reader methods of the deleted DorisCharFilter.
- Behavior changed: No. The terms and positions of every analyzer are unchanged
  (above). An analyzer error now comes from the component as a doris::Exception
  instead of through the CLuceneError conversion, with the same
  INVERTED_INDEX_ANALYZER_ERROR code and a slightly different message.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@airborne12
airborne12 marked this pull request as ready for review November 13, 2026 03:16
@airborne12

airborne12 commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

airborne12 commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

/review

@airborne12
airborne12 force-pushed the snii-analysis-t4-components branch from d90ab96 to 045c743 Compare November 13, 2026 03:29
@airborne12

airborne12 commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner
@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner
@eldenmoon eldenmoon Awaiting requested review from eldenmoon eldenmoon is a code owner
@csun5285 csun5285 Awaiting requested review from csun5285 csun5285 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@airborne12 @hello-stephen