refactor: try upgrade regex-automata

[tisonkun]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This refers to #3043.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR does not require documentation updates.

[tisonkun]

Seems some internal impl issues exist; the following cases failed:

index inverted_index::search::fst_apply::intersection_apply::tests::test_intersection_fst_applier_with_valid_pattern
mito2 sst::index::creator::tests::test_create_and_query_regex

[codecov]

Codecov Report

Attention: Patch coverage is 96.87500% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 84.88%. Comparing base (2b2fd80) to head (8b9d6f8).
Report is 4 commits behind head on main.

❗ Current head 8b9d6f8 differs from pull request most recent head d46b413. Consider uploading reports for the commit d46b413 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3575      +/-   ##
==========================================
- Coverage   85.41%   84.88%   -0.53%     
==========================================
  Files         911      917       +6     
  Lines      152425   152890     +465     
==========================================
- Hits       130195   129784     -411     
- Misses      22230    23106     +876     

[zhongzc]

Great job!

[tisonkun]

@zhongzc It just compiled and passed test. I suggest you do some end to end test for where you need it in the index functionality ..

Also, to glue the regex-automata and fst together, "extra" checks are involved. I don't test the performance after this change.

[zhongzc]

Also, to glue the regex-automata and fst together, "extra" checks are involved. I don't test the performance after this change.

I'm not currently motivated to add automated testing to check the performance of fst matching regex statements. In my manual end-to-end bench, it is not a critical impact point.

I suggest you do some end to end test for where you need it in the index functionality ..

Once we have e2e testing one day, I will add enough test cases for index

[tisonkun]

@zhongzc Thank you! This makes sense to me.

Could you add some other reviews to this PR? I don't know who is familiar with this logic.

[zhongzc]

I feel like it’s more correct to write it like this,
ref to https://github.com/BurntSushi/regex-automata/blob/0ba880134d649866fa15809dec9c6eae89cd7591/src/dfa/transducer.rs#L41-L74

impl fst::Automaton for DfaFstAutomaton {
    type State = StateID;

    #[inline]
    fn start(&self) -> Self::State {
        let config = Config::new();
        self.0.start_state(&config).unwrap()
    }

    #[inline]
    fn is_match(&self, state: &Self::State) -> bool {
        self.0.is_match_state(*state)
    }

    #[inline]
    fn can_match(&self, state: &Self::State) -> bool {
        !self.0.is_dead_state(*state)
    }

    #[inline]
    fn accept_eof(&self, state: &StateID) -> Option<StateID> {
        if self.0.is_match_state(*state) {
            return Some(*state);
        }
        Some(self.0.next_eoi_state(*state))
    }

    #[inline]
    fn accept(&self, state: &Self::State, byte: u8) -> Self::State {
        if self.0.is_match_state(*state) {
            return *state;
        }
        self.0.next_state(*state, byte)
    }
}

[tisonkun]

@zhongzc Cool. Let me try it out.

[tisonkun]

It works. Burnt gives a more concrete implementation in https://github.com/BurntSushi/aho-corasick/blob/56256dca1bcd2365fd1dc987c1c06195429a2e2c/src/transducer.rs, which handles Anchored and Unanchored separately.

In our case, it should be always unanchored, meaning we don't ensure the patter match the whole string, but part of the string is OK.

[evenyag]

Mostly LGTM