multiple texts #33
Draft
multiple texts #33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ajalab
reviewed
Jan 18, 2025
| @@ -24,6 +25,53 @@ pub trait SearchIndex: BackwardIterableIndex { | |||
| { | |||
| Search::new(self).search(pattern) | |||
| } | |||
|
|
|||
| // If we created a HasDoc trait (or something better named) for those | |||
There was a problem hiding this comment.
It might be also an option that we implement new APIs on a new trait/struct such as MultiTextSearchIndex that wraps a BackwardIterableIndex instance.
- We can avoid adding
Docdata to every FM-Index variant likeFMIndexandRLFMIndex. - We can consider different APIs for single-text search index
SearchIndexand multi-texts search indexMultiTextSearchIndex. For instance,locatequery in the former trait may just return pattern positions, whereas the latter may return positions withTextId.
There was a problem hiding this comment.
Yes, I think it would be nice to have a search index API that doesn't allow any \0 (or at least one at the end), and a multi index that accepts \0 (or a builder pattern), that offers the extended APIs.
So let's explore this. I'm a still not entirely sure how to implement doc so I hope to get this in a shape so you can help me implement it.
There was a problem hiding this comment.
Got it! I'll also try to implement the multi-text search based on the APIs you outlined.
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed on #24, right now FM-Index doesn't properly support multiple 0s in the text.
It would be nice to support this. This PR is a draft for what the API to support this feature could look like:
TextBuildermaking it easy to construct zero-separated texts.*a
LocationInfostruct that lets you obtain the text id for a location, as well as the text belonging to a 0 separated locationa
textmethod to get the original text belonging to such a zero separated location (both on the index as well as on theLocationInfovarious new search methods that are boundary aware. Each of them should forbid the use of \0 in the query itself (maybe this should be forbidden in general; that depends on how well it works):
contains (which is basically the same as
search)starts_with
ends_with
exact
maybe in the future lexicographical queries too.
We could gate all these methods with a
HasDoctrait so that they don't exist if you don't construct an index with a doc. That would allow us to implement this functionality onFMIndexonly (or first) without worrying about RFLMIndex yet.