Is there a way to create distributed inverted index?

[chenkovsky]

I want to create fts index on big corpus. but it seems that currently we cannot create an index that contains multiple index files. does anyone have any suggestion? or I want to create a PR to support it.

[BubbleCal]

hi @chenkovsky
now lance doesn't support to build the inverted index in a distributed way directly, but it's doable.
check the lines

you can see that the algorithm dispatches the tokens into multiple shards by a simple hashing, generally:

• the main thread tokenizes the documents, then dispatches the tokens into corresponding shards

  lance/rust/lance-index/src/scalar/inverted/builder.rs

  Line 191 in 6cd6ae8

  let shard = hasher.finish() as usize % num_shards;

• the index worker threads, each one for a shard, receives tokens from the stream, and index them

  lance/rust/lance-index/src/scalar/inverted/builder.rs

  Line 451 in 6cd6ae8

  async fn add(&mut self, row_id: u64, tokens: StringArray, positions: Int32Array) -> Result<()> {

• after all index workers finish the indexing, the algorithm collect and merge the results

  lance/rust/lance-index/src/scalar/inverted/builder.rs

  Line 344 in 6cd6ae8

  let mut merged_stream = stream::select_all(posting_streams);

so for distributed indexing, exposing the index worker may be a good start point.