Skip to content

Panic while optimizing vector indices #5312

@wjones127

Description

@wjones127

Getting large number of panics when optimizing vector indices:

thread 'main' panicked at 'index out of bounds: the len is 117 but the index is 117': /src/lance/rust/lance-index/src/vector/ivf/storage.rs:95
   0: log_panics::Config::install_panic_hook::{{closure}}
   1: indexer::install_panic_handler::{{closure}}
   2: std::panicking::panic_with_hook
   3: std::panicking::panic_handler::{{closure}}
   4: std::sys::backtrace::__rust_end_short_backtrace
   5: __rustc::rust_begin_unwind
   6: core::panicking::panic_fmt
   7: core::panicking::panic_bounds_check
   8: <lance::index::vector::ivf::v2::IVFIndex<S,Q> as lance_index::vector::VectorIndex>::partition_size
   9: lance::index::vector::builder::IvfIndexBuilder<S,Q>::should_split
  10: lance::index::vector::builder::IvfIndexBuilder<S,Q>::build_partitions::{{closure}}
  11: lance::index::vector::builder::IvfIndexBuilder<S,Q>::build::{{closure}}
  12: lance::index::vector::ivf::optimize_vector_indices_v2::{{closure}}
  13: lance::index::vector::ivf::optimize_vector_indices::{{closure}}
  14: lance::index::<impl lance_index::traits::DatasetIndexExt for lance::dataset::Dataset>::optimize_indices::{{closure}}::{{closure}}

This bug produces IVF_PQ indices that cannot be optimized, even by older version of the libraries. Can work around by dropping and recreating indices.

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions