-
Notifications
You must be signed in to change notification settings - Fork 489
Description
Getting large number of panics when optimizing vector indices:
thread 'main' panicked at 'index out of bounds: the len is 117 but the index is 117': /src/lance/rust/lance-index/src/vector/ivf/storage.rs:95
0: log_panics::Config::install_panic_hook::{{closure}}
1: indexer::install_panic_handler::{{closure}}
2: std::panicking::panic_with_hook
3: std::panicking::panic_handler::{{closure}}
4: std::sys::backtrace::__rust_end_short_backtrace
5: __rustc::rust_begin_unwind
6: core::panicking::panic_fmt
7: core::panicking::panic_bounds_check
8: <lance::index::vector::ivf::v2::IVFIndex<S,Q> as lance_index::vector::VectorIndex>::partition_size
9: lance::index::vector::builder::IvfIndexBuilder<S,Q>::should_split
10: lance::index::vector::builder::IvfIndexBuilder<S,Q>::build_partitions::{{closure}}
11: lance::index::vector::builder::IvfIndexBuilder<S,Q>::build::{{closure}}
12: lance::index::vector::ivf::optimize_vector_indices_v2::{{closure}}
13: lance::index::vector::ivf::optimize_vector_indices::{{closure}}
14: lance::index::<impl lance_index::traits::DatasetIndexExt for lance::dataset::Dataset>::optimize_indices::{{closure}}::{{closure}}
This bug produces IVF_PQ indices that cannot be optimized, even by older version of the libraries. Can work around by dropping and recreating indices.