Problem
create_scalar_index validates preconditions on both sides of the destructive drop_index call. With the default replace=True, any check that runs after the drop destroys the existing index and then fails, leaving the dataset with no index and no rebuild.
lance_ray/index.py:
if replace:
if _index_exists(dataset, name):
dataset.drop_index(name) # ~L618 destructive
fragments = dataset.get_fragments()
if not fragments:
raise ValueError("Dataset contains no fragments") # ~L634
if fragment_ids is not None:
...
if invalid_fragments:
raise ValueError(f"Fragment IDs {invalid_fragments} do not exist in dataset") # ~L639
Reproduction
lr.create_scalar_index(uri=path, column="id", index_type="BTREE", name="my_idx")
lr.create_scalar_index(
uri=path, column="id", index_type="BTREE", name="my_idx",
fragment_ids=[999], # does not exist
replace=True, # default
)
# raises ValueError -- and `my_idx` is now gone
Because name is caller-supplied, reusing an existing index name for a call that fails validation drops an unrelated index.
The column-type check is currently on the correct side of the drop (see #5219), but nothing in the code or tests signals that this ordering is load-bearing, so a future contributor adding a precondition has no reason to put it in the right place.
Suggested fix
Hoist all non-destructive validation above the if replace: block — fragment existence, fragment_ids membership, and the segment-count adjustment — so the drop is the last thing that can happen before work starts.
Add a regression test that locks the invariant down: pre-create an index, call create_scalar_index with replace=True and an input that fails validation, then assert both that it raises and that the original index still appears in describe_indices().
Related
Surfaced while reviewing #5219, which restores large_string support for BTREE. Pre-existing, not introduced there.
Problem
create_scalar_indexvalidates preconditions on both sides of the destructivedrop_indexcall. With the defaultreplace=True, any check that runs after the drop destroys the existing index and then fails, leaving the dataset with no index and no rebuild.lance_ray/index.py:Reproduction
Because
nameis caller-supplied, reusing an existing index name for a call that fails validation drops an unrelated index.The column-type check is currently on the correct side of the drop (see #5219), but nothing in the code or tests signals that this ordering is load-bearing, so a future contributor adding a precondition has no reason to put it in the right place.
Suggested fix
Hoist all non-destructive validation above the
if replace:block — fragment existence,fragment_idsmembership, and the segment-count adjustment — so the drop is the last thing that can happen before work starts.Add a regression test that locks the invariant down: pre-create an index, call
create_scalar_indexwithreplace=Trueand an input that fails validation, then assert both that it raises and that the original index still appears indescribe_indices().Related
Surfaced while reviewing #5219, which restores
large_stringsupport for BTREE. Pre-existing, not introduced there.