Skip to content

create_scalar_index: replace=True drops the existing index before some preconditions are validated #5249

Description

@FANNG1

Problem

create_scalar_index validates preconditions on both sides of the destructive drop_index call. With the default replace=True, any check that runs after the drop destroys the existing index and then fails, leaving the dataset with no index and no rebuild.

lance_ray/index.py:

if replace:
    if _index_exists(dataset, name):
        dataset.drop_index(name)          # ~L618  destructive

fragments = dataset.get_fragments()
if not fragments:
    raise ValueError("Dataset contains no fragments")        # ~L634

if fragment_ids is not None:
    ...
    if invalid_fragments:
        raise ValueError(f"Fragment IDs {invalid_fragments} do not exist in dataset")   # ~L639

Reproduction

lr.create_scalar_index(uri=path, column="id", index_type="BTREE", name="my_idx")
lr.create_scalar_index(
    uri=path, column="id", index_type="BTREE", name="my_idx",
    fragment_ids=[999],       # does not exist
    replace=True,             # default
)
# raises ValueError -- and `my_idx` is now gone

Because name is caller-supplied, reusing an existing index name for a call that fails validation drops an unrelated index.

The column-type check is currently on the correct side of the drop (see #5219), but nothing in the code or tests signals that this ordering is load-bearing, so a future contributor adding a precondition has no reason to put it in the right place.

Suggested fix

Hoist all non-destructive validation above the if replace: block — fragment existence, fragment_ids membership, and the segment-count adjustment — so the drop is the last thing that can happen before work starts.

Add a regression test that locks the invariant down: pre-create an index, call create_scalar_index with replace=True and an input that fails validation, then assert both that it raises and that the original index still appears in describe_indices().

Related

Surfaced while reviewing #5219, which restores large_string support for BTREE. Pre-existing, not introduced there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions