[python] Search vector index shards concurrently

[TheR1sing3un]

Purpose

Python vector index readers run their native search synchronously before returning a completed Future, so collecting those futures still opens and searches shards one at a time.

Schedule complete shard open/search/close operations with a bounded thread pool for both single and batch vector queries. Reuse the existing global-index.thread-num table option (positive integer, default 32), matching Java vector reads. The worker count is capped by the number of splits; setting the table option to 1 keeps searches serial. Native query options do not override this table-level concurrency setting.

Results are merged in split order, persisted metric checks are synchronized without serializing index loading, and pending tasks are cancelled while started tasks finish cleanup on failure. Single-shard and serial searches keep the direct path.

Tests

• python -m pytest -q pypaimon/tests/vector_index_parallel_search_test.py pypaimon/tests/vector_search_filter_test.py pypaimon/tests/vector_metric_consistency_test.py pypaimon/tests/vindex_vector_index_test.py pypaimon/tests/lumina_vector_index_test.py pypaimon/tests/batch_vector_raw_scan_test.py: 120 passed, 1 skipped (lumina_data is not installed).
• Coverage exercises concurrent index loading, default and configured worker limits, single/batch result ordering, offset pre-filters, metric mismatch and search-error cleanup, option validation, table-level configuration, and serial fast paths.
• Flake8, license-header checks, and git diff --check passed.

Benchmark

Measurement revision: 80dd2c34ba07. The results below use explicit worker limits of 1/2/4/8; they do not measure the default worker limit of 32. The benchmark was not rerun for the configuration revision.

macOS 26.4.1 arm64, Python 3.9.6, paimon-vindex 0.4.0, NumPy 2.0.2. Each native IVF-flat shard contains 4,096 vectors of 64 dimensions with 16 clusters. Queries use Top-K=10 and nprobe=4; per-shard Vindex I/O parallelism is fixed at 1.

The temporary harness runs the production index read/merge path against local index files. Timings include index opening, native initialization/search, result conversion, merging, and closing; table planning and scalar pre-filter evaluation are excluded. Each process performs one warm-up and five measured iterations; values below are medians. A delay of 1 ms is injected per positional read to isolate storage-latency sensitivity; this is not a live object-store measurement.

Shards	Queries	Read delay	Master (ms)	Parallelism 1 (ms)	2 (ms)	4 (ms)	8 (ms)
8	1	1 ms	61.957	61.746	32.263	16.291	8.787
8	8	0 ms	2.827	2.883	2.998	4.005	4.218
8	8	1 ms	165.151	165.171	83.661	42.373	22.892
16	8	1 ms	331.570	331.942	167.545	86.012	44.061

At 16 shards / 8 queries, parallelism 4 improves latency by 3.85x and parallelism 8 by 7.53x versus master. Peak process RSS for parallelism 1/2/4/8 was 139.8/139.6/153.6/155.7 MiB. All result IDs and score bytes matched the serial baseline, and open stream counts never exceeded the configured parallelism.

Local tiny searches do not benefit from additional threads; set global-index.thread-num=1 when serial shard searches are preferable. A separate 50-iteration local single-query check measured 1.075 ms on master versus 1.085 ms with parallelism 1. Real gains depend on shard size, storage, native thread settings, and concurrent query load.

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: CLEAN.

Reviewed 80dd2c34ba07. The existing native readers finish their searches synchronously, so the previous list of futures did not overlap shard open/search work. Scheduling the complete operation with a bounded pool gives multi-shard reads a useful concurrency path while the default remains serial. Metric synchronization, per-shard ownership, ordered result merging and error cleanup were checked independently.

Validation: 106 tests passed, 7 skipped and 54 subtests passed across the six specified suites. Current head CI is green. Optional native SDK integrations were skipped locally; the latency benchmark was not reproduced.

No actionable implementation regression found in this review.

[JingsongLi]

Reviewed the shard-search concurrency control. Reusing the existing global-index.thread-num option gives bounded parallelism without introducing a conflicting configuration surface. The end-to-end behavior looks good to me.