[python] Group LeRobot video frame decoding by payload

[TheR1sing3un]

Purpose

Random LeRobot frame requests are currently decoded in caller order. Alternating between videos can repeatedly evict decoder sessions, while out-of-order frames from one video force the backend to seek backwards.

Group each batch by the physical video payload, decode every group in ascending frame ordinal, and restore the original row order before applying collate_fn. LeRobot dataset reads now submit base and delta rows to the collator together, so frames from the same payload can be scheduled across row groups. Decoder factories, PyAV/TorchCodec backends, process-local caching, output order, and the public API remain unchanged. A custom decode_fn will be invoked in grouped frame order rather than input order; its returned rows are still restored to input order.

Benchmark

A temporary local PyAV benchmark used an H.264 video at 160x90, 30 fps, 180 frames, GOP 30. It ran on macOS 26.4.1 arm64 with Python 3.12.13 and PyAV 18.1.0, warmed codec/filesystem caches, took the median of five runs, and verified every decoded result and final row order.

Workload	Scheduling	Time (s)	Decoder opens	Explicit seeks	Speedup
4 videos x 120 frames, cache 8	Input order	0.5380	4	445	1.00x
4 videos x 120 frames, cache 8	Group only	0.5405	4	445	1.00x
4 videos x 120 frames, cache 8	Group + sort	0.1836	4	0	2.93x
16 videos x 12 frames, cache 8	Input order	0.2496	107	131	1.00x
16 videos x 12 frames, cache 8	Group only	0.1263	16	82	1.98x
16 videos x 12 frames, cache 8	Group + sort	0.1212	16	0	2.06x

The ablation separates the benefit of payload grouping from frame sorting: grouping avoids decoder-cache churn when the working set exceeds the cache, while sorting removes backward seeks within each video. Results will vary with codec, GOP structure, backend, cache size, and access pattern. TorchCodec uses the same scheduling path but was not installed in this benchmark environment. The temporary benchmark harness is not included in this change.

Tests

• python -m pytest -q pypaimon/tests/multimodal_video_test.py pypaimon/tests/multimodal_lerobot_test.py -k 'video or delta' (15 passed, 2 skipped, 70 deselected)
• python -m flake8 --config=dev/cfg.ini pypaimon/multimodal/video.py pypaimon/multimodal/lerobot/dataset.py pypaimon/tests/multimodal_video_test.py

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: CLEAN.

Reviewed 94925e735933. Grouping by physical video and sorting frame requests reduces decoder churn/backward seeks for shuffled LeRobot batches. The implementation restores caller row order, preserves null descriptors, and passes base/delta rows through the same scheduling batch. The changed invocation order for custom decoders is documented.

Validation: the focused video/delta selection passed 15 tests with 2 skipped. A broader local run hit a pytest compatibility failure in an unchanged test that mocks sys.version_info; optional integrations were skipped. Current CI separately fails a Daft-on-Ray subprocess with exit -11 in Python 3.13. That test does not exercise this video scheduling path; its cause is not established by this review. The native video benchmark was not rerun.

No actionable implementation regression found in this review.