Add a custom implementation LocalFileSystem::list_with_offset

[corwinjoy]

Which issue does this PR close?

Closes #7018.

Rationale for this change

Performance enhancement needed for network file system as described in the PR.

What changes are included in this PR?

Add an implementation LocalFileSystem::list_with_offset so that it can intelligently scan files and drastically reduce the number of statx system calls. Basically, we add a prefilter to pull only what is needed in-place.

Are there any user-facing changes?

None.

[corwinjoy]

The initial work for this PR was done by @dplasto, who reported the issue and provided an initial implementation. Further suggestions were provided by @adamreeve .

[tustvold]

It would appear this is legitimately failing CI

[alamb]

Thanks @corwinjoy and @tustvold -- marking this as a draft as we work through the test failures

[corwinjoy]

I'm not sure what is happening with the CI. It seems to be failing formatting. But, I ran 'cargo format' and clippy as specified in Contributing.md so I am not sure what is wrong.

…

On Sat, Jan 25, 2025, 9:14 AM Andrew Lamb ***@***.***> wrote: Thanks @corwinjoy <https://github.com/corwinjoy> and @tustvold <https://github.com/tustvold> -- marking this as a draft as we work through the test failures — Reply to this email directly, view it on GitHub <#7019 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBF42SFNON4OYCDVEG36AT2MPA6RAVCNFSM6AAAAABV2WCNBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUGAZTINRXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[etseidl]

I'm not sure what is happening with the CI. It seems to be failing formatting. But, I ran 'cargo format' and clippy as specified in Contributing.md so I am not sure what is wrong.

Are you running from the arrow-rs directory or the object_store directory. When I run cargo fmt --all -- --check from the object_store directory I get similar errors to CI.

[corwinjoy]

Ah, thanks for the clarification @etseidl! From the guidelines I didn't realize that I had to run these commands in the object-store rather than the arrow-rs directory. Doing that, I do get suggested changes which I have applied. Hopefully this will clear the CI checks.

[alamb]

Ah, thanks for the clarification @etseidl! From the guidelines I didn't realize that I had to run these commands in the object-store rather than the arrow-rs directory. Doing that, I do get suggested changes which I have applied. Hopefully this will clear the CI checks.

The emulator tests appear to be failing: https://github.com/apache/arrow-rs/actions/runs/12970828706/job/36180799417?pr=7019

---- local::tests::test_path_with_offset stdout ----
thread 'local::tests::test_path_with_offset' panicked at src/local.rs:1440:66:
called `Result::unwrap()` on an `Err` value: Generic { store: "LocalFileSystem", source: UnableToCanonicalize { path: "../testing/data/arrow-ipc-file", source: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
stack backtrace:

[corwinjoy]

OK. I have changed this test to use a temporary test and files which should make it more portable. We will see what the emulator says about this.

[alamb]

OK. I have changed this test to use a temporary test and files which should make it more portable. We will see what the emulator says about this.

Looks like the tests are good

[alamb]

This PR looks ready for review to me. @corwinjoy shall we mark it as ready for review?

[corwinjoy]

@alamb Yes please. Mark it as ready for review.