Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Hadoop FileSystem.openFile() to open files #3078

Open
steveloughran opened this issue Nov 25, 2024 · 0 comments · May be fixed by #3079
Open

Use Hadoop FileSystem.openFile() to open files #3078

steveloughran opened this issue Nov 25, 2024 · 0 comments · May be fixed by #3079
Labels
Type: enhancement

Comments

@steveloughran
Copy link
Contributor

steveloughran commented Nov 25, 2024 •
edited
Loading

Describe the enhancement requested

Use the hadoop openFile() builder API to open a file, passing down

  • the existing FileStatus (saves a HEAD on cloud stores)
  • a read policy appropriate for parquet files. This ensures that the object stores know what kind of GET/prefect/cache to do.

Component(s)

Core
@steveloughran steveloughran changed the title Hadoop FileSystem.openFile() to open a file, passing down length and read policy Use Hadoop FileSystem.openFile() to open a file Nov 26, 2024
@steveloughran steveloughran changed the title Use Hadoop FileSystem.openFile() to open a file Use Hadoop FileSystem.openFile() to open files Nov 26, 2024
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Nov 26, 2024
@steveloughran
Use Hadoop FileSystem.openFile() to open files
550bf8f 
* Open files with FileSystem.openFile(), passing in file status
* And read policy of "parquet, vector, random, adaptive"

Addresses apacheGH-3078
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Nov 26, 2024
@steveloughran
apacheGH-3078: Use Hadoop FileSystem.openFile() to open files
b1b490b 
* Open files with FileSystem.openFile(), passing in file status
* And read policy of "parquet, vector, random, adaptive"
@steveloughran steveloughran linked a pull request Nov 26, 2024 that will close this issue
Open
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Nov 27, 2024
@steveloughran
apacheGH-3078: Use Hadoop FileSystem.openFile() to open files
ab31f8f 
* Open files with FileSystem.openFile(), passing in file status
* And read policy of "parquet, vector, random, adaptive"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

GH-3078: Use Hadoop FileSystem.openFile() to open files steveloughran/parquet-mr
1 participant
@steveloughran