Skip to content

Conversation

BrynjarM
Copy link

@BrynjarM BrynjarM commented Oct 9, 2025

FILE_SIZE_BYTES did not influence CSV COPY operations to S3 since, unlike the filesystem variant, the file length was consistently reported as 0. The implementation now updates the length as data is written + test added (based on https://github.com/duckdb/duckdb/blob/main/test/sql/copy/file_size_bytes.test).

Fixes duckdb/duckdb#18381

@BrynjarM
Copy link
Author

@Tmonster, @carlopi, @samansmink
I'm bringing this pull request to your attention as I'm unable to formally request reviewers. I noticed you are all active contributors and was hoping you might have a moment to look.
My primary goal with this PR was to clearly illustrate the issue from the bug report (that I found after running into this exact issue). As I'm new to the codebase, please feel free to take a completely different approach to the solution.
The key outcome is to ensure the FILE_SIZE_BYTES setting is correctly respected when writing CSVs to S3, allowing us to control the number of output files.
Thank you for your time and consideration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSV - COPY TO doesn't respect the FILE_SIZE_BYTES option

1 participant