Skip to content

[FLINK-40147] Improve Native S3 FileSystem docs when using S3-compatible servers that do not support AWS checksum-validation #28817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Samrat002 wants to merge 1 commit into
base: master
Choose a base branch
from
+12 −1
Open

[FLINK-40147] Improve Native S3 FileSystem docs when using S3-compatible servers that do not support AWS checksum-validation #28817

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/content/docs/deployment/filesystems/s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,14 @@ In addition to the [common configuration](#common-configuration) options (`s3.ac

View the detailed configuration at [native-s3-fs](https://github.com/apache/flink/tree/master/flink-filesystems/flink-s3-fs-native#configuration-options)

For S3-compatible endpoints, the AWS SDK requires a region. Use the server's region, or `us-east-1` if the server ignores it. Disable checksum validation only if the server does not support it:

```yaml
s3.endpoint: https://your-s3-compatible-server
s3.region: us-east-1
s3.checksum-validation.enabled: false
```

---

### Presto S3 FileSystem
Expand Down
5 changes: 4 additions & 1 deletion flink-filesystems/flink-s3-fs-native/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,7 @@ For S3-compatible servers (SeaweedFS, LocalStack, Ceph RGW, etc.), set the endpo

```yaml
s3.endpoint: http://localhost:8333
s3.region: us-east-1
fs.s3.aws.credentials.provider: AnonymousCredentialsProvider

# Required: SeaweedFS serves the S3 API in path-style and does not support
Expand All @@ -289,7 +290,9 @@ s3.chunked-encoding.enabled: false
s3.checksum-validation.enabled: false
```

The exact subset of compatibility flags needed depends on the server and version — consult its documentation. The defaults in this filesystem (`path-style-access=false`, `chunked-encoding.enabled=true`, `checksum-validation.enabled=true`) are tuned for AWS S3.
The AWS SDK requires a region for custom endpoints. Use the server's region, or `us-east-1` if the server ignores it.

Required compatibility flags vary by server and version. Disable checksum validation only if the server does not support it. The defaults (`path-style-access=false`, `chunked-encoding.enabled=true`, `checksum-validation.enabled=true`) target AWS S3.

## Memory Optimization for Large Files

Expand Down