Skip to content

fix(s3): verify direct-transfer redirects with the instance access keys - #3158

Open
xrgzs wants to merge 1 commit into
mainfrom
fix/s3-302
Open

xrgzs wants to merge 1 commit into
mainfrom
fix/s3-302

Conversation

@xrgzs

@xrgzs xrgzs commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Summary / 摘要

S3 direct-transfer redirects (302 for downloads, 307 for direct uploads) have never triggered since the gofakes3 dependency moved to the per-instance key store. s3RequestAuthorized verified the request with signature.V4SignVerify (and signature.V2SignVerify as the V2 fallback), both of which resolve the access key ID through the signature package's process-wide key store.

gofakes3's 706a863 moved WithV4Auth keys onto the instance and stopped writing that store. OpenList picked that up in #2813 (gofakes3 v0.8.1) and #3071 (v0.8.2-0.20260911142347-cd3c030a83b4). From then on the check returned InvalidAccessKeyId for every request whenever S3 access keys are configured, so directObjectURL and directUploadURL always bailed out at the first condition and every object download fell back to a server-side relay.

User-visible effect: the S3 endpoint silently stopped saving bandwidth. With web_proxy off and a storage that provides a direct link, downloads still ran through the OpenList server, with no error anywhere — the redirect path simply never fires.

Root cause, in one line: the redirect path asked a key store that is always empty, so it always concluded the caller was unauthorized.

Changes

  • server/s3/redirect.go
    • verify redirect requests with signature.V4SignVerifyWithLookup, falling back to signature.V2SignVerifyWithLookup, against the access keys this server was configured with
    • keep s3RequestAuthorized's signature unchanged, so server.go, redirectHandler, directObjectURL and directUploadURL are untouched
  • go.mod
    • bump github.com/OpenListTeam/gofakes3 to the commit that adds V2SignVerifyWithLookup

V4 first, then V2 on ErrUnsupportAlgorithm — the same order the gofakes3 auth middleware uses, so both paths agree on which requests are authorized.

Compatibility

  • No configuration change. web_proxy, webdav_policy and proxy_types keep deciding whether a request may be redirected; this PR only makes the existing check able to pass.
  • Behaviour change (intended, and what the docs already describe): with web_proxy off and a storage that provides a direct link, an authorized S3 GET may now return 302 to the provider URL, and a direct upload may return 307. This is the behaviour introduced in feat(s3): support direct transfer redirects #2598 and observed for 139Yun in fix(s3): only skip direct redirect for true sub-resource queries #2604's manual test — that test predates the gofakes3 dependency switch, which is why it passed then. Administrators who need the S3 client to always receive the object body keep that by enabling Web Proxy on the storage, as the S3 guide already recommends for repository tools such as restic and rustic.
  • Relates to [Feature] 为 S3 direct-link 302 与 Web Proxy 增加显式兼容性契约 #2796: that issue assumes an authenticated object GET may return 302. Before this fix that outcome was only reachable when no S3 access keys were configured, so the compatibility contract requested there was mostly theoretical. Landing this makes the redirect path work as documented, which makes that contract worth settling — it is not addressed here.
  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。
  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。
  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

  • OpenList-Frontend:
  • OpenList-Docs:

Related Issues / 关联 Issue

Testing / 测试

  • go test ./...
  • Manual test / 手动测试: After this patch, S3-redirect works properly.

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING。
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmt、go fmt 或 prettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明): DeepSeek

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

- verify redirect requests with V4SignVerifyWithLookup and
  V2SignVerifyWithLookup against the access keys this server was configured
  with, instead of the signature package's key store
- fall back from V4 to V2 in the same order the gofakes3 auth middleware uses
- restore the 302 download and 307 upload redirects: V4SignVerify and
  V2SignVerify read a package-wide key store that gofakes3 no longer writes, so
  every signed request failed that check and all traffic fell back to a
  server-side relay
- bump github.com/OpenListTeam/gofakes3 to the commit providing
  V2SignVerifyWithLookup

Co-authored-by: DeepSeek V4.1 Flash <noreply@deepseek.com>
Generated-by: WorkBuddy 5.6.2
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant