Skip to content

refactor: Refactor S3 configuration structure and fix Python 3.9 compatibility issues#73

Merged
MaojiaSheng merged 2 commits intovolcengine:mainfrom
baojun-zhang:s3_format
Feb 5, 2026
Merged

refactor: Refactor S3 configuration structure and fix Python 3.9 compatibility issues#73
MaojiaSheng merged 2 commits intovolcengine:mainfrom
baojun-zhang:s3_format

Conversation

@baojun-zhang
Copy link
Collaborator

@baojun-zhang baojun-zhang commented Feb 5, 2026

Description

Refactor S3 configuration structure and fix Python 3.9 compatibility issues

Related Issue

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • Extract S3-related configurations from AGFSConfig into a new S3Config class
  • Update agfs_manager.py to use the new S3Config structure
  • Modify test_config_validation.py to adapt to the new configuration structure
  • Update configuration example files to reflect the new configuration structure
  • Fix Python 3.9 compatibility issue in embedding_queue.py by replacing EmbeddingMsg | None with Optional[EmbeddingMsg]

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

image

Additional Notes

ov.conf example

{
  # .....
  "storage": {
    "agfs": {
      "port": 8080,
      "log_level": "warn",
      "path": "./data",
      "backend": "s3",
      "url": "http://localhost:8080",
      "timeout": 10,
      "retry_times": 3,
      "s3": {
        "bucket": "openviking-test",
        "endpoint": "http://127.0.0.1:8333",
        "region": "us-east-1",
        "access_key": "any",
        "secret_key": "any",
        "prefix": "",
        "use_ssl": "false"
      }
    }
  }
}

…atibility issues

- Extract S3-related configurations from AGFSConfig into a new S3Config class
- Update agfs_manager.py to use the new S3Config structure
- Modify test_config_validation.py to adapt to the new configuration structure
- Update configuration example files to reflect the new configuration structure
- Fix Python 3.9 compatibility issue in embedding_queue.py by replacing `EmbeddingMsg | None` with `Optional[EmbeddingMsg]`
@MaojiaSheng MaojiaSheng merged commit 3ffde58 into volcengine:main Feb 5, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants