Skip to content

Conversation

@fenos
Copy link
Contributor

@fenos fenos commented Nov 9, 2025

What kind of change does this PR introduce?

Feature

What is the current behavior?

Analytics buckets currently use a single undeline catalogue to store tables and namespaces.
Sometimes, Iceberg catalogues have limitations in the maximum number of iceberg tables they can accommodate.

What is the new behavior?

We implement sharding for analytics buckets, meaning that we can shard iceberg tables across multiple catalogues and being able to multiply the number of available tables that we can create and offer. Thus, it becomes a federated catalogue.

Additional context

sharding is optional, but needed when offering analytics buckets as multi-tenant service

@fenos fenos force-pushed the fix/improve-analytics-buckets branch 4 times, most recently from 2e14ad1 to 336924e Compare November 10, 2025 16:46
@coveralls
Copy link

coveralls commented Nov 10, 2025

Pull Request Test Coverage Report for Build 19363040917

Details

  • 945 of 2098 (45.04%) changed or added relevant lines in 36 files are covered.
  • 50 unchanged lines in 7 files lost coverage.
  • Overall coverage decreased (-1.8%) to 75.913%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/storage/events/migrations/run-migrations.ts 0 1 0.0%
src/storage/protocols/iceberg/catalog/index.ts 2 3 66.67%
src/storage/events/workers.ts 3 6 50.0%
src/storage/events/lifecycle/bucket-created.ts 5 9 55.56%
src/internal/sharding/strategy/catalog.ts 14 20 70.0%
src/storage/protocols/vector/vector-store.ts 0 7 0.0%
src/internal/sharding/knex.ts 18 26 69.23%
src/internal/queue/database.ts 12 24 50.0%
src/internal/queue/event.ts 17 29 58.62%
src/internal/errors/codes.ts 11 24 45.83%
Files with Coverage Reduction New Missed Lines %
src/internal/errors/codes.ts 1 69.41%
src/storage/events/workers.ts 1 50.0%
src/storage/protocols/iceberg/catalog/rest-catalog-client.ts 1 65.68%
src/storage/storage.ts 1 75.4%
src/storage/database/knex.ts 4 82.46%
src/storage/protocols/iceberg/catalog/tenant-catalog.ts 6 83.15%
src/storage/protocols/iceberg/knex.ts 36 64.6%
Totals Coverage Status
Change from base Build 19332422696: -1.8%
Covered Lines: 25386
Relevant Lines: 33164

💛 - Coveralls

@fenos fenos force-pushed the fix/improve-analytics-buckets branch 4 times, most recently from 6bbc299 to 1990c98 Compare November 12, 2025 15:50
@fenos fenos force-pushed the fix/improve-analytics-buckets branch 3 times, most recently from e96c81e to e4cf9b4 Compare November 14, 2025 11:14
@fenos fenos force-pushed the fix/improve-analytics-buckets branch from e4cf9b4 to 0bc327f Compare November 14, 2025 11:21
@fenos fenos merged commit 90250f2 into master Nov 14, 2025
2 of 6 checks passed
@fenos fenos deleted the fix/improve-analytics-buckets branch November 14, 2025 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants