Skip to content

[Bug] Major Performance degradation with 20GB dataset. Timeouts with 200GB dataset. #124

Description

@RossWilliams

Describe the bug

Update operations become unreasonably slow if they change a GSI due to lack of an index. Problematic SQL missing indexes:

DELETE FROM "_ddb_<gsi_table>" WHERE base_pk = $1 AND base_sk_s = $2

At just 20GB I'm seeing updates take from 700ms-11s . At 200GB a single update_item call fails with a timeout. I'm using gsi_propagation_delay_ms=0.

To Reproduce

  1. Set gsi_propagation_delay_ms=0.
  2. Create a table with 3 gsis
  3. Load 20GB of data using all GSIs for the load
  4. with concurrency of 5 start updating the existing data with update_item. The updates should update a gsi key.
  5. Observe latency and max throughput

Expected behavior

DynamoDB offers consistent performance for common production databases.

Actual behavior

Over 30 seconds running:

Requests per second achieved: 4.4
p50 update latency: 730ms
p99 update latency: 11,677ms

A small database shows unreasonable performance degradation, likely severely impacting user-visible latency. A reasonably sized database will cease to operate.

Environment

  • ExtendDB version: 140a1e5
  • Operating system: macOS 26
  • Rust version (if building from source): 1.94
  • Client SDK/driver and version: https
  • Deployment method (binary, container, source): source

Checklist

  • I have searched existing issues for duplicates
  • I have included the ExtendDB version and environment details
  • I can reproduce this with the latest release

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions