Skip to content

feat(plugins): add replication plugin for external database sync (#72)#106

Open
voldemort9999 wants to merge 1 commit intoouterbase:mainfrom
voldemort9999:feat/issue-72-replication
Open

feat(plugins): add replication plugin for external database sync (#72)#106
voldemort9999 wants to merge 1 commit intoouterbase:mainfrom
voldemort9999:feat/issue-72-replication

Conversation

@voldemort9999
Copy link

/claim #72

Description

Adds a lightweight replication plugin that automatically pulls data from a configured external database (PostgreSQL, MySQL, Turso) into the internal local SQLite database.

This implementation deeply integrates with the existing architecture and alarm mechanism without altering the core StarbaseDBDurableObject system:

  • Implements plugins/replication/index.ts cleanly within ~110 lines.
  • Extends the existing executeExternalQuery mapping logic with INSERT OR REPLACE INTO natively to replicate rows idempotently.
  • Utilizes the cursorColumn (default: id) and state tracking via an automatically provisioned tmp_replication_cursors SQLite table to ensure duplicate data isn't fetched and replication scales independently per table.
  • Relies cleanly on the existing application ecosystem structure, avoiding unnecessary code bloat or intrusive dependencies across do.ts or route handlers.

Local Testing Performed

  • Validated external source (PostgreSQL) SELECT batching via dynamically mapped cursors.
  • Successfully verified edge cases when external results return empty batches (cursor tracks cleanly, upsert logic exits gracefully).
  • Validated full vitest pass on local logic simulating large schema syncs to ensure robust state management during the mapping step.
  • Checked TypeScript compiler to assure zero type defects or regressions.

Demo Video

os.01.mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant