Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistent tasks database implementation #350

Merged
merged 10 commits into from
Jul 2, 2024
Merged

Persistent tasks database implementation #350

merged 10 commits into from
Jul 2, 2024

Conversation

rdaum
Copy link
Owner

@rdaum rdaum commented Jul 1, 2024

Initial WiredTiger tasks DB implementation

This is a series of commits which pileup to implement persistent task save and load (on server restart). This is a feature that LambdaMOO has, allowing it to resume background tasks between server restarts, and is required for us to be at feature parity with 1.8.0.

Successfully persists stored tasks, and reconstitutes them on restart of the server.

Caveats:
.

  • Implementation is for WiredTiger only, no Relbox yet
  • The actual save/commit for each task currently blocks the Scheduler thread, and so could be a performance problem in the future that we will need to address by putting it into its own background thread.
  • No tweaking has been done on the wiredtiger DB configuration to e.g. commit more cache and keep more memory resident, etc.
  • No task loading from textdump yet, which is something we'd need before calling this feature complete.

Still this is major progress towards a 1.0 release.

rdaum added 10 commits July 2, 2024 18:47
To make serializing/deserializing this easier/better.

These are transient values which are now propagated down the call
chain instead.
Successfully persists stored tasks, and reconstitutes them on load.

Caveats:

* Design problem when reconstituting... these tasks need a Session
  that isn't NoopSession if they do background `notify` events. But how
  to get one? Some re-thinking and refactoring of session management
  has to be worked through, to separate the notion of "my" session
  created out of a connection event, a from "background" session --
  still attached to the same daemon/RpcServer, and to make it possible
  to go in and update these suspended tasks to have the latter.
* Implementation is for WiredTiger only, no Relbox yet
* No task loading from textdump yet.
  * Adds the concept of a SessionFactory to produce "background"
    sessions.
  * Resuming a task that does e.g. a `notify` now works.
  * Had to refactor `RpcServer` a bit to make this work.
@rdaum rdaum merged commit bea3bc7 into main Jul 2, 2024
11 of 12 checks passed
@rdaum rdaum deleted the rd/task-db branch July 6, 2024 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant