Initial support for wal_checkpoint pragma #694

sonhmai · 2025-01-15T07:26:25Z

Wire pragma wal_checkpoint to checkpoint infra

add basic support for parsing and instruction emitting pragma wal_checkpoint;
checkpoint opcode for instruction
checkpoint execution in virtual machine
cli test

Part of #696.

Before

limbo> pragma wal_checkpoint;

  × Parse error: Not a valid pragma name

After

Enter ".help" for usage hints.
limbo> pragma wal_checkpoint;
0|0|0

sonhmai · 2025-01-15T08:20:48Z

The build on windows failed because rexpect is not supported in windows.
This one is only a dev-dependencies for integration testing the CLI.

Will need to look for a way to not building the cli tests in cli/tests in windows.
Although it does not work in windows, I still find that the integration tests of calling through cli written in Rust to be beneficial

dev in rust, integration test in rust. no context switching to python or TCL. use cargo for all.
advanced testing features (concurrency in tcl tests, etc.) in Rust instead of TCL.

sonhmai · 2025-01-16T05:44:27Z

resolved the issue by 465f14f.

this PR for scaffold is ready for review. cc @penberg

penberg

We already have infrastructure for WAL checkpointing so please just wire it up in this PR.

cli/tests/test_journal.rs

core/translate/mod.rs

core/vdbe/insn.rs

sonhmai · 2025-01-18T04:49:22Z

We already have infrastructure for WAL checkpointing so please just wire it up in this PR.

@penberg yeah, it makes sense. Let me bundle them.

core/vdbe/mod.rs

core/translate/mod.rs

sonhmai · 2025-01-19T02:22:22Z

@penberg this PR ready for review again now. Thanks.

sonhmai · 2025-01-19T08:52:27Z

Updated the PR title and description to reflect the change more accurately

Part of #696, more info and todos mentioned in the issue.

core/vdbe/mod.rs

penberg · 2025-01-20T07:20:43Z

cli/tests/test_journal.rs

+/// rexpect does not work on Windows.
+/// https://github.com/rust-cli/rexpect/issues/11
+#[cfg(not(target_os = "windows"))]
+mod tests {


There's testing/shelltests.py for CLI tests

Hi @penberg, yes I was aware of the python test.

My rationals of adding tests written in Rust

these are targeted to be integration tests for like wal features (similar to the TCL concurrency tests for WAL), not only cli. For example, have multiple processes connecting to database and asserts the wal behaviors through CLI interfaces.

dev in rust, integration test in rust. no context switching to python (shelltests.py) or TCL. use cargo for all.

advanced testing features (concurrency in tcl tests, etc.) in Rust instead of TCL.

adding more advanced concurrency tests later (complex ones in TCL for example, not only for wal)

I'm a bit hesitant to expand the integration tests in python, that's why I decided to go with Rust.

I'm fine with porting if the overall direction of the project is to go with Python for these advanced integration tests via shell interface.

Let me know your thoughts.

I am just bit hesitant of adding yet another test harness. That said, if we move all the integration tests to a top-level tests directory similar to what Deno, for example, does, I guess it's fine.

makes sense. let me centralize them in another PR.

penberg · 2025-01-20T07:31:15Z

core/vdbe/mod.rs

+                            // 2nd col: # modified pages written to wal file
+                            state.registers[*dest + 1] = OwnedValue::Integer(0);
+                            // 3rd col: # pages moved to db after checkpoint
+                            state.registers[*dest + 2] = OwnedValue::Integer(0);


Why set all of these to zeros? Is there some problem with checkpoint() not returning enough information?

Correct, checkpoint currently does not return necessary info. I noted in the TODO for another issue and PR to resolve it.

hard-coding 2nd and 3rd cols for now because checkpoint result does not provide these info. TODO recorded in #696.

sonhmai changed the title ~~add scaffolding for supporting wal checkpoint~~ draft: add scaffolding for supporting wal checkpoint Jan 15, 2025

sonhmai marked this pull request as draft January 15, 2025 07:33

sonhmai force-pushed the feat/wal-checkpoint branch from e466847 to 807e0f4 Compare January 15, 2025 07:49

sonhmai marked this pull request as ready for review January 15, 2025 07:57

sonhmai marked this pull request as draft January 15, 2025 07:57

sonhmai mentioned this pull request Jan 15, 2025

Wal checkpoint support #696

Open

6 tasks

sonhmai force-pushed the feat/wal-checkpoint branch from 3ebde0c to 465f14f Compare January 16, 2025 05:30

sonhmai marked this pull request as ready for review January 16, 2025 05:44

sonhmai changed the title ~~draft: add scaffolding for supporting wal checkpoint~~ add scaffolding for supporting wal checkpoint Jan 17, 2025

penberg reviewed Jan 17, 2025

View reviewed changes

cli/tests/test_journal.rs Outdated Show resolved Hide resolved

core/translate/mod.rs Show resolved Hide resolved

core/vdbe/insn.rs Outdated Show resolved Hide resolved

sonhmai marked this pull request as draft January 18, 2025 04:49

sonhmai force-pushed the feat/wal-checkpoint branch 3 times, most recently from f6e4c48 to aaaa6dc Compare January 18, 2025 05:58

sonhmai commented Jan 18, 2025

View reviewed changes

core/vdbe/mod.rs Outdated Show resolved Hide resolved

sonhmai commented Jan 18, 2025

View reviewed changes

core/translate/mod.rs Outdated Show resolved Hide resolved

sonhmai force-pushed the feat/wal-checkpoint branch from aaaa6dc to f8f9f09 Compare January 19, 2025 02:15

sonhmai changed the title ~~add scaffolding for supporting wal checkpoint~~ Wire pragma wal_checkpoint to checkpoint infra Jan 19, 2025

sonhmai marked this pull request as ready for review January 19, 2025 02:20

sonhmai commented Jan 19, 2025

View reviewed changes

core/vdbe/mod.rs Show resolved Hide resolved

sonhmai requested a review from penberg January 19, 2025 10:53

sonhmai and others added 4 commits January 20, 2025 08:34

add scaffolding for supporting wal checkpoint

66d6291

add dev dependencies for testing wal_checkpoint

6243ffb

feat: wire checkpoint to bytecode execution

cb631da

core: allocate 2 registers for checkpoint opcode execution

e45a807

sonhmai force-pushed the feat/wal-checkpoint branch from f8f9f09 to e45a807 Compare January 20, 2025 01:52

penberg reviewed Jan 20, 2025

View reviewed changes

penberg changed the title ~~Wire pragma wal_checkpoint to checkpoint infra~~ Initial support for wal_checkpoint pragma Jan 20, 2025

penberg closed this in 9369f06 Jan 20, 2025

penberg merged commit 9369f06 into tursodatabase:main Jan 20, 2025
37 checks passed

sonhmai deleted the feat/wal-checkpoint branch January 20, 2025 08:06

sonhmai mentioned this pull request Jan 20, 2025

Centralize Rust integration and regression tests #753

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial support for wal_checkpoint pragma #694

Initial support for wal_checkpoint pragma #694

sonhmai commented Jan 15, 2025 •

edited by penberg

Loading

sonhmai commented Jan 15, 2025

sonhmai commented Jan 16, 2025 •

edited

Loading

penberg left a comment

sonhmai commented Jan 18, 2025

sonhmai commented Jan 19, 2025

sonhmai commented Jan 19, 2025

penberg Jan 20, 2025

sonhmai Jan 20, 2025

penberg Jan 20, 2025

sonhmai Jan 20, 2025

penberg Jan 20, 2025

sonhmai Jan 20, 2025 •

edited

Loading

Initial support for wal_checkpoint pragma #694

Initial support for wal_checkpoint pragma #694

Conversation

sonhmai commented Jan 15, 2025 • edited by penberg Loading

sonhmai commented Jan 15, 2025

sonhmai commented Jan 16, 2025 • edited Loading

penberg left a comment

Choose a reason for hiding this comment

sonhmai commented Jan 18, 2025

sonhmai commented Jan 19, 2025

sonhmai commented Jan 19, 2025

penberg Jan 20, 2025

Choose a reason for hiding this comment

sonhmai Jan 20, 2025

Choose a reason for hiding this comment

penberg Jan 20, 2025

Choose a reason for hiding this comment

sonhmai Jan 20, 2025

Choose a reason for hiding this comment

penberg Jan 20, 2025

Choose a reason for hiding this comment

sonhmai Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

sonhmai commented Jan 15, 2025 •

edited by penberg

Loading

sonhmai commented Jan 16, 2025 •

edited

Loading

sonhmai Jan 20, 2025 •

edited

Loading