[Bug]: Headless nightly server crashes with JavaScript heap out of memory after ~23.5 hours

[Prgm-code]

Before submitting

• I searched existing issues and did not find a report with this exact environment and crash signature.
• I included enough detail to reproduce or investigate the problem.

Possibly related reports: #4178, #2761
apps/server

Steps to reproduce

1. On an Ubuntu VM, start the published nightly server with:

   npx --yes t3@nightly serve --host "::" --port 3773

2. Connect to the headless server using the remote web client.

3. Use the server normally with the persisted T3 state under ~/.t3.

4. Leave the server running continuously.

I do not yet have a smaller deterministic sequence beyond long-running use. In this occurrence, V8 reported an elapsed runtime of 84,604,456 ms immediately before the fatal error, equivalent to approximately 23 hours, 30 minutes, and 4 seconds.

Expected behavior

The headless server should remain available during long-running use and its memory consumption should remain bounded.

A large persisted history should not cause the Node process to reach the V8 heap limit and abort.

Actual behavior

After approximately 23.5 hours, the Node process reached a heap size close to 2 GiB. Mark-Compact garbage collection reclaimed very little memory and the process terminated with:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

The process then exited with Aborted (core dumped), making the remote T3 server unavailable until it was restarted manually.

Restarting the same command successfully restored the service with the existing state.

A read-only inspection after the restart showed:

• Restarted process RSS after approximately 29 minutes: 756,416 KiB (~738.7 MiB)
• SQLite state database: 181,133,312 bytes (~172.7 MiB)
• Projection threads: 39 total, 2 deleted, 0 archived
• Projected thread activities: 25,601 rows
• Activity JSON payload size: 44,106,381 bytes (~42.1 MiB)
• Orchestration events: 31,800 rows
• Event JSON payload size: 62,380,841 bytes (~59.5 MiB)
• Largest thread: 5,999 activities and approximately 9.35 MiB of activity JSON payload
• Messages: 1,503 rows and approximately 0.88 MiB of text

No NODE_OPTIONS or custom --max-old-space-size value was configured.

Impact

Blocks work completely

Version or commit

t3 0.0.29-nightly.20260725.899

Started through npx --yes t3@nightly.

Environment

• Mode: remote/headless t3 serve
• OS: Ubuntu 24.04.4 LTS
• Kernel: Linux 6.8.0-136-generic, x86_64
• Node.js: v24.18.0
• CPU: 16 virtual CPUs, Intel Xeon E5645 @ 2.40 GHz
• Virtualization: full virtualization
• RAM: 7.8 GiB
• Swap: 4.0 GiB
• T3 data directory: default ~/.t3
• Server command: npx --yes t3@nightly serve --host "::" --port 3773

Logs or stack traces

<--- Last few GCs --->

[269722:0x29fa8000] 84604456 ms: Mark-Compact 2044.3 (2056.1) -> 2009.5 (2040.2) MB, pooled: 13 MB, 233.83 / 0.03 ms (average mu = 0.214, current mu = 0.151) allocation failure; scavenge might not succeed

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x74eae8 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0xc42680  [node]
 3: 0xc4276f  [node]
 4: 0xee5fc5  [node]
 5: 0xee5ff2  [node]
 6: 0xee62ea  [node]
 7: 0xef6fea  [node]
 8: 0xefb390  [node]
 9: 0x198d931  [node]

Aborted (core dumped)

Screenshots, recordings, or supporting files

None currently. The crash was visible in the terminal output. No retrievable entry for the original process was available through coredumpctl.

Workaround

Manually restarting the same command restores the server temporarily:

npx --yes t3@nightly serve --host "::" --port 3773

The persisted state does not need to be deleted for the restart to succeed. It is not yet known how long the restarted process will remain stable.