Reduce archived conversation disk usage with cold storage

[Quicksaver]

Summary

This adds compressed cold storage for archived conversations. Once a thread is archived, its conversation data and attachments are moved out of the hot state database into a dedicated archive.sqlite bundle, while unarchiving restores the complete thread tree before applying the domain command.

The lifecycle is durable and restart-safe: migrations 035/036 queue existing archived and deleted threads, background work resumes interrupted archive/delete jobs, and restoration keeps the cold bundle authoritative until the unarchive command commits successfully. Attachment ownership is preserved from exact persisted IDs and cold-bundle filenames throughout archive, restore, and durable deletion.

Archived thread shells remain consistent across clients without relying on compacted event history: each WebSocket session reconciles an authoritative shell snapshot, then replays live changes without losing archive removals during the handoff.

What Changed

• Added a dedicated cold-storage service that gzip-compresses thread-scoped SQL rows and attachments into chunked records in archive.sqlite.
• Required every cold-storage service error to retain its underlying failure cause.
• Removed successfully archived rows, attachments, and per-thread provider logs from hot storage, with incremental vacuuming to reclaim database pages.
• Restored archived thread trees transactionally before unarchive commands, with rollback on command failure and cleanup of the cold bundle only after a successful commit.
• Retried idempotent cold-bundle finalization when the same accepted unarchive command is dispatched again, without emitting a duplicate event or touching a mismatched thread.
• Reserved still-hot archived rows before unarchive dispatch so queued lifecycle work cannot move them cold between the restore check and command commit, and rejected unarchive when storage cannot restore or reserve the conversation.
• Scoped restored reservations to their original archive timestamp so a failed post-commit bundle finalization cannot block a later archive from moving the conversation cold.
• Serialized archive-tree lifecycle operations with idle lock eviction, restored cleanup_pending bundles, and prevented queued archive work or post-commit failures from undoing a successful restore.
• Added migrations 035/036 with persistent manifests, maintenance records, and delete queues so interrupted work and legacy archived/deleted threads are recovered after restart without replacing fork-local migration metadata.
• Rechecked archived shell state at the destructive transaction boundary and required active provider, terminal, and log writers to quiesce before archiving hot rows.
• Changed thread deletion to permanently remove hot rows, cold bundles, attachments, provider logs, and related cleanup records.
• Kept non-missing attachment/provider-log directory failures retryable instead of treating them as empty directories.
• Matched attachment ownership by exact persisted IDs and cold-bundle filenames so colliding sanitized thread segments cannot remove another live thread's files.
• Resumed cleanup_pending manifests even when the archived shell row is missing, while preserving exact attachment metadata until durable delete cleanup succeeds.
• Added one-time compaction of legacy hot storage and startup processing for pending archive/delete work.
• Closed per-thread provider log writers before lifecycle cleanup so files can be removed safely.
• Applied optimistic archive visibility consistently to rendered rows, project sorting, and keyboard navigation on web, and kept in-progress unarchive states across the web and mobile archive interfaces.
• Excluded optimistically archived threads from every web project-activity sort consumer and command-palette thread/project action so a hidden latest thread cannot reorder projects or remain a navigation target.
• Sorted invalid or missing mobile archive timestamps after valid timestamps for both newest-first and oldest-first views.
• Refreshed thread shells from an authoritative socket-owned snapshot on every completion-capable WebSocket session, with live buffering before the projection read so archived and deleted conversations cannot linger after event compaction.
• Retried failed thread-detail cache evictions on later shell items while the thread remains absent, and revived tombstoned caches before persisting authoritative active detail snapshots after event compaction.
• Added the supported t3-sqlite-state inspection path for both hot state.sqlite data and cold archive.sqlite manifests/chunks.
• Added coverage for archive/restore round trips, binary SQL values, attachment handling, schema evolution, bounded restore reads, failure recovery, migrations, orchestration integration, and sidebar archive visibility.

Why

With fairly moderate usage, the user .t3 folder grew to more than 30 GB. That is far too much local data growth for normal use, and without lifecycle controls it gets out of hand very quickly. Archived and deleted conversations need to stop accumulating indefinitely in hot storage while remaining reliably recoverable when a user unarchives them.

Validation

• pnpm exec vp check passed with exit 0; it reported 9 pre-existing lint warnings outside the changed files.
• pnpm exec vp run typecheck passed with exit 0.
• pnpm exec vp run lint:mobile passed with exit 0; optional SwiftLint, ktlint, and detekt checks were skipped because those tools are not installed locally.
• Focused archive storage, deletion, orchestration, migration, SQLite inspection, server snapshot handoff, client shell sync, sidebar, and mobile archive tests passed: 10 files, 214 tests.
• Review follow-up cache synchronization tests passed: 2 files, 26 tests; changed-file formatting and lint passed, and the @t3tools/client-runtime typecheck passed with one pre-existing Effect suggestion in src/relay/discovery.ts.
• Review follow-up unarchive finalization tests passed: 1 file, 16 tests; changed-file formatting and lint passed, and the server typecheck passed with three pre-existing Effect suggestions in src/orchestration/decider.ts.
• Review follow-up project/mobile archive sorting tests passed: 2 files, 97 tests; changed-file formatting and lint passed. Web and mobile package typechecks remain blocked only by pre-existing errors outside the changed files.
• Review follow-up cold-storage error tests passed: 2 files, 22 tests; changed-file formatting and lint passed, and the server typecheck passed with six pre-existing Effect suggestions in src/orchestration/decider.ts.
• Playwright against isolated ports 5213/14213 created and archived a disposable thread; after the exit animation, the row was absent from the rendered and keyboard-addressable sidebar and the project showed No threads yet.
• Playwright against isolated ports 5736/13776 paired successfully, rendered the authoritative empty shell, connected both HMR and application WebSockets, and reported no console or page errors.
• Playwright against isolated ports 5736/13776 rendered seeded project activity in the expected order without page exceptions; the exact optimistic archive transition is covered by the focused sidebar test.
• Playwright against isolated ports 5736/13776 showed the seeded thread in the command palette before optimistic archive, removed it afterward, and opened a fresh project draft instead of navigating to the hidden thread, with no console or page errors.
• The iOS development client built, installed, and launched on an iPhone 17 Pro simulator. The archive-screen UI pass was blocked before application code by this worktree's Metro resolution of strict pnpm symlinks; focused mobile archive-list tests cover both sort directions.

Proof

No attached visual proof is necessary; the data lifecycle and sidebar interaction are covered by the automated and isolated Playwright validation above.

---

Note

High Risk
Large architectural change to conversation persistence, destructive hot-row moves, and unarchive/restore boundaries; failures are designed to be retry-safe but bugs could lose or strand archived data.

Overview
Cold storage moves archived conversation SQL, events, and attachments out of hot state.sqlite into gzip-chunked bundles in archive.sqlite, leaving a lightweight shell for the Archive UI. A new ThreadColdStorage service handles archive, restore, rollback, finalize, and permanent delete, with per-tree locking, cleanup_pending retries, and incremental vacuum on both databases. Migrations 035/036 seed manifests and delete queues for existing archived and soft-deleted threads; t3-sqlite-state gains --database archive for inspection.

Lifecycle expands ThreadDeletionReactor into a background queue for archive, delete, and legacy compaction: it stops providers/terminals/log writers before archiving, coalesces failures into delayed rescans, and keeps delete cleanup until filesystem work succeeds. Unarchive in OrchestrationEngine restores (or hot-reserves) the cold bundle before the command commits, rolls back on dispatch failure, and finalizes the bundle only after success—including on receipt retry.

Clients: web sidebar, command palette, and project removal use optimistic archive hiding and per-member deleteArchivedThreads / force options; mobile archive lists omit invalid timestamps, sort them last, and show unarchive-in-progress. Mobile and web clear stale thread-detail caches on schema upgrade. Provider NDJSON loggers add closeThread for safe teardown during archive/delete.

^{Reviewed by Cursor Bugbot for commit 219c60c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Reduce archived conversation disk usage by introducing cold storage for threads

• Adds a ThreadColdStorageLive layer backed by the filesystem and SQLite that archives, restores, rolls back, and finalizes thread data, including provider event logs and attachments.
• Integrates cold storage into the orchestration engine: thread.unarchive dispatches now restore the archived bundle before appending, roll back on failure, and finalize after a successful commit.
• Adds ThreadDeletionReactor lifecycle job scheduling for archive, delete, and compact-legacy-storage operations with 30-second retry and deduplication.
• Introduces DB migrations 35 and 36 to create thread_archive_manifests and thread_cleanup_queue tables and backfill pending entries for existing archived/deleted threads.
• Adds optimistic archive hiding in the web and mobile UIs: threads disappear from the sidebar, command palette, and project sorting immediately on archive and reappear if the mutation fails.
• Evicts cached thread detail from IndexedDB (web, version bump to 5) and SQLite (mobile, schema version 2) for archived threads; stale cache writes are guarded by a per-thread generation counter.
• Extends project.delete to accept deleteArchivedThreads: true, allowing deletion of projects that contain only archived shells without requiring force.
• Risk: existing clients below IndexedDB version 5 will have their entire thread object store cleared on upgrade.

^{Macroscope summarized 219c60c.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c56f20e1-078d-42d4-90de-adc7aca0c4e5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Needs human review

Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

One Effect service convention issue found in the new ThreadColdStorage service. See the inline comment.

^{Posted via Macroscope — Effect Service Conventions}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3f385ab. Configure here.}

[macroscopeapp]

One convention finding on the new ThreadColdStorage service. The prior run's ThreadColdStorageShape concern has been resolved — the service interface is now inline in the Context.Service declaration.

^{Posted via Macroscope — Effect Service Conventions}