Reduce archived conversation disk usage with cold storage

.agents/skills/test-t3-app/SKILL.md

@@ -73,7 +73,7 @@ Always pass `--dev-url` for a dev-runner environment so the generated pairing UR
 
 Read [references/sqlite-fixtures.md](references/sqlite-fixtures.md) before changing the database.
 
-- Use `node apps/server/scripts/t3-sqlite-state.ts query` for schema discovery and read-only checks.
+- Use `node apps/server/scripts/t3-sqlite-state.ts query` for schema discovery and read-only checks. It targets `state.sqlite` by default; pass `--database archive` when inspecting cold archive manifests or chunks.
 - Stop the dev server before using `node apps/server/scripts/t3-sqlite-state.ts exec`, then restart it with the same base directory.
 - Seed projection tables only for disposable UI fixtures. Use application commands and APIs when testing business behavior or projection correctness.
 - Use the auth CLI, not direct `auth_*` table edits, for pairing and sessions.

.agents/skills/test-t3-app/references/sqlite-fixtures.md

@@ -4,7 +4,7 @@ Load this reference only when inspecting or seeding local T3 state directly.
 
 ## Select the correct database
 
-When `--base-dir` or `--home-dir` is explicit, runtime state lives under `<base-dir>/userdata` and the database path is `<base-dir>/userdata/state.sqlite`. The `<base-dir>/dev` state directory is only the fallback for an implicit development home, preventing an ordinary `vp run dev` from touching production state.
+When `--base-dir` or `--home-dir` is explicit, runtime state lives under `<base-dir>/userdata`. Hot projections and runtime state are stored in `state.sqlite`; cold archive manifests and compressed chunks are stored in `archive.sqlite`. The `<base-dir>/dev` state directory is only the fallback for an implicit development home, preventing an ordinary `vp run dev` from touching production state.
 
 Start the target runtime once before seeding so all migrations have run. Use an isolated base directory. Stop the server before writes to avoid racing application state or an active projection.
 
@@ -18,6 +18,15 @@ node apps/server/scripts/t3-sqlite-state.ts query \
   --sql "SELECT name FROM sqlite_schema WHERE type = 'table' ORDER BY name"
 ```
 
+The helper targets `state.sqlite` by default. Select the cold archive database explicitly for read-only inspection:
+
+```bash
+node apps/server/scripts/t3-sqlite-state.ts query \
+  --base-dir <base-dir> \
+  --database archive \
+  --sql "SELECT t.thread_id, t.archive_version, COUNT(c.chunk_index) AS chunk_count FROM archive_threads AS t LEFT JOIN archive_thread_chunks AS c ON c.thread_id = t.thread_id GROUP BY t.thread_id, t.archive_version ORDER BY t.thread_id"
+```
+
 Inspect current columns before writing a fixture:
 
 ```bash
@@ -36,6 +45,8 @@ node apps/server/scripts/t3-sqlite-state.ts exec \
 
 Use one statement per invocation for both `query` and `exec`; the helper wraps writes in a transaction and prints the backup path after a successful mutation. Use a single insert with multiple value rows when a fixture needs several records.
 
+Cold archive manifests and chunks are authoritative recovery data. Prefer `--database archive` for queries only. Exercise archive and restore behavior through application commands and APIs instead of constructing or changing archive bundles directly; only use archive writes for a deliberately isolated corruption or compatibility fixture.
+
 ## Seed projection data carefully
 
 The web UI primarily reads these projection tables:

BRANCH_DETAILS.md

@@ -0,0 +1,55 @@
+# Conversation Data Savings
+
+Archived conversations use cold storage instead of retaining full hot projections and diagnostics in `state.sqlite`.
+
+- `state.sqlite` keeps only the lightweight archived thread shell needed by the Archive page plus command receipts required for retry idempotency. Conversation events, messages, activities, turns, checkpoints, plans, session/runtime rows, and content attachments are written as bounded gzip-compressed chunks in the separate `archive.sqlite` database, then removed from hot storage. Command receipts remain hot until permanent deletion so retrying an accepted archive command returns its original sequence.
+- `apps/server/scripts/t3-sqlite-state.ts` is the supported isolated-database inspection path. It targets hot `state.sqlite` by default and accepts `--database archive` for cold manifest and chunk queries, with both paths derived from the shared server configuration. Archive and restore behavior should use application commands rather than direct archive-bundle writes.
+- Content attachments are part of the archive bundle and return on unarchive. Attachment collection follows exact ids persisted in thread messages, while retry cleanup may reuse exact attachment chunk filenames already recorded in the cold bundle; normalized thread-name collisions cannot claim another thread's files. Provider diagnostic logs and terminal history logs are deliberately destructive on archive: they are deleted, never copied into the bundle, and never restored. Provider-log cleanup matches only the exact thread log and its numeric rotations so similarly prefixed thread ids are not affected.
+- Cold restore preserves binary SQL values, validates attachment entry names, and atomically replaces attachment files before marking SQL rows restored. It keeps compressed chunks authoritative on failure, pages chunk reads to bound memory, rejects unknown tables/chunk kinds, and intersects archived row columns with the current schema so older bundles remain recoverable after compatible migrations.
+- Permanent thread deletion removes the shell, event stream, command receipts, every thread-owned projection/runtime/checkpoint row, attachments, terminal history, provider logs and rotations, and any cold-archive manifest/chunks. Exact attachment ownership metadata remains available until external attachment and provider-log cleanup succeeds, after which the SQL and cold-chunk rows are removed; cross-thread plan references are cleared rather than leaving dangling ids, and the durable cleanup-queue entry is retained until filesystem cleanup and free-page reclamation succeed so interrupted deletes retry safely. Startup recovery also rediscovers soft-deleted shells that have not reached the cleanup queue, covering commits made while the lifecycle reactor was offline.
+- Sidebar V2 project removal in `apps/web/src/components/SidebarV2.tsx` applies the archived-history warning and command options from `apps/web/src/components/Sidebar.logic.ts` independently to each selected project member. Members with known live threads use forced deletion; when a member has no live shell threads, it requests archived-thread deletion without force because archived shells are excluded from normal navigation state. The server removes archived cold bundles but still rejects any unseen live thread. Project removal emits the same per-thread deletion events for archived shells, so already-cold threads pass through the durable lifecycle worker and lose their hot shell, archive manifest, and compressed chunks before cleanup completes. `apps/web/src/components/Sidebar.logic.test.ts` covers standalone and grouped warning text plus the per-member thread partitioning that selects each member's command options, `apps/server/src/orchestration/decider.delete.test.ts` covers the archived-only live-thread precondition, and `apps/server/src/orchestration/Layers/ThreadDeletionReactor.test.ts` covers the forced-project decider, lifecycle reactor, and cold-storage boundary together.
+- Archive/delete filesystem work runs through a durable background lifecycle queue so command acknowledgement and the UI are not blocked by compression or large cleanup operations. Archive and restore lifecycle work is serialized per project tree, with reference-counted idle-lock eviction after the final user or waiter. Archive eligibility and provider/terminal/log-writer shutdown run under that same tree lock, so a queued archive job that becomes stale after unarchive cannot stop the newly active session or delete its terminal history. Archive creation uses a retry-safe two-phase boundary: compressed chunks and hot-row deletion commit before the manifest enters `cleanup_pending`, and destructive attachment/log cleanup must finish before it becomes `cold`. Destructive archive transitions recheck the archived shell inside the transaction and require terminal/log-writer shutdown to succeed. Provider shutdown also remains required while the projected session is `starting` or `running`; a missing provider binding is terminal for a settled or absent projected session, allowing legacy subagent shells that never owned a provider binding to move cold safely. Failed lifecycle work requests one coalesced delayed rescan of the durable pending state instead of being dropped or introducing a permanent polling loop, enqueue interruptions release their deduplication reservation, and restart recovery preserves the same retry boundary. Filesystem failures other than a genuinely missing directory also keep retry state. Incomplete cleanup, including a `cleanup_pending` manifest whose archived shell has already been removed, and archived shells missing a manifest resume after restart without rebuilding an already durable bundle.
+- Unarchive restores `cold` or `cleanup_pending` bundles before dispatching the domain command and treats an existing `restored` manifest as an in-flight or post-commit reservation without replaying its stale chunks over live rows or attachments. A rejected or failed command re-archives the restored rows and files, while a successful command finalizes the cold bundle only when that request actually performed a restore; this prevents unrelated or already-hot unarchive commands from deleting archive data. Once SQL restoration commits, later publication or metrics failures cannot re-archive the restored data. Retrying an accepted unarchive receipt also retries idempotent bundle finalization for that same thread, so a transient post-commit cleanup failure cannot strand restored archive chunks.
+- Unarchive reserves still-hot archived rows with a `restored` manifest before releasing the archive-tree lock. This prevents queued lifecycle work from moving those rows cold between the restore check and the unarchive command commit; command failure archives the reserved rows, while success removes the reservation. Process-local restore ownership distinguishes those live reservations from `restored` manifests abandoned by a crashed process, which startup lifecycle discovery moves cold again while the shell remains archived. If storage cannot restore or reserve the archived conversation, the command is rejected before an active-shell event can commit.
+- A `restored` manifest protects only the archive epoch that an unarchive command reserved. If post-commit bundle finalization fails and the user later archives the thread again, the newer shell timestamp lets lifecycle work replace the stale manifest and move the conversation cold.
+- Migration `035_ThreadColdArchive` registers existing archived conversations for background conversion on the next update. Migration `036_DeletedThreadCleanupQueue` registers legacy soft-deleted conversations for the same permanent cleanup used by new deletes.
+- After the legacy queues drain, a retryable one-time `VACUUM` physically compacts `state.sqlite` and enables incremental auto-vacuum. Later lifecycle operations reclaim bounded free-page batches from both `state.sqlite` and `archive.sqlite`, avoiding a full compaction on every archive.
+- Normal provider, server, trace, and terminal logging behavior is unchanged. Space is reclaimed at the conversation lifecycle boundary instead of by weakening diagnostics for active work.
+
+Primary files:
+
+- `apps/server/src/orchestration/Layers/ThreadColdStorage.ts`
+- `apps/server/src/orchestration/Layers/ThreadDeletionReactor.ts`
+- `apps/server/src/orchestration/Layers/ThreadDeletionReactor.test.ts`
+- `apps/server/src/persistence/Migrations/035_ThreadColdArchive.ts`
+- `apps/server/src/persistence/Migrations/036_DeletedThreadCleanupQueue.ts`
+- `apps/server/scripts/t3-sqlite-state.ts`
+- `apps/web/src/browser/previewThreadLifecycle.ts`
+- `apps/web/src/browser/usePreviewThreadLifecycleCleanup.ts`
+- `apps/web/src/components/settings/ArchivedThreadsPanel.tsx`
+- `apps/web/src/components/Sidebar.logic.ts`
+- `apps/web/src/components/Sidebar.logic.test.ts`
+- `apps/web/src/components/SidebarV2.tsx`
+
+## Sidebar And Shell Consistency
+
+Sidebar archive visibility is centralized across persisted and optimistic archive states. Project rows, project status, sorting, keyboard navigation, prewarming, and command-palette thread/project actions exclude an optimistically archived conversation immediately, so durable cold-storage work cannot leave a stale row or navigation target visible while the server shell catches up.
+
+The web Archive settings route renders `apps/web/src/components/settings/ArchivedThreadsPanel.tsx`, which reads archived snapshots across connected environments, groups archived threads by project, and exposes unarchive plus permanent-delete actions. Keeping this branch-owned surface separate from the general panels in `apps/web/src/components/settings/SettingsPanels.tsx` limits archive behavior to the Archive route.
+
+Authoritative shell synchronization is shared runtime behavior in `packages/client-runtime/src/state/shell.ts`, `packages/client-runtime/src/rpc/client.ts`, and `apps/server/src/ws.ts`, rather than a branch-specific subscription implementation. HTTP snapshots provide an early shell, while completion-capable WebSocket sessions revalidate it with a socket-owned snapshot after live buffering begins; the same refresh runs when the app returns to the foreground. Cold archive storage relies on this contract because compacted per-thread history cannot prove that cached projects and threads still exist.
+
+Persisted web and mobile thread details are also fast-paint caches, not an archive source of truth. A `thread.archived` detail event, an authoritative archived detail snapshot, successful local archive acknowledgement, or shell removal evicts the persisted detail through `packages/client-runtime/src/state/threadCache.ts`. Per-thread cache generations, eviction tombstones, and write locks prevent queued or idle-finalizer writes from recreating the snapshot. Shell additions and `thread.unarchived` events revive the cache while invalidating only writes captured during the tombstoned generation. Web IndexedDB migration version 5 and mobile SQLite migration version 2 each clear legacy thread-detail caches once; active details repopulate on demand.
+
+Failed shell-driven detail-cache removals remain pending while the authoritative shell omits the thread and retry on later shell items. An authoritative active detail snapshot also revives a cache tombstone before it queues persistence, covering reconnects where the corresponding unarchive event was compacted.
+
+`apps/server/src/orchestration/Layers/ThreadDeletionReactor.ts` schedules lifecycle cleanup when it observes a thread archive or deletion event, including deletions expanded from a project command; it makes a best-effort attempt to close every server preview session at event observation time so delayed archive work cannot close a preview reopened after unarchive. `apps/web/src/browser/ElectronBrowserHost.tsx` calls `apps/web/src/browser/usePreviewThreadLifecycleCleanup.ts`, whose renderer lifecycle effect uses the pure reconciliation logic in `apps/web/src/browser/previewThreadLifecycle.ts` to retain the last authoritative per-environment thread baseline while a shell synchronizes, then detect removals once that environment is live or discard the baseline when the environment leaves the catalog. It clears both `apps/web/src/previewStateStore.ts` and `apps/web/src/previewMiniPlayerStore.ts` state even when server preview-close delivery arrives first. Removing the preview state unmounts the cross-thread background webview, whose `apps/web/src/browser/desktopTabLifetime.ts` lease stops recording/capture before it closes the desktop tab. Unarchive therefore starts with clean preview state instead of reviving a stale mini-player or capture session. `apps/server/src/orchestration/Layers/ThreadDeletionReactor.test.ts`, `apps/web/src/browser/previewThreadLifecycle.test.ts`, and `apps/web/src/previewMiniPlayerStore.test.ts` cover the server and renderer cleanup boundaries.
+
+`apps/server/src/server.test.ts` verifies that HTTP-seeded shell subscriptions preserve archive removals published while WebSocket catch-up reads persisted events, then emit the synchronization completion marker. Initial-snapshot event replay in `apps/server/src/ws.ts` and deferred active-thread cache writes in `packages/client-runtime/src/state/threads.ts` remain complementary to authoritative shell refresh. Cache lifecycle behavior is covered through `packages/client-runtime/src/state/shell-sync.test.ts`, `packages/client-runtime/src/state/threadCommands.test.ts`, `packages/client-runtime/src/state/threads-sync.test.ts`, `apps/web/src/connection/storage.test.ts`, and the mobile storage/database tests.
+
+Mobile Archive rows omit invalid lifecycle timestamps instead of presenting corrupt or legacy values as newly archived, and sort missing or invalid archive dates after valid dates in both sort directions. General mobile time rendering is unchanged.
+
+## Development Ports
+
+- Web: `5736`
+- Server/WebSocket: `13776`

apps/mobile/src/features/archive/ArchivedThreadsRouteScreen.tsx

@@ -16,13 +16,18 @@ import {
   refreshArchivedThreadsForEnvironment,
   useArchivedThreadSnapshots,
 } from "./useArchivedThreadSnapshots";
+import type { EnvironmentThreadShell } from "@t3tools/client-runtime/state/shell";
+import { scopedThreadKey } from "../../lib/scopedEntities";
 
 export function ArchivedThreadsRouteScreen() {
   const { expand } = useClerkSettingsSheetDetent();
   const { savedConnectionsById } = useSavedRemoteConnections();
   const [searchQuery, setSearchQuery] = useState("");
   const [selectedEnvironmentId, setSelectedEnvironmentId] = useState<EnvironmentId | null>(null);
   const [sortOrder, setSortOrder] = useState<ArchivedThreadSortOrder>("newest");
+  const [unarchivingThreadKeys, setUnarchivingThreadKeys] = useState<ReadonlySet<string>>(
+    () => new Set(),
+  );
   const environments = useMemo<ReadonlyArray<ArchivedThreadsHeaderEnvironment>>(
     () =>
       Arr.sort(
@@ -67,6 +72,22 @@ export function ArchivedThreadsRouteScreen() {
   );
   const { unarchiveThread, confirmDeleteThread } =
     useArchivedThreadListActions(refreshChangedEnvironment);
+  const handleUnarchiveThread = useCallback(
+    async (thread: EnvironmentThreadShell) => {
+      const threadKey = scopedThreadKey(thread.environmentId, thread.id);
+      setUnarchivingThreadKeys((current) => new Set(current).add(threadKey));
+      try {
+        await unarchiveThread(thread);
+      } finally {
+        setUnarchivingThreadKeys((current) => {
+          const next = new Set(current);
+          next.delete(threadKey);
+          return next;
+        });
+      }
+    },
+    [unarchiveThread],
+  );
 
   useFocusEffect(
     useCallback(() => {
@@ -86,10 +107,11 @@ export function ArchivedThreadsRouteScreen() {
       onRefresh={refresh}
       onSearchQueryChange={setSearchQuery}
       onSortOrderChange={setSortOrder}
-      onUnarchiveThread={unarchiveThread}
+      onUnarchiveThread={(thread) => void handleUnarchiveThread(thread)}
       searchQuery={searchQuery}
       selectedEnvironmentId={selectedEnvironmentId}
       sortOrder={sortOrder}
+      unarchivingThreadKeys={unarchivingThreadKeys}
     />
   );
 }