draft: database layer refactor#962
Draft
ZacharyZcR wants to merge 180 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status
Draft / important branch. This PR is the long-running database layer refactor branch. It is continuing repository-boundary migration while keeping every migrated domain behind gray-rollout controls.
Goal
Replace the current in-memory encrypted SQLite snapshot model with a safer persistent database architecture that can support SQLite, PostgreSQL, and MySQL/MariaDB.
Gray rollout scope
DATABASE_LAYER_REPOSITORY_ROLLOUT.Current progress
SettingsRepository.HostRepositoryandCredentialRepository.HostRepositoryandCredentialRepository.DismissedAlertRepository.HomepageLayoutRepository.HomepageItemRepository.NetworkTopologyRepository.DashboardServiceLinkRepository.SessionRecordingRepository.CommandHistoryRepository.RecentActivityRepository.SshCredentialUsageRepository.TransferRecentRepository.FileManagerBookmarkRepository.FileManagerBookmarkRepository.C2sTunnelPresetRepository.TmuxSessionTagRepository.OpksshTokenRepository.VaultTokenRepository.VaultProfileRepository.HostMetricsPreferenceRepository.HostHealthRepository.HostMetricsHistoryRepository.AlertRepository.UserDataExportRepository.RecentActivityRepository, with dashboard host access checks routed through current host resolution, role, and RBAC access repositories.HostFolderRepository.HostResolutionRepository.HostResolutionRepositorywhile preserving request-user credential decryption behavior.CredentialRepository, preserving user-key field encryption and system-key copies for shared credentials.RawSqliteUserEncryptionMigrationStore, leavingDataCryptoandLazyFieldEncryptionon a storage boundary.DataCryptocurrent-runtime migration boundary, removing direct SQLite opening fromauth-manager.tsand keeping current SQLite resolution behindcreateCurrentUserEncryptionMigrationStore.DatabaseSaveTrigger, removing direct SQLite snapshot-save imports from those routes.DatabaseSaveTrigger.withSqliteForeignKeysDisabled, restoring constraints through afinallypath, routed the import route through the current SQLite boundary without importinggetDb(), and centralized current SQLite resolution throughcurrent-repository-runtime.LegacySqliteDatabaseCopyStore, leavingDatabaseMigrationas the file backup/encrypt/rename orchestrator.HostRepository.CredentialRepository.termix_identityrepository rollout domain and migrated Termix ID identity handle CRUD/resolution, public key publish/list/update/delete, linked credential lookup, and certificate target key lookup toTermixIdentityRepository.termix_identity_carepository rollout domain and migrated Termix ID CA public lookup, encrypted private-key create/rotate/delete, and certificate signing reads toTermixIdentityCaRepository.UserRepository.HostResolutionRepository.HostResolutionRepository.HostRepository, preserving user-key field encryption.HostResolutionRepository.HostRepositoryandUserRepository.HostResolutionRepositorywhile preserving route-level own-host decryption.HostRepository, with overwrite lookup and credential fallback reads routed through current host resolution and credential repositories.HostRepository.snippetsrepository rollout domain and migrated snippet folder list, owned lookup, visible-list owned reads, reorder, create/update/delete, export reads, bulk import, and folder create/metadata/rename/delete writes toSnippetRepository.SnippetRepository, and host folder record cleanup toHostFolderRepository.shared_credentialsrepository rollout domain and migrated shared credential material create/update/delete, pending re-encryption, and user cleanup persistence toSharedCredentialRepository.SimpleDBOpsdependencies from host metrics, host metrics viewer, tmux monitor, Docker, tunnel, and Proxmox paths; those gates now useDataCryptodirectly.SimpleDBOpscompatibility helper after source imports reached zero..envare copied under<DATA_DIR>/backups/pre-database-layer-refactor-<timestamp>/, with a marker to avoid repeat backups and fail-closed startup unless explicitly skipped.current-repository-runtimeand migrated settings, user, host, API key, session, trusted device, audit log, user preference, open tab, dismissed alert, homepage layout/item, network topology, dashboard service link, session recording, command history, recent activity, transfer recent, file-manager bookmark, C2S tunnel preset, tmux session tag, OPKSSH token, Vault token/profile, SSH credential usage, role, SSO provider, host folder, alert, host health, host metrics preference/history, credential, host resolution, snippet, shared credential, Termix ID identity/CA, RBAC access, and user data export current repository factories to shared SQLite context and write-save hook construction.Gray controls
DATABASE_LAYER_REPOSITORY_ROLLOUT=allenables all migrated repository domains.DATABASE_LAYER_REPOSITORY_ROLLOUT=settings,users,sessions,api_keys,trusted_devices,credentials,termix_identity,termix_identity_ca,hosts,snippets,sso_providers,audit_logs,user_preferences,open_tabs,dismissed_alerts,homepage_layouts,homepage_items,network_topology,dashboard_service_links,session_recordings,command_history,recent_activity,ssh_credential_usage,transfer_recent,file_manager_bookmarks,c2s_tunnel_presets,tmux_session_tags,opkssh_tokens,vault_tokens,vault_profiles,host_metrics_preferences,host_health,host_metrics_history,alerts,user_data_exports,host_folders,host_resolution,roles,rbac_access,shared_credentialsenables the intended gray slice explicitly.DATABASE_LAYER_REPOSITORY_ROLLOUT=offdisables migrated repository domains and fails closed.settings,users.repository_rollout_configwith the parsed mode and domains.GET /database/migration/statusincludesrepositoryRolloutfor live gray verification.DATABASE_LAYER_SKIP_PREUPGRADE_BACKUP=1bypasses the automatic pre-upgrade backup only when an external backup has already been verified.DATABASE_LAYER_PREUPGRADE_BACKUP_KEEP=<count>controls how many automatic pre-upgrade backups are retained; the default is 3.Pre-gray validation
npm run type-checkdatabase.ts; remainingfile_manager_*text is legacy SQLite import/export file formatdatabase.ts; remaining text is summary variables or repository-owned accessgit diff --checkDOMAIN_ALIASES/DOMAINSidentifier text matched the broadAIsubstring checkIntended direction