chat: Bound external ingest reconciliation scans - #331053
Draft
roblourens wants to merge 1 commit into
Draft
Conversation
Cap external-ingest workspace discovery before remote search results can exhaust the renderer heap. Preserve the existing database and skip ingestion when discovery is incomplete, while allowing later retries. Refs #328547 (Written by Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Bounds external-ingest reconciliation to prevent memory exhaustion in large workspaces.
Changes:
- Adds capped workspace discovery with truncation reporting.
- Preserves stored index data and supports reconciliation retries.
- Adds coverage for truncated scans and retries.
Show a summary per file
| File | Description |
|---|---|
externalIngest.spec.ts |
Tests bounded reconciliation and retry behavior. |
externalIngestIndex.ts |
Caps discovery and gates ingestion on reconciliation. |
codeSearchRepo.ts |
Adds a workspace-scan error. |
searchServiceImpl.ts |
Reports search-limit information around exclusions. |
searchService.ts |
Adds the limit-aware search API. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 5/5 changed files
- Comments generated: 3
- Review effort level: Balanced
Comment on lines
36
to
+40
| const results = await this._findFilesWithDefaultExcludesAndExcludes(include, copilotIgnoreExclude, maxResults, token); | ||
| if (!this._ignoreService.isRegexExclusionsEnabled || !results) { | ||
| return results; | ||
| } else if (Array.isArray(results)) { | ||
| return await filterIngoredResources(this._ignoreService, results); | ||
| } else { | ||
| return await this._ignoreService.isCopilotIgnored(results) ? undefined : results; | ||
| } | ||
| const files = Array.isArray(results) ? results : results ? [results] : []; | ||
| return { | ||
| files: this._ignoreService.isRegexExclusionsEnabled ? await filterIngoredResources(this._ignoreService, files) : files, | ||
| limitReached: maxResults !== undefined && files.length >= maxResults, |
Comment on lines
905
to
+906
| for (const folder of workspaceFolders) { | ||
| const remainingFileCount = maxExternalIngestFileCount - candidateFiles.size; |
| } | ||
|
|
||
| await this.reconcileDbFiles(); | ||
| this._isReconciliationComplete = await this.reconcileDbFiles(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #328547
Summary
Why
The crash dump attached to #328547 shows the workbench renderer terminating after V8 heap exhaustion while accumulating remote file-search matches. External-ingest reconciliation previously searched each workspace folder with
**/*,Number.MAX_SAFE_INTEGER, and no cancellation token. In a broad Remote SSH workspace, that can retain millions of search results in the renderer.This deliberately avoids a global search cap. It only bounds external-ingest initialization and refuses to reconcile or upload from an incomplete filesystem view.
Validation
npm run test:unit -- src/platform/workspaceChunkSearch/test/node/externalIngest.spec.ts(21 passed)npx tsc --noEmit --project tsconfig.jsonfromextensions/copilotgit diff --checkReviewer notes
The 100,000 limit matches the existing default local workspace-index maximum. Reaching the limit is treated conservatively as possible truncation, so external ingest is unavailable for workspaces at or above that scale until the workspace is narrowed or exclusions reduce the raw search result count. Existing database contents are preserved in that case.
(Written by Copilot)