[Bug] grep fails for the whole tree when one file has an oversized or non-UTF-8 line

[sahrizvi]

Description

A single file with a very long line — a minified bundle, a source map, a one-line JSON/CSV fixture — makes grep fail for the entire search, not just that file. Matches already collected from unrelated files are discarded along with it.

The same defect has a second trigger: a single line that is not valid UTF-8 anywhere in the tree, which fails with a different message (Invalid ripgrep match output).

Steps to Reproduce

In any repository, add one ordinary source file and one file with a very long single line:

printf 'const needle = 1\n' > src.ts
{ printf 'var a="%s";needle;var b="%s"\n' "$(printf 'x%.0s' {1..90000})" "$(printf 'y%.0s' {1..90000})"; } > bundle.min.js

altimate-code debug rg search needle

For the second trigger, add a file containing a non-UTF-8 byte:

printf 'needle \xff\xfe tail\n' > weird.txt

Expected Behavior

Matches are returned for every file that contains one. A file whose line cannot be handled should not affect the rest of the search.

Actual Behavior

The command fails outright and returns nothing — the match in src.ts is lost too:

Error: Unexpected error
Ripgrep JSON record exceeded 65536 bytes

Root cause

packages/core/src/ripgrep.ts parses ripgrep's --json output inside Stream.mapEffect, so any per-record failure aborts the whole stream. A match record embeds the entire matched line, so a long line exceeds the 64 KiB per-record ceiling and kills the search.

There are three ways one record can end a search — oversized, unparseable JSON, and schema rejection. The third also fires on valid ripgrep output: every path/lines/match field is a union of {"text": …} (valid UTF-8) and {"bytes": "<base64>"} (anything else), and only the text arm was modelled.

A second, separately reachable parser (packages/opencode/src/file/ripgrep.ts, behind the /find route) has the same defect.

Two things that are not the fix, both checked:

• Raising the buffer alone misses the point — Stream.splitLines has already materialized the line before the size check runs, so that ceiling never bounded the transient allocation. It did bound what the search retains, which has to be handled separately.
• ripgrep cannot prevent this at the source: --max-columns / --max-columns-preview are ignored by its JSON printer (verified — the record stayed 200,189 bytes).

Impact

Telemetry over a 7-day window attributed this to ~74 machines on 0.9.3 and 0.9.4. Any repository containing a bundled or minified asset can hit it, and the symptom is a totally failed search rather than a degraded one.

Environment

• OS: macOS 15 (platform-independent — the parser is shared)
• CLI version: 0.9.3, 0.9.4