VS F# editor: race in Tokenizer `SourceTextData` cache can corrupt per-line token data under concurrent access

[xperiandri]

Please provide a succinct description of the issue.

Tokenizer stores per-line tokenization/classification state in a shared per-document cache. That cache is read and updated from concurrent editor paths (e.g., classification/tagging + symbol lookup). With non-thread-safe storage this can race and produce inconsistent/corrupted cache state.

Repro steps

Provide the steps required to reproduce the problem:

1. Open a medium/large F# file in Visual Studio.
2. Trigger concurrent editor activity (rapid edits/scrolling, semantic classification refreshes, and repeated symbol-at-caret requests).
3. Observe intermittent failures or unstable behavior in token/classification flows (historically seen as index/cache inconsistency under contention).

If possible attach a zip file with the repro case. This often makes it easier for others to reproduce.
The zip file should ideally represent the situation just before the call/step that is problematic.

Expected behavior

Per-line tokenization cache should remain consistent under concurrent reads/writes and never corrupt internal state.

Actual behavior

Shared cache operations can race under concurrent callers and may lead to inconsistent line cache state and downstream failures.

Known workarounds

No reliable user workaround; reducing concurrent editor activity can make repro less frequent.

Related information

• Operating system: Windows
• .NET Runtime kind (.NET Framework, .NET 8, .NET 10 in repo targets)
• Editing Tools: Visual Studio 2026 Insiders (18.9)

---

Proposed fix (based on current Tokenizer changes)

In vsintegration/src/FSharp.Editor/LanguageService/Tokenizer.fs:

1. Replace non-thread-safe per-line storage with thread-safe storage in SourceTextData:
   • use ConcurrentDictionary<int, SourceLineData>
   • keep Item getter/setter semantics via TryGetValue, index assignment, TryRemove
2. Keep explicit invalidation API ClearFrom(n) that removes cached lines from n forward.
3. Preserve existing call sites that invalidate after edits (e.g., sourceTextDataCache.ClearFrom(endLine + 1)).
4. Keep per-document cache as ConcurrentDictionary<string list, SourceTextData> so cache entries are safe across concurrent access.

This keeps behavior the same functionally, while making per-line cache mutation safe when multiple editor features touch the same document cache concurrently.

[xperiandri]

@T-Gro I had been catching exceptions there during debugging before this fix