Skip to content

Add telemetry to diagnose intermittent NES model switching#308755

Draft
ulugbekna wants to merge 2 commits intomainfrom
ulugbekna/nes-model-switch-telemetry
Draft

Add telemetry to diagnose intermittent NES model switching#308755
ulugbekna wants to merge 2 commits intomainfrom
ulugbekna/nes-model-switch-telemetry

Conversation

@ulugbekna
Copy link
Copy Markdown
Contributor

Problem

Investigation of provideinlineedit telemetry revealed that ~0.04% of sessions with successful NES fetches experience intermittent model switching — the NES model bounces between two different models (e.g., nes-callisto-003nes-callistones-callisto-003) mid-session.

We couldn't determine the root cause from existing telemetry because the modelConfig property didn't include how the model was selected (fetched from /models, experiment config, hardcoded default, etc.).

Changes

1. nesModelChanged telemetry event

Fires whenever _currentModelObs produces a different model than the previous one. Captures:

  • previousModel / newModel — model names before and after
  • newModelSource — how the model was picked (fetched, expConfig, expDefaultConfig, localConfig, hardCodedDefault)
  • modelListNames / modelListSources — full snapshot of the aggregated model list
  • fetchedNesModels — raw NES models from latest /models response
  • hasCopilotToken / isFreeUser — token state at switch time

2. source field in modelConfig JSON on every provideInlineEdit event

The existing modelConfig telemetry property now includes a source field, so every NES request self-documents how its model was chosen. Query via:

| extend modelSource = tostring(parse_json(Properties["modelconfig"]).source)

3. selectedModel() API on IInlineEditsModelService

Replaces the old selectedModelConfiguration() with selectedModel() returning { config, source } atomically from a single observable read — avoids a race between config and source.

Possible causes under investigation

  1. Server /models endpoint returning different NES models across token refreshes
  2. Transient loss of /models data falling through to hardcoded defaults
  3. Experiment assignment changes mid-session

This telemetry will confirm which.

- Add 'nesModelChanged' event that fires when the selected NES model
  changes mid-session, capturing source, model list, and token state
- Include model source in provideinlineedit's modelConfig JSON via
  new selectedModel() method that returns config+source atomically
- This helps diagnose whether switches are caused by /models endpoint
  changes, experiment flips, or fallback to hardcoded defaults
Copilot AI review requested due to automatic review settings April 9, 2026 11:03
@ulugbekna ulugbekna marked this pull request as draft April 9, 2026 11:04
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds diagnostic telemetry around Next Edit Suggestions (NES) model selection to help identify the cause of rare mid-session model switching, and updates the inline edits model service API to return model + source atomically.

Changes:

  • Add a new nesModelChanged MSFT telemetry event emitted when the selected NES model name changes.
  • Include a source field in the serialized modelConfig telemetry for each provideInlineEdit flow.
  • Replace selectedModelConfiguration() with selectedModel() returning { config, source } to avoid config/source races.
Show a summary per file
File Description
extensions/copilot/src/platform/inlineEdits/node/inlineEditsModelService.ts Adds nesModelChanged telemetry via an autorun and introduces the new selectedModel() implementation.
extensions/copilot/src/platform/inlineEdits/common/inlineEditsModelService.ts Updates the service interface to selectedModel(): SelectedModel.
extensions/copilot/src/extension/xtab/node/xtabProvider.ts Plumbs model source into modelConfig telemetry JSON and switches to selectedModel().
extensions/copilot/src/extension/xtab/test/node/xtabProvider.spec.ts Updates mocks and tests to use selectedModel() instead of selectedModelConfiguration().

Copilot's findings

  • Files reviewed: 4/4 changed files
  • Comments generated: 2

…ranch

- Move hasCopilotToken/isFreeUser to measurements bag as numbers (1/0)
  instead of string properties, matching the isMeasurement GDPR annotation
- Add modelSource to forceUseDefaultModel branch return value and update
  determineModelConfiguration return type to always include modelSource
  so source is never undefined in the serialized modelConfig JSON
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants