[Remote-SSH] Extension Host with Copilot Fails on VS Code 1.95.3 but Works on 1.93.0

[technic960183]

Type: Bug

Description

We are encountering an issue with the GitHub Copilot extension when using Visual Studio Code with Remote-SSH on an HPC login node. The extension works flawlessly with VS Code 1.93.0 (4849ca9) but fails on VS Code 1.95.3 (f1a4fb1). Interestingly, VS Code 1.95.3 works fine on the computing node of the same cluster, accessed via SSH forwarding from the login node.

To eliminate variables, we have tested the setup thoroughly:

• Two personal computers were used:
  • One running VS Code 1.93.0, Copilot 1.245.0 and Copilot Chat 0.20.? (Will recheck this on Monday.).
  • The other running VS Code 1.95.3, Copilot 1.245.0 and Copilot Chat 0.22.4.
• Both with GitHub Copilot 1.245.0 and GitHub Copilot Chat 0.22.4
• Both connect to the same HPC login node and same account using Remote-SSH.
• Before each login, the .vscode-server directory on the remote machine was cleaned up to ensure a fresh environment.
• On the remote machine, GitHub Copilot and GitHub Copilot Chat were the only extensions installed.

Expected Behavior

The GitHub Copilot extension should initialize and function correctly on both VS Code versions.

Actual Behavior

• On VS Code 1.95.3, the Copilot extension causes the Extension Host process to crash only on the login node.
• The issue does not occur on the computing node or when using VS Code 1.93.0 on the login node.
• Disabling the GitHub Copilot extensions eliminates the issue.

Test by VS Code Bisect

Done on our login node with VS Code 1.95.3. Result:
Extension Bisect is done and has identified github.copilot as the extension causing the problem.

Before, we don't know that it is possible to disable GitHub Copilot alone but not GitHub Copilot Chat.
Later, we found that when VS Code bisect disable GitHub Copilot 1.245.0 alone, GitHub Copilot Chat 0.22.4 works normally on our login node with VS Code 1.95.3.

Logs

Below are logs captured from the Extension Host (remote) process on both the login and computing nodes running VS Code 1.95.3. These logs highlight differences observed during initialization:

Login Node (Fails)

2024-11-21 16:34:43.523 [trace] ExtHostCommands#registerCommand github.copilot.openLogs
2024-11-21 16:34:43.523 [trace] ExtHostCommands#registerCommand github.copilot.signIn
2024-11-21 16:34:43.530 [trace] ExtensionService#_callActivateOptional GitHub.copilot-chat
2024-11-21 16:34:43.555 [trace] extHostWorkspace#findFiles2: fileSearch, extension: GitHub.copilot-chat, entryPoint: findFiles2
2024-11-21 16:34:43.556 [trace] ProxyResolver#tls.connect [{"highWaterMark":16384,"servername":"default.exp-tas.com","session":"null","localAddress":"null","ALPNProtocols":"http/1.1","port":443,"host":"default.exp-tas.com"}]
2024-11-21 16:34:43.562 [debug] ProxyResolver#resolveProxy unconfigured http://169.254.169.254/metadata/instance/compute DIRECT 
2024-11-21 16:34:43.693 [trace] ProxyResolver#tls.connect [443, "default.exp-tas.com", {"servername":"default.exp-tas.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.723 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.760 [debug] ExtHostSearch /work1/user141421/.vscode-server/cli/servers/Stable-f1a4fb101478ce6ec82fe9627c43efbf9e98c813/server/node_modules/@vscode/ripgrep/bin/rg --files --hidden --case-sensitive --no-require-git -g '!**/.git' -g '!**/.svn' -g '!**/.hg' -g '!**/CVS' -g '!**/.DS_Store' -g '!**/Thumbs.db' -g '!**/node_modules' -g '!**/bower_components' -g '!**/*.code-search' --no-ignore-parent --follow --no-config --no-ignore-global
 - cwd: /home/user141421
 - Sibling clauses: {}
2024-11-21 16:34:43.964 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.998 [trace] ExtHostCommands#registerCommand github.copilotChat.signIn
[skip some lines]
2024-11-21 16:34:44.005 [trace] ExtHostCommands#registerCommand github.copilot.buildLocalWorkspaceIndex
2024-11-21 16:34:44.020 [debug] ExtHostSearch Search finished. Stats: {"cmdTime":266,"fileWalkTime":266,"directoriesWalked":0,"filesWalked":0,"cmdResultCount":28852}
2024-11-21 16:34:44.020 [debug] Ext host file search time: 266ms
2024-11-21 16:34:44.174 [trace] ExtHostCommands#registerCommand codereferencing.showOutputPane
2024-11-21 16:34:44.176 [trace] ProxyResolver#tls.connect [443, "copilot-telemet[39 chars]", {"servername":"copilot-telemet[39 chars]","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:44.215 [trace] ExtHostCommands#executeCommand setContext
2024-11-21 16:34:44.215 [trace] ExtHostCommands#executeCommand _setContext
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.generate
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.acceptCursorPanelSolution
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.previousPanelSolution
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.nextPanelSolution
2024-11-21 16:34:44.216 [trace] ExtHostCommands#registerCommand _github.copilot.ghostTextPostInsert
[Process Crash]

Computing Node (Works)

2024-11-21 16:30:49.433 [trace] ExtHostCommands#registerCommand github.copilot.openLogs
2024-11-21 16:30:49.433 [trace] ExtHostCommands#registerCommand github.copilot.signIn
[Compare to above, no extra message here.]
2024-11-21 16:30:49.454 [trace] ExtHostCommands#registerCommand github.copilotChat.signIn
[skip the same lines as above]
2024-11-21 16:30:49.459 [trace] ExtHostCommands#registerCommand github.copilot.buildLocalWorkspaceIndex
2024-11-21 16:30:49.664 [trace] ProxyResolver#tls.connect [443, "default.exp-tas.com", {"servername":"default.exp-tas.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:49.694 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:49.981 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:50.116 [trace] ExtHostCommands#registerCommand codereferencing.showOutputPane
2024-11-21 16:30:50.118 [trace] ProxyResolver#tls.connect [443, "copilot-telemet[39 chars]", {"servername":"copilot-telemet[39 chars]","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:50.156 [trace] ExtHostCommands#executeCommand setContext
2024-11-21 16:30:50.156 [trace] ExtHostCommands#executeCommand _setContext
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.generate
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.acceptCursorPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.previousPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.nextPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand _github.copilot.ghostTextPostInsert
2024-11-21 16:30:50.176 [debug] ProxyResolver#loadSystemCertificates count 137
2024-11-21 16:30:50.191 [debug] ProxyResolver#loadSystemCertificates count filtered 134
2024-11-21 16:30:50.191 [debug] ProxyResolver#resolveProxy unconfigured https://mobile.events.data.microsoft.com/OneCollector/1.0?cors=true&content-type=application/x-json-stream DIRECT 
2024-11-21 16:30:50.192 [trace] ProxyResolver#tls.connect [{"protocol":"https:","hostname":"mobile.events.d[32 chars]","port":443,"path":"null","method":"POST","headers":"[object Object]","agent":"[object Object]","_defaultAgent":"[object Object]","host":"mobile.events.d[32 chars]","lookupProxyAuthorization":"[Function: bound dz]","noDelay":true,"servername":"mobile.events.d[32 chars]","secureEndpoint":true,"_vscodeAdditionalCaCerts":"[134 certs]","keepAlive":true,"scheduling":"lifo","timeout":5000,"_agentKey":"mobile.events.d[57 chars]","encoding":"null","keepAliveInitialDelay":1000}]
[Process keep running without crashing]
2024-11-21 16:30:50.176 [debug] ProxyResolver#loadSystemCertificates count 137
2024-11-21 16:30:50.191 [debug] ProxyResolver#loadSystemCertificates count filtered 134
2024-11-21 16:30:50.191 [debug] ProxyResolver#resolveProxy unconfigured https://mobile.events.data.microsoft.com/OneCollector/1.0?cors=true&content-type=application/x-json-stream DIRECT 
2024-11-21 16:30:50.192 [trace] ProxyResolver#tls.connect [{"protocol":"https:","hostname":"mobile.events.d[32 chars]","port":443,"path":"null","method":"POST","headers":"[object Object]","agent":"[object Object]","_defaultAgent":"[object Object]","host":"mobile.events.d[32 chars]","lookupProxyAuthorization":"[Function: bound dz]","noDelay":true,"servername":"mobile.events.d[32 chars]","secureEndpoint":true,"_vscodeAdditionalCaCerts":"[134 certs]","keepAlive":true,"scheduling":"lifo","timeout":5000,"_agentKey":"mobile.events.d[57 chars]","encoding":"null","keepAliveInitialDelay":1000}]

Additional Notes

• Network configuration remains the same across tests on both the login node and the computing node. Firewall are disable for all nodes.
• The system administrator has confirmed that no OS-level rules (e.g., process termination) are in effect on the login node.
• All of the hardware been exactly the same for the login node and the computing nodes in our cluster.

Key questions

• Why it works on VS Code 1.93.0 but not on VS Code 1.95.3 ?
• Why VS Code 1.95.3 works on our computing node but not on the login node?

VS Code version: Code 1.95.3 (f1a4fb1, 2024-11-13T14:50:04.152Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 5.15.0-78-generic

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (8 x 2808)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	7.89GB (1.49GB free)
Process Argv	--log info --crash-reporter-id edf07c68-6f96-49c4-a777-c6a188cd4542
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: Spock
OS	Linux x64 5.15.0-78-generic
CPUs	AMD Ryzen Threadripper PRO 5975WX 32-Cores (64 x 1793)
Memory (System)	251.70GB (244.89GB free)
VM	0%

Extensions (7)

Extension	Author (truncated)	Version
copilot	Git	1.245.0
copilot-chat	Git	0.22.4
jupyter-keymap	ms-	1.1.2
remote-ssh	ms-	0.115.1
remote-ssh-edit	ms-	0.87.0
remote-explorer	ms-	0.4.3
vscode-speech	ms-	0.12.1

(1 theme extensions excluded)

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
g316j359:31013175
dvdeprecation:31068756
dwnewjupytercf:31046870
nativerepl2:31139839
pythonrstrctxt:31112756
cf971741:31144450
iacca1:31171482
notype1cf:31157160
5fd0e150:31155592
dwcopilot:31170013
stablechunks:31184530

[lramos15]

Does Copilot Chat work?

Please try using VS Code bisect to help us narrow down the version which broke this

[technic960183]

Thank you for your response.

The result of VS Code bisect confirm that the issue is caused by the github.copilot extension (Version 1.245.0).

When VS Code bisect disable GitHub Copilot alone, GitHub Copilot Chat works normally.

Additionally, we have verified that both GitHub Copilot (1.245.0) and GitHub Copilot Chat (0.22.4) extensions are using the same version on VS Code 1.93.0 and VS Code 1.95.3 during our previous tests. (Wrong information, we are sorry.) I've updated these information to the issue description.

Please let us know if further information is needed.

[lramos15]

Are you saying that you are using copilot 1.245.0 on both 1.93 and 1.95.3. It's not possible for Copilot Chat 0.22.4 to be used on 1.93 due to breaking API changes.