TypeScript file Watcher consuming over 5k watchers #46351

oof2win2 · 2021-10-12T15:30:16Z

Issue Type: Bug

Since the latest update of Insiders, it has begun to use a new file watcher (from what I see here and the new Files: Legacy Watcher setting). My issue however is that this is not working and file watchers are still being consumed.

This is the output of inotify-consumers, npm i and then running inotify-consumers again. To note, node_modules/ was removed prior to the first consumer check.

oof2win2@koelkast:/opt/factorio/AwF-Bot-Test$ rm -rf node_modules/
oof2win2@koelkast:/opt/factorio/AwF-Bot-Test$ ~/cmds/inotify-consumers 

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
     235  2812552  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node --max-old-space-size=3072 /home
      96  2812533  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
       4    46817  /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
       4  2812522  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
       4  2812450  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
       2    46787  /lib/systemd/systemd --user
       1    46787  /lib/systemd/systemd --user
       1    46787  /lib/systemd/systemd --user
       0  2812582  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i

     347     WATCHERS TOTAL COUNT
oof2win2@koelkast:/opt/factorio/AwF-Bot-Test$ npm i

npm installation here

oof2win2@koelkast:/opt/factorio/AwF-Bot-Test$ ~/cmds/inotify-consumers 

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
    2540  2812552  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node --max-old-space-size=3072 /home
    2305  2812582  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
      96  2812533  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
       4    46817  /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
       4  2812522  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
       4  2812450  /home/oof2win2/.vscode-server-insiders/bin/284e0db6c4cac1e557a43cd6691babdaafc2e1a9/node /home/oof2win2/.vscode-server-i
       2    46787  /lib/systemd/systemd --user
       1    46787  /lib/systemd/systemd --user
       1    46787  /lib/systemd/systemd --user

    4957     WATCHERS TOTAL COUNT
oof2win2@koelkast:/opt/factorio/AwF-Bot-Test$

Here are my settings.json. Used this as just "**/node_modules/**: true, which is included, did not work.

{
  "files.watcherExclude": {
    "**": true,
    "**/**": true
  }
}

VS Code version: Code - Insiders 1.62.0-insider (284e0db6c4cac1e557a43cd6691babdaafc2e1a9, 2021-10-12T05:44:21.888Z)
OS version: Darwin x64 20.5.0
Restricted Mode: No
Remote OS version: Linux x64 5.4.0-80-generic

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-1030NG7 CPU @ 1.10GHz (8 x 1100)
GPU Status	2d_canvas: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled webgl: enabled webgl2: enabled
Load (avg)	3, 3, 3
Memory (System)	8.00GB (0.11GB free)
Process Argv	--crash-reporter-id 11b22232-fe4c-4ed0-a0a1-d5910ebb956b
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: awf
OS	Linux x64 5.4.0-80-generic
CPUs	Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz (12 x 1200)
Memory (System)	31.27GB (4.74GB free)
VM	0%

Extensions (4)

Extension	Author (truncated)	Version
remote-ssh	ms-	0.65.8
remote-ssh-edit	ms-	0.65.8
todo-tree	Gru	0.0.214
html-preview-vscode	tht	0.2.5

A/B Experiments

vsliv695:30137379
vsins829:30139715
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
pythonvspyt602:30291494
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30291487
pythontb:30258533
pythonvspyt551:30291412
pythonptprofiler:30281269
vsdfh931cf:30280410
vshan820:30294714
pythondataviewer:30285072
pythonvsuse255:30319630
vscod805:30301674
pythonvspyt200:30323110
vsccppwt:30377800
pythonvssor306:30340298
bridge0708:30335490
pygetstartedt2:30353727
bridge0723:30353136
pythonrunftest32:30365365
pythonf5test824:30361779
javagetstartedt:30350119
pythonvspyt187:30365360
pydsgst2:30361790
vsqsis400:30374798
vsaa593cf:30376535

The text was updated successfully, but these errors were encountered:

bpasero · 2021-10-13T05:13:51Z

@oof2win2 thanks for testing this. As a first question, can you run code-insiders --verbose, open a folder and then in devtools see which file watcher is being used? You should see something like this:

It would also give a hint what excludes are actually being used.

oof2win2 · 2021-10-13T06:12:49Z

No watcher seems to be used?

When I open a normal window (non-SSH), it shows me Parcel is being used with this output

bpasero · 2021-10-13T06:48:51Z

@oof2win2 oh, so this is an issue you see with connecting remotely via SSH to a Linux remote? If that is the case you will have to change the log level via this command:

To "trace" and then see the output via "Log (Server)" in output panel.

oof2win2 · 2021-10-13T06:51:40Z

Ah, then the output is this:

[2021-10-13 06:50:20.658] [remoteagent] [debug] [File Watcher (parcel)] Request to start watching: /opt/factorio/AwF-Bot-Test (excludes: **/.git/objects/**,**/.git/subtree-cache/**,**/node_modules/*/**,**/.hg/store/**,**,**/**,/home/oof2win2/.vscode-server-insiders/extensions/**),/opt/factorio/servers/test/scenarios/AwF-Scenario (excludes: **/.git/objects/**,**/.git/subtree-cache/**,**/node_modules/*/**,**/.hg/store/**,**,**/**,/home/oof2win2/.vscode-server-insiders/extensions/**)
[2021-10-13 06:50:20.658] [remoteagent] [debug] [File Watcher (parcel)] Request to stop watching: 
[2021-10-13 06:50:20.659] [remoteagent] [debug] [File Watcher (parcel)] Started watching: '/opt/factorio/AwF-Bot-Test' with backend 'inotify' and native excludes '/opt/factorio/AwF-Bot-Test/.git/objects, /opt/factorio/AwF-Bot-Test/.git/subtree-cache, /opt/factorio/AwF-Bot-Test/node_modules, /opt/factorio/AwF-Bot-Test/.hg/store, /opt/factorio/AwF-Bot-Test'
[2021-10-13 06:50:20.659] [remoteagent] [debug] [File Watcher (parcel)] Started watching: '/opt/factorio/servers/test/scenarios/AwF-Scenario' with backend 'inotify' and native excludes '/opt/factorio/servers/test/scenarios/AwF-Scenario/.git/objects, /opt/factorio/servers/test/scenarios/AwF-Scenario/.git/subtree-cache, /opt/factorio/servers/test/scenarios/AwF-Scenario/node_modules, /opt/factorio/servers/test/scenarios/AwF-Scenario/.hg/store, /opt/factorio/servers/test/scenarios/AwF-Scenario'

bpasero · 2021-10-13T07:02:28Z

So it looks like 2 file watchers are installed (with parcel) and /opt/factorio/AwF-Bot-Test/node_modules and /opt/factorio/servers/test/scenarios/AwF-Scenario/node_modules are excluded. Do you think that does not work?

oof2win2 · 2021-10-13T07:05:15Z

Yeah, I think that doesn't work. The excludes have ** and **/** in them, so everything should be excluded, right?

oof2win2 · 2021-10-13T07:09:35Z

~~Nevermind, seems that the node_modules exclusion does work. It was **/.git/** that reduced the watchers to just 70. Still think that ** and **/** should work though~~ Seems I was wrong there. The file watchers don't spawn until I open a file.

bpasero · 2021-10-13T07:19:58Z

Yes, a files.watcherExclude of ** will essentially ignore the workspace folder and you should not see file handles being used as a result. The log message should confirm that if you look for "native excludes", because those are the paths we send to parcel for exclusion.

That said, it is possible that something else is consuming file handles that is not the file watcher maybe?

oof2win2 · 2021-10-13T07:30:11Z

Like a different app or service? Not sure what it would be. Without any extensions (except SSH), the file watchers stay. Disconnecting from the remote frees it of any watchers from the VSCode server executable.

bpasero · 2021-10-13T07:38:13Z

@oof2win2 can you validate this again without opening any files? the file watcher will start independent from individual files opening as long as you open the folder /opt/factorio/AwF-Bot-Test as workspace.

oof2win2 · 2021-10-13T07:40:06Z

With no files open, the system total of watchers is 70. The workspace is open.

bpasero · 2021-10-13T08:39:30Z

@oof2win2 ok that confirms to me that something else is starting to install watchers. Is it possible that the files you open start an extension (such as "TypeScript") and they install further file watchers?

To proof that point, you could try to disable all extensions (on the server as well as client).

oof2win2 · 2021-10-13T11:45:04Z

The @enabled tag shows only Remote - SSH and Remote - SSH: Editing Configuration Files. Since I am however opening a TS file, I created a new file - testing.malloc which shouldn't run any built-in extensions. It showed only the 70 file watchers across the system. JSON files are the same. JS files however start the extra file watchers. With no extensions, could it be the integrated JS/TS language support?

bpasero · 2021-10-13T11:54:23Z

Yes quite possibly the TS extension is setting up their own file watching. This is nothing I have control over.

mjbvz · 2021-10-13T20:47:23Z

Yes the typescript server has its own file watcher. Moving upstream to TS

bpasero · 2021-10-14T04:21:47Z

Would be good to understand why that is required.

andrewbranch · 2021-10-14T16:19:23Z

@oof2win2 could you collect a TS Server log and post it here?

oof2win2 · 2021-10-14T17:29:19Z

Here it is

andrewbranch · 2021-10-14T22:05:10Z

That log shows that we add 5 directory watchers and 4 file watchers. Granted, file watching is complicated and I know very little about it, but this sounds expected for us and doesn’t sound like we are adding 5k watchers.

bpasero · 2021-10-15T05:25:30Z

but this sounds expected for us

Why can you not rely on the file events VS Code sends for a workspace to your extension?

oof2win2 · 2021-10-15T05:52:55Z

Looked at the log just now and it seems that TS finds all typescript stuff scross the system and then watches it just if something happens? I agree with @bpasero that TS should listen to VSCode's ignores and not watch files if they are in the ignore path.

bpasero · 2022-03-30T08:45:00Z

Fyi microsoft/vscode#146066 makes the createFileSystemWatcher API more powerful: if you watch inside the workspace and your watcher is not recursive, the files.watcherExclude setting will not apply to you. This will help you watching inside node_modules folder because that folder is ignored by default via settings.

My understanding is that TypeScript does not require recursive watching because you only need to watch the @types folder for changes and that would work with a flat watcher 🚀

sheetalkamat · 2022-05-09T20:59:54Z

If anyone wants to do custom watching, we do allow users to exclude directories and folders from watches.. So users can opt to disable watching and refresh the state as they like but our watching tries to cater for various types of scale and uses and also os variants.

typescript-bot · 2022-05-12T20:00:01Z

This issue has been marked 'Working as Intended' and has seen no recent activity. It has been automatically closed for house-keeping purposes.

lramos15 assigned bpasero Oct 12, 2021

bpasero assigned mjbvz and unassigned bpasero Oct 13, 2021

bpasero changed the title ~~File Watcher consuming over 5k watchers~~ TypeScript file Watcher consuming over 5k watchers Oct 13, 2021

mjbvz transferred this issue from microsoft/vscode Oct 13, 2021

mjbvz removed their assignment Oct 13, 2021

andrewbranch added the Needs More Info The issue still hasn't been fully clarified label Oct 14, 2021

andrewbranch mentioned this issue Oct 20, 2021

tsserver does not seem to respect any of the exclude or watch options #46207

Closed

andrewbranch removed the Needs More Info The issue still hasn't been fully clarified label Oct 20, 2021

andrewbranch added the Needs Investigation This issue needs a team member to investigate its status. label Oct 20, 2021

andrewbranch added this to the TypeScript 4.6.0 milestone Oct 20, 2021

andrewbranch assigned sheetalkamat Oct 20, 2021

bpasero mentioned this issue Oct 27, 2021

File watcher resulting in high CPU use on Mac/Linux for large folders microsoft/vscode#3998

Closed

jrmyio mentioned this issue Oct 27, 2021

Type checking/VS Code slow when using MUI #34801

Closed

adicandra1 mentioned this issue Dec 3, 2021

Continual CPU usage from typingsInstaller process #36950

Closed

RyanCavanaugh modified the milestones: TypeScript 4.6.0, TypeScript 4.6.1 Feb 3, 2022

sheetalkamat added Working as Intended The behavior described is the intended behavior; this is not a bug and removed Needs Investigation This issue needs a team member to investigate its status. labels May 9, 2022

typescript-bot closed this as completed May 12, 2022

bpasero mentioned this issue Jun 10, 2022

Opening a .js file when node_modules is present causes explosion in number of watchers #49474

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeScript file Watcher consuming over 5k watchers #46351

TypeScript file Watcher consuming over 5k watchers #46351

oof2win2 commented Oct 12, 2021

bpasero commented Oct 13, 2021 •

edited

Loading

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

oof2win2 commented Oct 13, 2021 •

edited

Loading

bpasero commented Oct 13, 2021 •

edited

Loading

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

mjbvz commented Oct 13, 2021

bpasero commented Oct 14, 2021

andrewbranch commented Oct 14, 2021

oof2win2 commented Oct 14, 2021

andrewbranch commented Oct 14, 2021

bpasero commented Oct 15, 2021

oof2win2 commented Oct 15, 2021

bpasero commented Mar 30, 2022 •

edited

Loading

sheetalkamat commented May 9, 2022

typescript-bot commented May 12, 2022

TypeScript file Watcher consuming over 5k watchers #46351

TypeScript file Watcher consuming over 5k watchers #46351

Comments

oof2win2 commented Oct 12, 2021

bpasero commented Oct 13, 2021 • edited Loading

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

oof2win2 commented Oct 13, 2021 • edited Loading

bpasero commented Oct 13, 2021 • edited Loading

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

oof2win2 commented Oct 13, 2021

bpasero commented Oct 13, 2021

mjbvz commented Oct 13, 2021

bpasero commented Oct 14, 2021

andrewbranch commented Oct 14, 2021

oof2win2 commented Oct 14, 2021

andrewbranch commented Oct 14, 2021

bpasero commented Oct 15, 2021

oof2win2 commented Oct 15, 2021

bpasero commented Mar 30, 2022 • edited Loading

sheetalkamat commented May 9, 2022

typescript-bot commented May 12, 2022

bpasero commented Oct 13, 2021 •

edited

Loading

oof2win2 commented Oct 13, 2021 •

edited

Loading

bpasero commented Oct 13, 2021 •

edited

Loading

bpasero commented Mar 30, 2022 •

edited

Loading