Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File watches are created over-zealously #713

Open
hpdeifel opened this issue Mar 15, 2019 · 34 comments
Open

File watches are created over-zealously #713

hpdeifel opened this issue Mar 15, 2019 · 34 comments

Comments

@hpdeifel
Copy link
Contributor

It seems that lsp-mode now creates file-watches for all the things in the projects root directory, which uses up resources and isn't always needed. For example, rls registers delete/change/create watchers explicitly only for Cargo.lock, Cargo.toml plus a deletion watcher for the target/ directory. Instead of creating only three file watches, lsp-mode decides to recursively watch files and directories from e.g. the target/ directory, which leads to errors like:

directory-files-recursively: Opening directory: No such file or directory, /somewhere/target/rls/debug/deps/rmeta31dHOO

Please correct me if I'm wrong, I haven't yet understood the file-watcher code in its entirety.

@yyoncho
Copy link
Member

yyoncho commented Mar 15, 2019

I agree that it is not the optimal solution, but at this point, this is the simplest solution having in mind that the language servers that are running in the folder are not defined and it is not trivial to deduplicate the paths. e. g. jdt-ls and boot-ls combined run watch 10+ file patterns and most of them are like **/*.java src/main/**. Also, they could unregister/register a file watch at any point. I agree that we might think for some optimizations, e. g. if the paths do not have wildcards - watch only them.

There is a similar issue for vscode:
microsoft/vscode#45295

directory-files-recursively: Opening directory: No such file or directory, /somewhere/target/rls/debug/deps/rmeta31dHOO

This is a bug that has to be fixed. You may ignore the target folder via lsp-file-watch-ignored for now.

yyoncho added a commit that referenced this issue Mar 16, 2019
- we need to sortout #713 before we
  reenable it again
@hpdeifel
Copy link
Contributor Author

I agree that creating only the required watches is a hard problem.

This is a bug that has to be fixed. You may ignore the target folder via lsp-file-watch-ignored for now.

Does that also ignore deletion events for the folder itself? Because that would be one of the three things that rls explicitly watches.

@yyoncho
Copy link
Member

yyoncho commented Mar 16, 2019

Does that also ignore deletion events for the folder itself?

The deletion events comes from the parent folder so it is fine to ignore the folder itself. I will fix the deletion event today, so you may ignore it.

@yyoncho
Copy link
Member

yyoncho commented Mar 16, 2019

I have pushed 1f7e807 which fixes the directory-files-recursively issue and I have documented the "Limitations" in the README page. I am not planning to address the issue related to optimizing the watches since it will become irrelevant if, for example, RLS starts to monitor **.rs files.

@yyoncho yyoncho added minor and removed bug labels Mar 16, 2019
@uqix
Copy link

uqix commented Sep 7, 2019

From README:

In case your project contains a lot of files you might want to disable file monitoring via lsp-enable-file-watchers (you may use dir-locals).

Is dir-locals for project root or any dir in the project?

@yyoncho
Copy link
Member

yyoncho commented Sep 7, 2019

@uqix the dir local solution will be per project. But you may use lsp-file-watch-ignored to skip a particular folder from being monitored.

@axelson
Copy link
Contributor

axelson commented Jan 14, 2021

Is there a chance for this to be re-visited? Or is there any way to work around this limitation from a language server implementation side?

@yyoncho
Copy link
Member

yyoncho commented Jan 14, 2021

Is there a chance for this to be re-visited? Or is there any way to work around this limitation from a language server implementation side?

No.

I am looking at elixir-lsp/elixir-ls#448 and given the fact that you are interested in **/*.<ext> there is even no room for optimization in elisp since we should watch everything...

In general, 1000 threshold limit is very aggressive. I think that emacs will perform fine after the initial watch establishment even with 10k files to watch.

@jturner314
Copy link

As an idea, it may be helpful to add a hook so that users can more easily customize which files to ignore (with more sophisticated logic than just regexps). Personally, I'd like to ignore everything according to .gitignore (via git-check-ignore) except for Cargo.lock.

@aldanor
Copy link

aldanor commented Jan 26, 2021

@jturner314 That would be the cleanest default, indeed - anything that's already in your .gitignore has to no reason to be watched.

@nbfalcon
Copy link
Member

nbfalcon commented Jan 26, 2021

@aldanor how about source code that is generated as part of the build process and can as such change any time? However, this might be more of an edge-case, and so if this were customizable, that should be good enough.

@aldanor
Copy link

aldanor commented Jan 26, 2021

@nbfalcon As you've said it's more of an edge case - it would be nice if you could prevent ignoring (or rather force-watching folders or files), bonus points if it can also be done on a per-project basis.

As for .gitignore, here's another simple example - the target/ folder will be present in most rust projects (and it doesn't have to be at project root, technically) and it will almost certainly be in .gitignore but in other projects there may be a valid target/ folder that has to be watched.

Cargo.lock may be another exception that will often be in .gitignore but would have to be watched.

@clayrisser
Copy link

Is this related to the fact that emacs keeps hanging on me with he message ". . . so watching the Repo may slow emacs down. Do you want to watch all files in . . ."?

@yyoncho
Copy link
Member

yyoncho commented Jan 31, 2021

Is this related to the fact that emacs keeps hanging on me with he message ". . . so watching the Repo may slow emacs down. Do you want to watch all files in . . ."?

@clayrisser here you are supposed to press y or n

@borkdude
Copy link

borkdude commented Feb 4, 2021

Personally, I'd like to ignore everything according to .gitignore

+1

@ericdallo
Copy link
Member

@yyoncho, what about a defcustom to make lsp-file-watch-ignored follow .gitignore?

@borkdude
Copy link

borkdude commented Feb 4, 2021

The reverse would also work for me: watch only files that are part of the git index (possibly using git itself for detecting the relevant directories).

@ordnungswidrig
Copy link

The reverse would also work for me: watch only files that are part of the git index (possibly using git itself for detecting the relevant directories).

What about all the freshly created file you did not commit yet?

@aldanor
Copy link

aldanor commented Feb 17, 2021

What about all the freshly created file you did not commit yet?

Yep, indeed. I'd rather just have it follow .gitignore by default (and maybe have some option of un-ignoring certain folders that may have code generated by the build process; although that would be a rare case; .gitignore path should work out of the box most of the time for most projects).

@Dieken
Copy link

Dieken commented Feb 20, 2021

Is this related to the fact that emacs keeps hanging on me with he message ". . . so watching the Repo may slow emacs down. Do you want to watch all files in . . ."?

@clayrisser No idea why lsp-mode picks some parent directory as workspace root directory, that will make lsp-mode traverse a lot of files.

https://github.com/emacs-lsp/lsp-mode/blob/2d1732a3b9bc27c1eeb4e69805bb6a8527ae7893/lsp-mode.el

Functions lsp--directory-files-recursively and lsp-watch-root-folder should exit early when number of files is greater than lsp-file-watch-threshold. At least lsp--directory-files-recursively really should exit early.

@yyoncho

@yyoncho
Copy link
Member

yyoncho commented Feb 20, 2021

Functions lsp--directory-files-recursively and lsp-watch-root-folder should exit early when number of files is greater than lsp-file-watch-threshold. At least lsp--directory-files-recursively really should exit early.

We could do that if we change the wording in the warning dialog.

As a side note, I have a prototype utilizing emacs threads which avoids the hanging.

@bhipple
Copy link

bhipple commented Mar 14, 2021

Every time I open a python file in any git repo, emacs hangs for ~20 seconds while I mash C-g, then I try to open the file a second time and am greeted with a y/n prompt that adding file watches to everything in my home dir will be slow. I've fixed it on my end by adding this:

(with-eval-after-load 'lsp-mode (add-to-list 'lsp-file-watch-ignored-directories (getenv "HOME")))

I'm wondering if watching $HOME is the intended behavior, or if it's a bug in either lsp-mode or spacemacs, or if it's something weird that only I'm seeing? My git clones don't have symlinks back to $HOME so I'm not sure how it "escaped".

@yyoncho
Copy link
Member

yyoncho commented Mar 14, 2021

I'm wondering if watching $HOME is the intended behavior, or if it's a bug in either lsp-mode or spacemacs,

most likely you have incidentally added the home folder as project root. Use M-x lsp-describe-session to check what are your project roots.

@Dieken
Copy link

Dieken commented Mar 14, 2021

yes, projectile or some other elisp plugins may set wrong project root.

I have added (setq lsp-enable-file-watchers nil) to my emacs config, haven't noticed any severe impact on user experience.

@Anton-Latukha
Copy link

Anton-Latukha commented Mar 17, 2021

Spitballing:

It is not widely known, Linux kernel & filesystems for many years ship the inotify watcher system with a directory mode, there is no need to create trigger on file basis - filesystems can be asked to notify when there would be some changes inside the directory:
old system was dnotify, then now it is inotify, LWN docs on it: Part 1, Part 2, ArchWiki section, Gentoo Wiki had me to mention inotify is Emacs USE flag since 24.4, dk if it is default one (probably is or should be on Linux), and lsp-mode most probably indirectly leverages it.

@Nathan-Furnal
Copy link

I don't understand what's in lsp-file-watched-ignored-directories ? I want to ignore my home folder my/home/ because anytime I start a random file without a project, lsp freezes Emacs and asks if I want to watch my entire home directory. It's never the case and I don't know how to stop it? @yyoncho

@yyoncho
Copy link
Member

yyoncho commented Mar 22, 2021

@Nathan-Furnal check whether you have accidentally added your home dir as a root folder. Do M-x lsp-describe-session and then use M-x lsp-workspace-folders-remove.

FWIW I will change the code that counts the file in workspace folders to stop when it hits the file limit.

@Nathan-Furnal
Copy link

Thanks that worked for me! Is there any way i can tell lsp to use the current folder when there's no project? I don't write files in my home folder and that'd be a useful alternative when i just want to write a short file.

@yyoncho
Copy link
Member

yyoncho commented Mar 22, 2021

@Nathan-Furnal press I when you are asked to select project root.

@Nathan-Furnal
Copy link

Nathan-Furnal commented Mar 22, 2021

Fair enough, I didn't know it was that obvious. Thanks for taking the time!

@mjkramer
Copy link

mjkramer commented Dec 3, 2021

For now, here's some advice for ignoring anything in .gitignore. At some point I'll try to submit a PR with per-workspace variables for toggling this behavior and manually un-ignoring specific directories.

(defun ++git-ignore-p (path)
  (let* (; trailing / breaks git check-ignore if path is a symlink:
         (path (directory-file-name path))
         (default-directory (file-name-directory path))
         (relpath (file-name-nondirectory path))
         (cmd (format "git check-ignore '%s'" relpath))
         (status (call-process-shell-command cmd)))
    (eq status 0)))

(defun ++lsp--path-is-watchable-directory-a
    (fn path dir ignored-directories)
  (and (not (++git-ignore-p (f-join dir path)))
       (funcall fn path dir ignored-directories)))

(advice-add 'lsp--path-is-watchable-directory
            :around #'++lsp--path-is-watchable-directory-a)

r0bobo pushed a commit to r0bobo/dotfiles that referenced this issue Jan 16, 2022
@dschrempf
Copy link

Has there been any work on this? Whenever I run computations which create a lot of output within a project, Emacs becomes unusable. The profiler report shows that lsp--folder-watch-callback is to blame :).

@yyoncho
Copy link
Member

yyoncho commented May 11, 2022

@dschrempf for now the plan is to move the establishment of the watches in a separate thread and use thread-yield. Thus we will have a responsive UI with the same number of file watches.

jdelStrother added a commit to jdelStrother/dotfiles that referenced this issue May 31, 2022
jdelStrother added a commit to jdelStrother/dotfiles that referenced this issue May 31, 2022
@jdelStrother
Copy link

For now, here's some advice for ignoring anything in .gitignore. At some point I'll try to submit a PR with per-workspace variables for toggling this behavior and manually un-ignoring specific directories.

(defun ++git-ignore-p (path)
  (let* (; trailing / breaks git check-ignore if path is a symlink:
         (path (directory-file-name path))
         (default-directory (file-name-directory path))
         (relpath (file-name-nondirectory path))
         (cmd (format "git check-ignore '%s'" relpath))
         (status (call-process-shell-command cmd)))
    (eq status 0)))

(defun ++lsp--path-is-watchable-directory-a
    (fn path dir ignored-directories)
  (and (not (++git-ignore-p (f-join dir path)))
       (funcall fn path dir ignored-directories)))

(advice-add 'lsp--path-is-watchable-directory
            :around #'++lsp--path-is-watchable-directory-a)

Nice idea, though it does make lsp take forever to start in my project, as it seems to run git check-ignore hundreds/thousands of times.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests