Severe performance issue -- file deletion

[ASerbinski]

I'm looking mostly for advice on how to track a performance issue I'm having with group folders;

Basics;
Nextcloud 20.0.8
Groupfolders 8.2.0
apache httpd 2.4.37 with h2 enabled
php 7.4.6
mariadb 10.3.17
centos 8
8-core x86, 32GB, 2TB SSD.

I have about 20 users connected to a group folder using webdav, the load is actually very light as the workflow is such that the users will open one pdf file at a time in a local annotator, spend a few minutes working in it, and save.

The problem is that some file operations are exceedingly slow. In particular, MOVE and DELETE take around 1.5 seconds for any file -- which adds up considerably if the operation has to be done on a larger number of files (i.e. mass delete or move). I've also checked via nextcloud web ui, and those operations take the same time there, which means it isn't a webdav issue.

Outside of the group folder, these operations take around 75 ms.
I've also created a second group folder and duplicated all of the data into it (12 GB, about 14k files), and in the second group folder, the operations take about 75 ms.
I've also tried disabling advanced permissions -- no change.
LOAD is averaging around 0.4 (of 8)

Where do I look?

[pierreozoux]

Looks like a duplicate of #1361

[ASerbinski]

Please reopen, this is not a duplicate of that issue.
This has nothing to do with maintenance mode, there is nothing logged. Its just slow to delete files.

[ASerbinski]

I found the problem. Please reopen, there is a valid bug that has nothing to do with the other thread.
The issue has to do with processing the trashcan during the file delete or move process. Once the trashcan was cleared sudo -u apache ./occ groupfolders:trashbin:cleanup -- performance is excellent.

I identified the problem by enabling the mysql general log and found big time steps between a few queries;

| 2021-04-08 12:08:03.820858 | owncloud[owncloud] @ localhost []   |    135738 |         1 | Query        | SELECT `filecache`.`fileid`, `storage`, `path`, `path_hash`, `filecache`.`parent`, `name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `permissions`, `checksum`, `metadata_etag`, `creation_time`, `upload_time` FROM `oc_filecache` `filecache` LEFT JOIN `oc_filecache_extended` `fe` ON `filecache`.`fileid` = `fe`.`fileid` WHERE `filecache`.`parent` = 734438 ORDER BY `name` ASC                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| 2021-04-08 12:08:04.809241 | owncloud[owncloud] @ localhost []   |    135738 |         1 | Query        | START TRANSACTION                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 2021-04-08 12:08:05.310703 | owncloud[owncloud] @ localhost []   |    135738 |         1 | Query        | COMMIT                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 2021-04-08 12:08:05.321364 | owncloud[owncloud] @ localhost []   |    135738 |         1 | Query        | UPDATE `oc_filecache` SET `size` = '7072693595' WHERE (`fileid` = 734438) AND ((`size` <> '7072693595') OR (`size` IS NULL))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

The queries themselves weren't particularly slow (about 70 ms), but it dumped out a list of 11k files under __groupfolders/trash/1/ -- which were clearly taking a while to be processed by php.

It obviously shouldn't be processing the trash directory like this.

[testbird]

@ASerbinski I would probably tend to try working around such frontend problems with something like filestash (most easy if already using nextcloud external storage).

[testbird]

Looking for "How to access your filesystem" with it that is.

[ASerbinski]

I worked around the issue by running the cleanup in my nightly cronjob. Solutions that use alternative products really don't apply since they don't fix the actual bug.

[testbird]

Yep, I understand. It would also involve using a brush-up shell-script for the classic unix group directories and user private group scheme. Which I didn't think of anymore, as they just work so silently, once these are set up.

[fschrempf]

The automatic expiration of the trashbin content is tracked in #930 and has been recently implemented in #1759. I guess this would prevent the issues described above. Would you agree, that we can close this?

[fschrempf]

Closing this as the main cause behind the issue is solved in recent versions. Please comment, reopen or create a new issue if you think this problem still exists. Thanks!