Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(local): thumbnails oom #7124

Merged
merged 8 commits into from
Sep 3, 2024
Merged

fix(local): thumbnails oom #7124

merged 8 commits into from
Sep 3, 2024

Conversation

Mmx233
Copy link
Contributor

@Mmx233 Mmx233 commented Sep 2, 2024 •
edited
Loading

Moved from #7120 for cleaning codes.

Allow limit thumbnail generating concurrent in local storage driver settings to avoid using too much memory.

Tested with image: mmx233/alist:v3.36.0-delta1 mmx233/alist:v3.36.0-delta1-ffmpeg

close #7082

@Mmx233
Copy link
Contributor Author

Mmx233 commented Sep 2, 2024 •
edited
Loading

Summary of Changes

  • Add static token bucket algorithm based on Go channels in local storage driver. cb9af35
  • Use token bucket to limit concurrent thumbnail generation, default concurrent is 16. 5bc1ed1 05f1c20
  • If concurrent is 0 or empty string, then there is no concurrent limit for thumbnail generation. In testing, after upgrading to the new version, this value is an empty string rather than the default value 16.
  • If thumbnail generation setting value changed, the theoretical maximum peak concurrency in a short period is the sum of old concurrency and new concurrency.

@xhofe xhofe merged commit 4874c9e into AlistGo:main Sep 3, 2024
3 checks passed
@Mmx233 Mmx233 deleted the fix/thumbnails-oom branch September 7, 2024 05:04
Three-taile-dragon pushed a commit to loognsss/blist that referenced this pull request Sep 26, 2024
@Mmx233 @Muione @Three-taile-dragon
fix(local): thumbnails oom (AlistGo#7124 close AlistGo#7082)
15f26fe 
* add my_build.sh

* Fix OOM of thumbnail generation of LoaclDrive by using a task queue to control thread count

* remove my_build.sh

* chore(local): allow ThumbConcurrency set to zero

* revert(local): changes to thumbnail generating functions

* feat(local): implement static token bucket

* feat(local): use static token bucket to limit thumbnails generating concurrent

---------

Co-authored-by: KKJas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xhofe xhofe xhofe approved these changes

@Muione Muione Muione approved these changes

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

生成缩略图疑似内存爆炸
3 participants
@Mmx233 @xhofe @Muione