Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

生成缩略图疑似内存爆炸 #7082

Closed
4 tasks done
asforest opened this issue Aug 26, 2024 · 3 comments · Fixed by #7124
Closed
4 tasks done

生成缩略图疑似内存爆炸 #7082

asforest opened this issue Aug 26, 2024 · 3 comments · Fixed by #7124
Labels
bug Something isn't working

Comments

@asforest
Copy link

Please make sure of the following things

  • I have read the documentation.
    我已经阅读了文档

  • I'm sure there are no duplicate issues or discussions.
    我确定没有重复的issue或讨论。

  • I'm sure it's due to AList and not something else(such as Network ,Dependencies or Operational).
    我确定是AList的问题,而不是其他原因(例如网络依赖操作)。

  • I'm sure this issue is not fixed in the latest version.
    我确定这个问题在最新版本中没有被修复。

AList Version / AList 版本

docker:xhofe/alist:v3.36.0-ffmpeg

Driver used / 使用的存储驱动

本地存储

Describe the bug / 问题描述

复现过程:
1.使用Alist添加一个本地存储,开启缩略图选项,其它保持默认
2.往这个存储里复制一些图片,每个图片约1m上下,总计200m左右的大小
3.访问Alist页面,打开对应的文件夹
4.将页面视图切换到“网格视图”,以开始生成缩略图
5.观察htop发现alist进程的内存从60m一瞬间上升到1.3G(我的系统总内存1.5G),然后整个系统卡死,只能断电重启
6.尝试使用docker compose参数mem_limit: '600M'将内存限制为600m
7.每当Alist触达这个内存限制都,容器都会重启。且docker日志中似乎看不到报错的字样

本地存储配置选项截图:
image

docker日志

image

Reproduction / 复现链接

这是我在本地本地部署的一个Alist实例,很抱歉实在没法提供公开访问的链接

Config / 配置

{
  "force": false,
  "site_url": "",
  "cdn": "",
  "jwt_secret": "xxxxxxxxxxxx",
  "token_expires_in": 24,
  "database": {
    "type": "sqlite3",
    "host": "",
    "port": 0,
    "user": "",
    "password": "",
    "name": "",
    "db_file": "data/data.db",
    "table_prefix": "x_",
    "ssl_mode": "",
    "dsn": ""
  },
  "meilisearch": {
    "host": "http://localhost:7700",
    "api_key": "",
    "index_prefix": ""
  },
  "scheme": {
    "address": "0.0.0.0",
    "http_port": 5244,
    "https_port": 5243,
    "force_https": false,
    "cert_file": "data/k.pem",
    "key_file": "data/v.key",
    "unix_file": "",
    "unix_file_perm": ""
  },
  "temp_dir": "data/temp",
  "bleve_dir": "data/bleve",
  "dist_dir": "",
  "log": {
    "enable": true,
    "name": "data/log/log.log",
    "max_size": 10,
    "max_backups": 5,
    "max_age": 28,
    "compress": false
  },
  "delayed_start": 0,
  "max_connections": 10,
  "tls_insecure_skip_verify": true,
  "tasks": {
    "download": {
      "workers": 5,
      "max_retry": 1
    },
    "transfer": {
      "workers": 5,
      "max_retry": 2
    },
    "upload": {
      "workers": 5,
      "max_retry": 0
    },
    "copy": {
      "workers": 5,
      "max_retry": 2
    }
  },
  "cors": {
    "allow_origins": [
      "*"
    ],
    "allow_methods": [
      "*"
    ],
    "allow_headers": [
      "*"
    ]
  },
  "s3": {
    "enable": false,
    "port": 5246,
    "ssl": false
  }
}

Logs / 日志

No response

@asforest asforest added the bug Something isn't working label Aug 26, 2024
@asforest
Copy link
Author

补充:即使对一个体积较小的目录成功生成了所有缩略图,其占用的内存也会居高不下,一直无法释放。疑似有内存泄漏

@Muione
Copy link
Contributor

Muione commented Aug 29, 2024

补充:即使对一个体积较小的目录成功生成了所有缩略图,其占用的内存也会居高不下,一直无法释放。疑似有内存泄漏

这个问题不是内存泄漏导致的,生成图片过一段时间后内存占用会恢复正常,但是确实没有及时释放。
至于爆内存这个应该是一次性开启了太多线程导致的,我在使用的时候也遇到过这样的问题,但是我的服务器内存24G不至于卡死。
这个问题我也在尝试修复,等待后续的PR吧。

@asforest
Copy link
Author

补充:即使对一个体积较小的目录成功生成了所有缩略图,其占用的内存也会居高不下,一直无法释放。疑似有内存泄漏

这个问题不是内存泄漏导致的,生成图片过一段时间后内存占用会恢复正常,但是确实没有及时释放。 至于爆内存这个应该是一次性开启了太多线程导致的,我在使用的时候也遇到过这样的问题,但是我的服务器内存24G不至于卡死。 这个问题我也在尝试修复,等待后续的PR吧。

谢谢大佬!

xhofe pushed a commit that referenced this issue Sep 3, 2024
* add my_build.sh

* Fix OOM of thumbnail generation of LoaclDrive by using a task queue to control thread count

* remove my_build.sh

* chore(local): allow ThumbConcurrency set to zero

* revert(local): changes to thumbnail generating functions

* feat(local): implement static token bucket

* feat(local): use static token bucket to limit thumbnails generating concurrent

---------

Co-authored-by: KKJas <[email protected]>
Three-taile-dragon pushed a commit to loognsss/blist that referenced this issue Sep 26, 2024
* add my_build.sh

* Fix OOM of thumbnail generation of LoaclDrive by using a task queue to control thread count

* remove my_build.sh

* chore(local): allow ThumbConcurrency set to zero

* revert(local): changes to thumbnail generating functions

* feat(local): implement static token bucket

* feat(local): use static token bucket to limit thumbnails generating concurrent

---------

Co-authored-by: KKJas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants