paddleOCR 识别，内存一直上涨，只要进程不杀死，内存就不会下降 #14331

Yins11 · 2024-11-12T05:51:25Z

Yins11
Nov 12, 2024

🔎 Search before asking

I have searched the PaddleOCR Docs and found no similar bug report.
I have searched the PaddleOCR Issues and found no similar bug report.
I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

python tools/infer/predict_rec.py
--rec_model_dir=./inference/rec_ppocr_v4/
--image_dir="xxxxxxxxxxxxxx"
--rec_algorithm="SVTR_LCNet"
--rec_image_shape="3, 48, 320"
--rec_batch_num=1
--rec_char_dict_path="./ppocr/utils/en_dict.txt"
--benchmark=True
--use_gpu=True
--use_tensorrt=True
--warmup=True
运行命令如上，服务器运行内存一直涨，直到挂掉,
paddlecor: v2.8.1
tensorrt: 8.6.1.6
python: 3.10
paddle: v2.6.1
这个问题只出现在使用tensorrt推理时。请问到底是什么原因

🏃‍♂️ Environment (运行环境)

linux

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

使用paddleocr v2.8.1代码训练-导出-推理，使用tensorrt推理出现上述问题，推理参数设置：
def init_args():
parser = argparse.ArgumentParser()
# params for prediction engine
parser.add_argument("--use_gpu", type=str2bool, default=True)
parser.add_argument("--use_xpu", type=str2bool, default=False)
parser.add_argument("--use_npu", type=str2bool, default=False)
parser.add_argument("--use_mlu", type=str2bool, default=False)
parser.add_argument("--ir_optim", type=str2bool, default=True)
parser.add_argument("--use_tensorrt", type=str2bool, default=True)
parser.add_argument("--min_subgraph_size", type=int, default=15)
parser.add_argument("--precision", type=str, default="fp32")
parser.add_argument("--gpu_mem", type=int, default=500)
parser.add_argument("--gpu_id", type=int, default=0)
parser.add_argument("--use_openvino", type=str2bool, default=False)

# params for text detector
parser.add_argument("--image_dir", type=str)
parser.add_argument("--page_num", type=int, default=0)
parser.add_argument("--det_algorithm", type=str, default="DB")
parser.add_argument("--det_model_dir", type=str)
parser.add_argument("--det_limit_side_len", type=float, default=960)
parser.add_argument("--det_limit_type", type=str, default="max")
parser.add_argument("--det_box_type", type=str, default="quad")

# DB parmas
parser.add_argument("--det_db_thresh", type=float, default=0.3)
parser.add_argument("--det_db_box_thresh", type=float, default=0.6)
parser.add_argument("--det_db_unclip_ratio", type=float, default=1.5)
parser.add_argument("--max_batch_size", type=int, default=10)
parser.add_argument("--use_dilation", type=str2bool, default=False)
parser.add_argument("--det_db_score_mode", type=str, default="fast")

# EAST parmas
parser.add_argument("--det_east_score_thresh", type=float, default=0.8)
parser.add_argument("--det_east_cover_thresh", type=float, default=0.1)
parser.add_argument("--det_east_nms_thresh", type=float, default=0.2)

# SAST parmas
parser.add_argument("--det_sast_score_thresh", type=float, default=0.5)
parser.add_argument("--det_sast_nms_thresh", type=float, default=0.2)

# PSE parmas
parser.add_argument("--det_pse_thresh", type=float, default=0)
parser.add_argument("--det_pse_box_thresh", type=float, default=0.85)
parser.add_argument("--det_pse_min_area", type=float, default=16)
parser.add_argument("--det_pse_scale", type=int, default=1)

# FCE parmas
parser.add_argument("--scales", type=list, default=[8, 16, 32])
parser.add_argument("--alpha", type=float, default=1.0)
parser.add_argument("--beta", type=float, default=1.0)
parser.add_argument("--fourier_degree", type=int, default=5)

# params for text recognizer
parser.add_argument("--rec_algorithm", type=str, default="SVTR_LCNet")#文本识别算法名称，目前支持CRNN, SRN, RARE, NETR, SAR, ViTSTR, ABINet, VisionLAN, SPIN, RobustScanner, SVTR, SVTR_LCNet
parser.add_argument("--rec_model_dir", type=str,default ="PaddleOCR/inference/rec_ppocr_v3/")#识别inference模型路径
parser.add_argument("--rec_image_inverse", type=str2bool, default=True)
parser.add_argument("--rec_image_shape", type=str, default="3, 48, 320")#识别时的图像尺寸
parser.add_argument("--rec_batch_num", type=int, default=6) #识别的batch size
parser.add_argument("--max_text_length", type=int, default=25)#识别结果最大长度，在SRN中有效
parser.add_argument(
    "--rec_char_dict_path", type=str, default="./ppocr/utils/ppocr_keys_v1.txt"
)#识别的字符字典文件
parser.add_argument("--use_space_char", type=str2bool, default=True)#是否包含空格，如果为True，则会在最后字符字典中补充空格字符
parser.add_argument("--vis_font_path", type=str, default="./doc/fonts/simfang.ttf")
parser.add_argument("--drop_score", type=float, default=0.5)

# params for e2e
parser.add_argument("--e2e_algorithm", type=str, default="PGNet")#端到端算法名称，目前支持PGNet
parser.add_argument("--e2e_model_dir", type=str)
parser.add_argument("--e2e_limit_side_len", type=float, default=768)
parser.add_argument("--e2e_limit_type", type=str, default="max")

# PGNet parmas
parser.add_argument("--e2e_pgnet_score_thresh", type=float, default=0.5)
parser.add_argument(
    "--e2e_char_dict_path", type=str, default="./ppocr/utils/ic15_dict.txt"
)
parser.add_argument("--e2e_pgnet_valid_set", type=str, default="totaltext")
parser.add_argument("--e2e_pgnet_mode", type=str, default="fast")

# params for text classifier
parser.add_argument("--use_angle_cls", type=str2bool, default=False)
parser.add_argument("--cls_model_dir", type=str)
parser.add_argument("--cls_image_shape", type=str, default="3, 48, 192")
parser.add_argument("--label_list", type=list, default=["0", "180"])
parser.add_argument("--cls_batch_num", type=int, default=6)
parser.add_argument("--cls_thresh", type=float, default=0.9)

parser.add_argument("--enable_mkldnn", type=str2bool, default=False)
parser.add_argument("--cpu_threads", type=int, default=10)
parser.add_argument("--use_pdserving", type=str2bool, default=False)
parser.add_argument("--warmup", type=str2bool, default=False)

# SR parmas
parser.add_argument("--sr_model_dir", type=str)
parser.add_argument("--sr_image_shape", type=str, default="3, 32, 128")
parser.add_argument("--sr_batch_num", type=int, default=1)

#
parser.add_argument("--draw_img_save_dir", type=str, default="./inference_results")
parser.add_argument("--save_crop_res", type=str2bool, default=False)
parser.add_argument("--crop_res_save_dir", type=str, default="./output")

# multi-process
parser.add_argument("--use_mp", type=str2bool, default=False)
parser.add_argument("--total_process_num", type=int, default=1)
parser.add_argument("--process_id", type=int, default=0)

parser.add_argument("--benchmark", type=str2bool, default=False)
parser.add_argument("--save_log_path", type=str, default="./log_output/")

parser.add_argument("--show_log", type=str2bool, default=True)
parser.add_argument("--use_onnx", type=str2bool, default=False)

liuhongen1234567 · 2024-11-12T07:20:19Z

liuhongen1234567
Nov 12, 2024

您好，请问在paddle3.0b1上也会内存上涨吗？您那边可以试试，paddle2.6我这边也同步看一下

0 replies

Yins11 · 2024-11-12T07:26:46Z

Yins11
Nov 12, 2024
Author

您好，请问在paddle3.0b1上也会内存上涨吗？您那边可以试试，paddle2.6我这边也同步看一下

未测试paddle3.0b1，不过我刚把paddle版本降级到2.4.2，就不上涨了，正常执行，不知道是不是paddle版本问题，或者是和tensorrt版本不匹配？？工作需要把paddle版本升级到2.6.1，ocr版本升级为2.8.1，所以还是要定位原因。麻烦了

0 replies

KyleWang-Hunter · 2024-11-13T01:26:30Z

KyleWang-Hunter
Nov 13, 2024

到底哪些版本会存在内存泄漏问题，官方能不能具体给个说明？

0 replies

liuhongen1234567 · 2024-11-13T09:43:42Z

liuhongen1234567
Nov 13, 2024

您好，您那边可以提供一下导出的配置的文件和官方权重的位置吗？

0 replies

Yins11 · 2024-11-14T01:57:32Z

Yins11
Nov 14, 2024
Author

您好，您那边可以提供一下导出的配置的文件和官方权重的位置吗？

model : https://github.com/PaddlePaddle/PaddleOCR/blob/v2.8.1/doc/doc_ch/models_list.md
v3 config:
`Global:
debug: false
epoch_num: 100
log_smooth_window: 20
print_batch_step: 10
save_model_dir: ./output/v3_en_mobile
save_epoch_step: 3
eval_batch_step:

0
100
cal_metric_during_train: true
pretrained_model: xxxxxxxxxxxxx/pretrain_model/en_PP-OCRv3_rec_train/best_accuracy.pdparams
checkpoints: null
save_inference_dir: null
use_visualdl: false
infer_img: doc/imgs_words/ch/word_1.jpg
character_dict_path: ppocr/utils/en_dict.txt
max_text_length: 25
infer_mode: false
use_space_char: true
distributed: true
save_res_path: ./output/rec/predicts_ppocrv3_en.txt
Optimizer:
name: Adam
beta1: 0.9
beta2: 0.999
lr:
name: Cosine
learning_rate: 0.001
warmup_epoch: 5
regularizer:
name: L2
factor: 3.0e-05
Architecture:
model_type: rec
algorithm: SVTR_LCNet
Transform: null
Backbone:
name: MobileNetV1Enhance
scale: 0.5
last_conv_stride:
- 1
- 2
  last_pool_type: avg
  last_pool_kernel_size:
- 2
- 2
  Head:
  name: MultiHead
  head_list:
- CTCHead:
  Neck:
  name: svtr
  dims: 64
  depth: 2
  hidden_dims: 120
  use_guide: true
  Head:
  fc_decay: 1.0e-05
- SARHead:
  enc_dim: 512
  max_text_length: 25
  Loss:
  name: MultiLoss
  loss_config_list:
CTCLoss: null
SARLoss: null
PostProcess:
name: CTCLabelDecode
Metric:
name: RecMetric
main_indicator: acc
ignore_space: false
Train:
dataset:
name: SimpleDataSet
data_dir: xxxxxxxxxxxxxxxxxxxxxxxxxx
ext_op_transform_idx: 1
label_file_list:
- xxxxxxxxxxxxxxxxxxxxxxx/train_list.txt
  transforms:
- DecodeImage:
  img_mode: BGR
  channel_first: false
- RecConAug:
  prob: 0.5
  ext_data_num: 2
  image_shape:
  - 48
  - 320
  - 3
    max_text_length: 25
- RecAug: null
- MultiLabelEncode: null
- RecResizeImg:
  image_shape:
  - 3
  - 48
  - 320
- KeepKeys:
  keep_keys:
  - image
  - label_ctc
  - label_sar
  - length
  - valid_ratio
    loader:
    shuffle: true
    batch_size_per_card: 128
    drop_last: true
    num_workers: 4
    Eval:
    dataset:
    name: SimpleDataSet
    data_dir: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
    label_file_list:
- xxxxxxxxxxxxxxxxxxxxxxx/val_list.txt
  transforms:
- DecodeImage:
  img_mode: BGR
  channel_first: false
- MultiLabelEncode: null
- RecResizeImg:
  image_shape:
  - 3
  - 48
  - 320
- KeepKeys:
  keep_keys:
  - image
  - label_ctc
  - label_sar
  - length
  - valid_ratio
    loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 128
    num_workers: 4
    profiler_options: null
    v4 config:Global:
    debug: false
    use_gpu: true
    epoch_num: 2
    amp_level: O1
    log_smooth_window: 20
    print_batch_step: 10
    save_model_dir: xxxxxxxxxxxxxxxxxxxxxxxx
    save_epoch_step: 10
    eval_batch_step:
0
80
cal_metric_during_train: true
pretrained_model: xxxxxxxxxxxxxxxxxxx/pretrain_model/en_PP-OCRv4_rec_train/best_accuracy.pdparams
checkpoints: null
save_inference_dir: null
use_visualdl: true
use_wandb: true
infer_img: doc/imgs_words/ch/word_1.jpg
character_dict_path: PaddleOCR/ppocr/utils/en_dict.txt
max_text_length: 25
infer_mode: false
use_space_char: true
distributed: true
save_res_path: PaddleOCR/output/rec/predicts_ppocrv3.txt
Optimizer:
name: Adam
beta1: 0.9
beta2: 0.999
lr:
name: Cosine
learning_rate: 0.0005
warmup_epoch: 5
regularizer:
name: L2
factor: 3.0e-05
Architecture:
model_type: rec
algorithm: SVTR_LCNet
Transform: null
Backbone:
name: PPLCNetV3
scale: 0.95
Head:
name: MultiHead
head_list:
- CTCHead:
  Neck:
  name: svtr
  dims: 120
  depth: 2
  hidden_dims: 120
  kernel_size:
  - 1
  - 3
  use_guide: true
  Head:
  fc_decay: 1.0e-05
- NRTRHead:
  nrtr_dim: 384
  max_text_length: 25
  Loss:
  name: MultiLoss
  loss_config_list:
CTCLoss: null
NRTRLoss: null
PostProcess:
name: CTCLabelDecode
Metric:
name: RecMetric
main_indicator: acc
ignore_space: false
Train:
dataset:
name: MultiScaleDataSet
ds_width: false
data_dir: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
ext_op_transform_idx: 1
label_file_list:
- xxxxxxxxxxxxxxxxxxxxxxxxxxxx/train_list.txt
  transforms:
- DecodeImage:
  img_mode: BGR
  channel_first: false
- RecConAug:
  prob: 0.5
  ext_data_num: 2
  image_shape:
  - 48
  - 320
  - 3
    max_text_length: 25
- RecAug: null
- MultiLabelEncode:
  gtc_encode: NRTRLabelEncode
- KeepKeys:
  keep_keys:
  - image
  - label_ctc
  - label_gtc
  - length
  - valid_ratio
    sampler:
    name: MultiScaleSampler
    scales:
- - 320
  - 32
- - 320
  - 48
- - 320
  - 64
    first_bs: 96
    fix_bs: false
    divided_factor:
- 8
- 16
  is_training: true
  loader:
  shuffle: true
  batch_size_per_card: 96
  drop_last: true
  num_workers: 8
  Eval:
  dataset:
  name: SimpleDataSet
  data_dir: xxxxxxxxxxxxxxxxxxxxxx
  label_file_list:
- xxxxxxxxxxxxxxxxxxxxx/val_list.txt
  transforms:
- DecodeImage:
  img_mode: BGR
  channel_first: false
- MultiLabelEncode:
  gtc_encode: NRTRLabelEncode
- RecResizeImg:
  image_shape:
  - 3
  - 48
  - 320
- KeepKeys:
  keep_keys:
  - image
  - label_ctc
  - label_gtc
  - length
  - valid_ratio
    loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 128
    num_workers: 4
    profiler_options: null
    `

0 replies

liuhongen1234567 · 2024-11-14T08:20:39Z

liuhongen1234567
Nov 14, 2024

您好，请问您那边测试的图片数量是多少？有测试过官方模型是否会内存泄漏吗？我用官方模型开tensorrt测了一下，并用memory_profiler 可视化了一下内存，发现它确实会涨一下，但是也没有持续的上涨。

0 replies

liuhongen1234567 · 2024-11-14T08:24:46Z

liuhongen1234567
Nov 14, 2024

我这边一张图重复了10万次

0 replies

liuhongen1234567 · 2024-11-14T08:26:24Z

liuhongen1234567
Nov 14, 2024

可视化内存的代码如下predict_rec.py.zip

0 replies

JoshonSmith · 2024-11-18T03:36:31Z

JoshonSmith
Nov 18, 2024

我这边一张图重复了10万次

每次将图做点修改后，执行，重复10万次呢

0 replies

Yins11 · 2024-11-20T11:54:11Z

Yins11
Nov 20, 2024
Author

可视化内存的代码如下predict_rec.py.zip

感谢解答！已定位到原因，是环境问题，现在可以正常推理了。

0 replies

forgeyao · 2024-11-26T08:42:37Z

forgeyao
Nov 26, 2024

什么原因，能说一下吗

1 reply

SWHL Dec 5, 2024
Maintainer

我也好奇，是啥原因了

GreatV · 2024-12-05T01:40:06Z

GreatV
Dec 5, 2024
Maintainer

根据您描述的问题,这是一个使用TensorRT推理时内存持续上涨的问题。以下是一些建议和解决方案:

内存泄漏的主要原因:

使用TensorRT推理时,默认会缓存engine,导致内存不断增长
批处理设置和模型本身的内存管理可能存在问题

建议的解决方案:

a) 调整批处理参数:

--rec_batch_num=1  # 将批处理大小设为1
--max_batch_size=1 # 限制最大批处理大小

b) 增加内存限制:

--gpu_mem=2000  # 根据显卡实际情况设置GPU显存限制

c) 关闭一些不必要的选项:

--warmup=False  # 关闭预热
--benchmark=False  # 关闭基准测试模式

如果仍然存在问题,可以尝试:

使用非TensorRT模式推理(设置 --use_tensorrt=False)
定期清理显存和缓存
使用较新版本的PaddlePaddle(>=2.6.1)和TensorRT(>=8.6)
考虑使用OpenVINO等其他推理后端

示例推理命令:

python tools/infer/predict_rec.py \
--rec_model_dir=./inference/rec_ppocr_v4/ \
--image_dir="xx" \
--rec_algorithm="SVTR_LCNet" \
--rec_image_shape="3, 48, 320" \
--rec_batch_num=1 \
--rec_char_dict_path="./ppocr/utils/en_dict.txt" \
--use_gpu=True \
--gpu_mem=2000 \
--use_tensorrt=True \
--precision=fp32 \
--warmup=False \
--benchmark=False

这些优化措施应该能够帮助控制内存使用。如果问题仍然存在，建议提交详细的日志信息到PaddleOCR的GitHub issues以获取更多支持。

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

paddleOCR 识别，内存一直上涨，只要进程不杀死，内存就不会下降 #14331

{{title}}

Replies: 12 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

paddleOCR 识别，内存一直上涨，只要进程不杀死，内存就不会下降 #14331

🔎 Search before asking

🐛 Bug (问题描述)

🏃‍♂️ Environment (运行环境)

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

Replies: 12 comments · 1 reply

Yins11 Nov 12, 2024 Author

Yins11 Nov 14, 2024 Author

Yins11 Nov 20, 2024 Author

SWHL Dec 5, 2024 Maintainer

GreatV Dec 5, 2024 Maintainer

Replies: 12 comments 1 reply

Yins11
Nov 12, 2024
Author

Yins11
Nov 14, 2024
Author

Yins11
Nov 20, 2024
Author

SWHL Dec 5, 2024
Maintainer

GreatV
Dec 5, 2024
Maintainer