torch.distributed.elastic.multiprocessing.errors.ChildFailedError（exitcode: -9）反复在同一个进度出现

[a241s]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

在lora微调的过程中，总是会准确的在进度为40%的时候报错相同的错误
报错信息如下：
torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -9) local_rank: 0 (pid: 1322) of binary: /root/miniconda3/bin/python
Traceback (most recent call last):
File "/root/miniconda3/bin/torchrun", line 8, in
sys.exit(main())
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 347, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
run(args)
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
elastic_launch(
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

finetune.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : (忽略)
host : autodl-container-8a9c4bbf6f-3c104f54
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 1322)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 1322

期望行为 | Expected Behavior

继续训练而不因这个error而停止

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:Ubuntu 22.04.1 LTS
- Python:3.10.8
- Transformers:4.40.0
- PyTorch:2.3.1+cu118
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):11.8

备注 | Anything else?

No response

[YuzaChongyi]

请问是用的哪一份微调代码进行训练的呢？ 目前的信息无法定位，如果稳定在同一个阶段出现比较像是数据问题

[a241s]

请问是用的哪一份微调代码进行训练的呢？ 目前的信息无法定位，如果稳定在同一个阶段出现比较像是数据问题

lora脚本，最近推进到了50%，exitcode 变为 1

[qyc-98]

您好 可以试试其他的数据先测试一下 目前看起来像是数据的问题