TypeError: UbufP2PCommOverlap(): incompatible function arguments.

[holmes313]

Hello,
When I configured --sequence-parallel and --tp-comm-overlap and started the training. It shows below information:
TypeError: UbufP2PCommOverlap(): incompatible function arguments. The following argument types are supported:
1. () -> None

Invoked with: tensor([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], device='cuda:3', dtype=torch.uint8), 3, 2, 16, 2, 0, 0, 3, 0, 0, tensor([])
How to fix it? Thanks.

[Zhihao06]

I have met the same problem, could you tell me how to fix it, thanks

[ZeroAGI]

I have met the same problem, could you tell me how to fix it, thanks

[sallyjunjun]

I have met the same problem, could you tell me how to fix it, thanks