Skip to content

[Operator Mechanism]fix cuda launch dimension checks for gpu kernels#79376

Open
feixi139 wants to merge 6 commits into
PaddlePaddle:developfrom
feixi139:fix_launch_uint32_gpu_kernels
Open

[Operator Mechanism]fix cuda launch dimension checks for gpu kernels#79376
feixi139 wants to merge 6 commits into
PaddlePaddle:developfrom
feixi139:fix_launch_uint32_gpu_kernels

Conversation

@feixi139

Copy link
Copy Markdown
Contributor

PR Category

Operator Mechanism

PR Types

Bug fixes

Description

This PR updates CUDA kernel launch configuration in the paddle/phi/kernels/funcs module.
The changes add explicit UINT32_MAX and device-limit checks for CUDA grid/block dimensions before kernel launches, and cast the validated launch parameters to uint32_t. This avoids implicit narrowing when int64_t or size_t values are passed as CUDA launch dimensions.
This PR is split from #79333 by directory/module scope to reduce review size. The changes are limited to CUDA launch
configuration handling and do not intend to change operator algorithms or numerical behavior.

是否引起精度变化

@paddle-bot paddle-bot Bot added the contributor External developers label Jun 25, 2026
PaddlePaddle-bot

This comment was marked as outdated.

@PaddlePaddle-bot

PaddlePaddle-bot commented Jun 26, 2026

Copy link
Copy Markdown

🤖 Paddle-CI-Agent | ci_status_monitor | 2026-07-06 13:53:44 UTC+08:00

CI报告基于以下代码生成(30分钟更新一次):
PR commit: 3a4674f | Merge base: 5d803b0 (branch: develop)


1 Required任务 : 43/48 通过

总执行(rerun次数) 总任务 ✅ 通过 ❌ 失败 ⏳ 运行中 ⏸️ 等待中 跳过
89(9) 80 75 2 3 0 0
任务 错误类型 置信度 日志
Fleet Unit test (single card) 环境问题 Job
Check approval 需要 Approval Job

2 失败详情

🔴 Fleet Unit test (single card) — 环境问题(置信度: 高)

分析器: 通用分析(fallback)
失败用例: 无,测试步骤未执行

用例 错误摘要
Check docker image and run container Docker 容器创建失败,NVIDIA runtime 初始化超时

关键日志:

docker: Error response from daemon: failed to create task for container: OCI runtime create failed
nvidia-container-cli: initialization error: driver rpc error: timed out: unknown.
##[error]Process completed with exit code 125.
  • 根因摘要: GPU 容器运行时初始化超时
    Docker 镜像已是最新,但 runner 在创建 CUDA 容器时,NVIDIA container hook 报 driver rpc error: timed out,容器未能启动,后续 Fleet 安装和单卡测试均被跳过。失败发生在 CI runner/驱动容器运行时初始化阶段,与 PR 代码尚未发生执行链路关联,判定为环境问题。

修复建议:

  1. 环境问题,请 rerun;若重跑仍失败,排查 runner 的 NVIDIA driver/container runtime 服务状态。

关联变更: 未分析(失败早于测试执行阶段)

🔴 Check approval — 需要 Approval(置信度: 高)

该 Job 需要人工 Approval,完成审批后 CI 才会继续执行。

  • 根因摘要: 等待人工审批

修复建议:

  1. 请通过人工审批。

关联变更: 不适用

PaddlePaddle-bot

This comment was marked as outdated.

PaddlePaddle-bot

This comment was marked as outdated.

PaddlePaddle-bot

This comment was marked as outdated.

@PaddlePaddle-bot PaddlePaddle-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Paddle-CI-Agent | pr_review | 2026-07-06 11:55:29

📋 Review 摘要

PR 概述:为多处 GPU kernel launch 参数增加设备上限 / uint32_t 检查,并显式转换 launch 维度。
变更范围paddle/phi/kernels/gpu 下 21 个 CUDA kernel 文件。
影响面 TagOperator Mechanism

问题

级别 文件 概述
🔴 Bug paddle/phi/kernels/gpu/gumbel_softmax_kernel.cu:91 height 的 64 位乘法仍使用已被截断的 int 维度,无法保护大 shape
🔴 Bug paddle/phi/kernels/gpu/gumbel_softmax_kernel.cu:142 Gumbel noise 的 size 计算同样在前置 int 参数处丢失 64 位范围
🔴 Bug paddle/phi/kernels/gpu/graph_reindex_kernel.cu:392 仅分别检查 bs / num_edges 不足以下游 int 相加溢出

📝 PR 规范检查

符合规范。

总体评价

本轮按风险优先审查了 CUDA launch 维度检查、grid-stride 覆盖语义和 C++ 整数收窄路径。大多数 launch cast 与设备维度检查匹配,但上述几处仍存在 64 位计数在进入新检查前被截断或下游 32 位相加溢出的风险,需要修复后再合入。

int64_t max_grid_dimx = dev_ctx.GetCUDAMaxGridDimSize()[0];
int64_t height = size_to_axis * size_out_axis;
int block_size = height < max_grid_dimx ? height : max_grid_dimx;
int64_t height = static_cast<int64_t>(size_to_axis) * size_out_axis;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug 这里把乘法提升到 int64_t,但 size_to_axis / size_out_axis 已经先用 int 承接了 funcs::SizeToAxis / SizeOutAxisint64_t 返回值。大 shape 会先在这些局部变量处截断,再参与这里的 launch 计算;OneHotCUDAKernel 内部的 int h / int w 也会继续截断,最终索引错误。

建议修复方式:把 size_to_axissize_from_axissize_out_axis 以及 kernel 内的 h / w 都改为 int64_t;如果该路径暂不支持大 tensor,则在赋给 int 前用 PADDLE_ENFORCE_LE_INT_MAX 明确拒绝,而不是只 cast 乘法。

const float temperature) {
DenseTensor random_tensor;
int64_t size = size_to_axis * size_from_axis;
int64_t size = static_cast<int64_t>(size_to_axis) * size_from_axis;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug GumbleNoiseGenerator::Transform 的形参仍是 int size_to_axis, int size_from_axis,调用方也用 int 保存 SizeToAxis/SizeFromAxis 的结果。这里的 static_cast<int64_t>(size_to_axis) 只能放大已经截断后的值,random_tensor.Resize({size}) 和后续 launch check 都是在错误 size 上运行。

建议修复方式:将 helper 形参和调用方的 size_to_axis / size_from_axis 改为 int64_t 并保持后续计算为 64 位;如果 SoftmaxFunctor 仍限制 int,则在调用前同时校验 size_to_axissize_from_axis 不超过 INT_MAX

errors::InvalidArgument("The first of dims should not be equal to 0."));
int64_t num_edges = neighbors.dims()[0];
PADDLE_ENFORCE_LE_INT_MAX(bs, "graph_reindex num_inputs");
PADDLE_ENFORCE_LE_INT_MAX(num_edges, "graph_reindex num_edges");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug 这里分别保证 bsnum_edges 能放进 int,但下游 Reindex / BufferReindex 的形参是 int,并在 out_nodes->resize(num_inputs + num_edges) 中先做 32 位加法。bsnum_edges 各自合法但总和超过 INT_MAX 时仍会发生有符号溢出,后续 hash table size / launch 检查都基于已损坏的大小。

建议修复方式:在 cast 前增加组合范围检查,例如 PADDLE_ENFORCE_LE(bs, std::numeric_limits<int>::max() - num_edges, ...),或者将 helper 的计数参数和 num_inputs + num_edges 计算整体改为 int64_t 后再对所有仍需 int 的 kernel 入口做显式检查。

@risemeup1111 risemeup1111 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复查新提交后,之前可见的 graph_reindex table_size 收窄问题和 index_elementwise_put tensor-value 路径 launch 检查问题已经补上。当前仍发现一个需要修复的边界问题,细节已放在 inline review comment 里。

  • P3 优先级:P3 非行级:PR 标题。当前标题 [Operator Mechanism]fix cuda launch dimension checks for gpu kernels 在 tag 后缺少空格,建议改为:[Operator Mechanism] fix cuda launch dimension checks for gpu kernels
Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.

Comment on lines +391 to +392
PADDLE_ENFORCE_LE_INT_MAX(bs, "graph_reindex num_inputs");
PADDLE_ENFORCE_LE_INT_MAX(num_edges, "graph_reindex num_edges");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级:P1
这里分别校验 bsnum_edges 不超过 INT_MAX,但后面马上把二者转成 int,并在 Reindex/BufferReindex 里执行 out_nodes->resize(num_inputs + num_edges)FillHashTable(..., out_nodes->size(), ...) 这类仍以 int 为长度的路径。只要 bs + num_edges > INT_MAX(例如二者各自都小于 INT_MAX),int 加法/实参收窄仍会溢出,后续 CUDA_KERNEL_LOOP 默认 int index 也无法覆盖这个组合长度。本 PR 的目标是避免隐式收窄,建议在转成 int 前同时约束组合长度,或把这些 helper 全链路改为 int64_t/CUDA_KERNEL_LOOP_TYPE(..., int64_t)

可先加上入口约束,避免继续进入不支持的 32-bit helper 路径:

Suggested change
PADDLE_ENFORCE_LE_INT_MAX(bs, "graph_reindex num_inputs");
PADDLE_ENFORCE_LE_INT_MAX(num_edges, "graph_reindex num_edges");
PADDLE_ENFORCE_LE_INT_MAX(bs, "graph_reindex num_inputs");
PADDLE_ENFORCE_LE_INT_MAX(num_edges, "graph_reindex num_edges");
PADDLE_ENFORCE_LE_INT_MAX(bs + num_edges, "graph_reindex num_inputs + num_edges");

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor External developers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants