Implement some missing element wise Add/Sub/Mul/Div/Neg operations for CPU and CUDA EPs

[Zyrin]

Description

• [CPU EP] Implement Add/Sub/Mul/Div element wise operations for (u)int8, (u)int16, uint32 and uint64.
• [CPU EP] Implement Neg unary operation for int16
• [CUDA EP] Implement Add/Sub/Mul/Div element wise operations for (u)int8 and (u)int16

Motivation and Context

This solves #23051

[tianleiwu]

This will increase binary size. Is the missing type used in any real model?

[Zyrin]

@microsoft-github-policy-service agree company="Cellumation"

[Zyrin]

I do not know if any "real" model use these types for these operations.
I tried to use uint8 operations in my model and found that onnxruntime was not supporting them, although the onnx api documentation has support for them. So I just went ahead and implemented all these missing types.
The binary size grows by <0.6% for libonnxruntime.so and <0.3% for libonnxruntime_providers_cuda.so.

[xadupre]

This may increase the binary size. +@scottmckay

[tianleiwu]

@Zyrin, please following https://github.com/microsoft/onnxruntime/blob/main/docs/Coding_Conventions_and_Standards.md#linting to format code.

Also need update documents (You can find the updated documents in artifacts of Windows GPU Doc Gen CI Pipeline from Checks).

[tianleiwu]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline

[tianleiwu]

/azp run Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-linux-gpu-ci-pipeline,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline,Android CI Pipeline

[tianleiwu]

/azp run iOS CI Pipeline,ONNX Runtime React Native CI Pipeline,CoreML CI Pipeline,Linux DNNL CI Pipeline,Linux MIGraphX CI Pipeline,Linux ROCm CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 9 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[Zyrin]

I applied the linting fixes. @tianleiwu could you restart the pipelines?

[tianleiwu]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline

[tianleiwu]

/azp run Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline

[tianleiwu]

/azp run Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline,CoreML CI Pipeline,Linux DNNL CI Pipeline,Linux MIGraphX CI Pipeline,Linux ROCm CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 7 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 8 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[tianleiwu]

@Zyrin, there are some build pipeline failed. You need update the unit tests to run on cuda and cpu provider only. See some examples in the same test file.

You will also need update operator documents (you can get them from artifacts of Windows GPU Doc Gen CI Pipeline).

[Zyrin]

@tianleiwu I assume you want me to only run the tests on the CPU and CUDA EPs like with the following code snipped from element_wise_ops_test.cc:1837:

if (nullptr != DefaultCpuExecutionProvider()) {
  std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
  execution_providers.push_back(DefaultCpuExecutionProvider());
  test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
}
if (nullptr != DefaultCudaExecutionProvider()) {
  std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
  execution_providers.push_back(DefaultCudaExecutionProvider());
  test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
}

Alternatively I could exclude the TensorRT and DNNL EPs, but I do not know if there are EPs that are not tested here, and thus would fail on someone else.

On that account should I only change the failing tests or all the tests I added?

[tianleiwu]

@tianleiwu I assume you want me to only run the tests on the CPU and CUDA EPs like with the following code snipped from element_wise_ops_test.cc:1837:

Right. You can follow the code snippet to fix failing tests that is introduced by this.

[Zyrin]

I fixed the tests. Is there a way for me to generate the docs, or is the easiest way to generate them to trigger the Windows GPU Doc Gen CI Pipeline?

[tianleiwu]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline

[tianleiwu]

/azp run Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-linux-gpu-ci-pipeline,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline,Android CI Pipeline

[tianleiwu]

/azp run iOS CI Pipeline,ONNX Runtime React Native CI Pipeline,CoreML CI Pipeline,Linux DNNL CI Pipeline,Linux MIGraphX CI Pipeline,Linux ROCm CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 9 pipeline(s).

[tianleiwu]

@Zyrin, I've triggered the pipelines

[Zyrin]

@tianleiwu The Linux GPU CI Pipeline fails on the test case ModelTests/ModelTest.Run/cuda__models_zoo_opset12_YOLOv312_yolov312 with a segmentation fault. How can I run this test locally to investigate the problem?

I tries to build the project with the flags from the CI. Unfortunately i get errors like this Load model ../models/opset7/test_bvlc_alexnet/model.onnx failed. File doesn't exist. So I'm guessing I am missing the models. Strangely the tests on my machine did not even include the models_zoo tests.

[tianleiwu]

@Zyrin, I rerun the failed test to see whether it is transient issue. The updated documents can be downloaded: https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1592734&view=artifacts&pathAsName=false&type=publishedArtifacts

[Zyrin]

@tianleiwu I rebased to the current main branche and pushed everything. Hopefully the CI should now run without an issue.

[tianleiwu]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline

[tianleiwu]

/azp run Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline,Android CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 9 pipeline(s).

[tianleiwu]

/azp run iOS CI Pipeline,ONNX Runtime React Native CI Pipeline,CoreML CI Pipeline,Linux DNNL CI Pipeline,Linux MIGraphX CI Pipeline,Linux ROCm CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI

[azure-pipelines]

Azure Pipelines successfully started running 7 pipeline(s).