Skip to content

Commit 10cc663

Browse files
hatala91facebook-github-bot
hatala91
authored and
facebook-github-bot
committed
Migrate torch::deploy test infra to pytorch/test-infra (#274)
Summary: With this PR, I propose to migrate [runtime_nightly.yaml](https://github.com/pytorch/multipy/blob/main/.github/workflows/runtime_nightly.yaml) and [runtime_tests.yaml](https://github.com/pytorch/multipy/blob/main/.github/workflows/runtime_tests.yaml) to use [pytorch/test_infra](https://github.com/pytorch/test-infra/wiki/). For this, I introduce a new reusable workflow in [build_test_release.yaml](https://github.com/pytorch/multipy/pull/274/files#diff-e9376faa19019165b5c9299c6e79ef11820184ea067877dd574bf357b7a7e01b) which is called from both the runtime tests and nightly build pipeline. The workflow then uses `pytorch/test_infra's` [linux_job.yml](https://github.com/pytorch/test-infra/blob/main/.github/workflows/) to build, test, run examples and benchmark. Example runs: - py37: https://github.com/pytorch/multipy/actions/runs/3708138595/jobs/6285368845 - py38: https://github.com/pytorch/multipy/actions/runs/3708138595/jobs/6285368958 - py39: https://github.com/pytorch/multipy/actions/runs/3708138595/jobs/6285369062 - py310: https://github.com/pytorch/multipy/actions/runs/3708138595/jobs/6285368633 Issue: #254 Pull Request resolved: #274 Reviewed By: seemethere, PaliC Differential Revision: D42069719 Pulled By: hatala91 fbshipit-source-id: c88a7863dda808a41ab49bfe56cc7b44e77566e4
1 parent 7dd2993 commit 10cc663

File tree

5 files changed

+183
-105
lines changed

5 files changed

+183
-105
lines changed

‎.github/workflows/build_test_release.yaml

+142
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
name: Reusable build, test and release
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
python3-minor-version:
7+
required: true
8+
type: string
9+
runner:
10+
required: true
11+
type: string
12+
distro:
13+
default: centos7
14+
type: string
15+
compat-tests:
16+
default: false
17+
type: boolean
18+
release:
19+
default: false
20+
type: boolean
21+
secrets:
22+
token:
23+
required: true
24+
25+
jobs:
26+
build-test:
27+
uses: pytorch/test-infra/.github/workflows/linux_job.yml@main
28+
with:
29+
runner: ${{ inputs.runner }}
30+
upload-artifact: multipy_runtime_python3.${{ inputs.python3-minor-version }}
31+
repository: pytorch/multipy
32+
gpu-arch-type: cuda
33+
gpu-arch-version: 116
34+
script: |
35+
python_version=3.${{ inputs.python3-minor-version }}
36+
37+
echo "::group::Install runtime build dependencies"
38+
xargs yum install -y <build-requirements-${{ inputs.distro }}.txt
39+
echo "::endgroup::"
40+
41+
echo "::group::Sync git submodules"
42+
git submodule update --init --recursive --jobs 0
43+
echo "::endgroup::"
44+
45+
echo "::group::Setup virtual environment"
46+
if [[ ${{ inputs.python3-minor-version }} -gt 7 ]];
47+
then
48+
conda install -y python=${python_version} mkl mkl-include conda-build pyyaml numpy ipython
49+
conda install -y -c conda-forge libpython-static=${python_version}
50+
conda install -y pytorch torchvision torchaudio cpuonly -c pytorch
51+
conda clean -ya;
52+
else
53+
conda deactivate
54+
pip install virtualenv
55+
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
56+
export CFLAGS="-fPIC -g"
57+
~/.pyenv/bin/pyenv install --force ${python_version}
58+
virtualenv -p ~/.pyenv/versions/$(~/.pyenv/bin/pyenv latest ${python_version})/bin/python3 ~/venvs/multipy
59+
source ~/venvs/multipy/bin/activate
60+
pip install \
61+
torch torchvision torchaudio \
62+
--extra-index-url https://download.pytorch.org/whl/cpu;
63+
fi
64+
echo "::endgroup::"
65+
66+
echo "::group::Install"
67+
python -m pip install -e . --install-option="--cudatests"
68+
echo "::endgroup::"
69+
70+
echo "::group::Generate examples"
71+
python multipy/runtime/example/generate_examples.py
72+
cd examples
73+
cmake -S . -B build/ -DMULTIPY_PATH=".." && cmake --build build/ --config Release -j
74+
cd -
75+
echo "::endgroup::"
76+
77+
export PYTHONPATH=$(pwd)
78+
export LIBTEST_DEPLOY_LIB=$(pwd)/multipy/runtime/build/libtest_deploy_lib.so
79+
export LD_LIBRARY_PATH=/opt/conda/lib/:$LD_LIBRARY_PATH
80+
81+
echo "::group::Test C++"
82+
multipy/runtime/build/test_deploy
83+
echo "::endgroup::"
84+
85+
echo "::group::Test Pybind"
86+
python multipy/runtime/test_pybind.py
87+
echo "::endgroup::"
88+
89+
echo "::group::Run examples"
90+
examples/build/hello_world_example
91+
python3 examples/quickstart/gen_package.py
92+
./examples/build/quickstart my_package.pt
93+
./examples/build/movable_example
94+
echo "::endgroup::"
95+
96+
echo "::group::Benchmark"
97+
./multipy/runtime/build/deploy_benchmark 2 none jit multipy/runtime/example/generated/resnet
98+
echo "::endgroup::"
99+
100+
if [[ ${{ inputs.python3-minor-version }} -gt 7 ]];
101+
then
102+
echo "::group::Compat test"
103+
pip install -r compat-requirements.txt
104+
multipy/runtime/build/interactive_embedded_interpreter --pyscript multipy/runtime/test_compat.py
105+
echo "::endgroup::";
106+
fi
107+
108+
echo "::group::Test GPU"
109+
# Separating GPU tests due to issues with py37 and py38: https://github.com/pytorch/multipy/issues/239
110+
pip install --upgrade --force-reinstall torch \
111+
--extra-index-url https://download.pytorch.org/whl/cu116
112+
multipy/runtime/build/test_deploy_gpu
113+
echo "::endgroup::"
114+
115+
echo "::group::Create tarball"
116+
cp -r multipy/runtime/build/dist/* .
117+
rm -rf multipy/runtime/build
118+
tar -czf multipy_runtime_python${python_version}.tar.gz multipy/
119+
echo "::endgroup::"
120+
121+
echo "::group::Move artifact for upload"
122+
mv multipy_runtime_python${python_version}.tar.gz ${RUNNER_ARTIFACT_DIR}/
123+
echo "::endgroup::"
124+
125+
release:
126+
needs: [build-test]
127+
runs-on: ubuntu-latest
128+
if: ${{ inputs.release }}
129+
steps:
130+
- name: Download artifact
131+
uses: actions/download-artifact@v3
132+
with:
133+
name: multipy_runtime_python3.${{ inputs.python3-minor-version }}
134+
path: ./
135+
136+
- name: Update nightly release
137+
uses: pyTooling/Actions/releaser@main
138+
with:
139+
tag: multipy_runtime_python3.${{ inputs.python3-minor-version }}
140+
rm: true
141+
token: ${{ secrets.token }}
142+
files: multipy_runtime_python3.${{ inputs.python3-minor-version }}.tar.gz

‎.github/workflows/runtime_nightly.yaml

+8-65
Original file line numberDiff line numberDiff line change
@@ -6,73 +6,16 @@ on:
66
workflow_dispatch:
77

88
jobs:
9-
unittest:
9+
build-test-release:
1010
strategy:
1111
matrix:
1212
python3-minor-version: [7,8,9,10]
1313
platform: [linux.4xlarge.nvidia.gpu]
1414
fail-fast: false
15-
runs-on: ${{ matrix.platform }}
16-
steps:
17-
- name: Checkout MultiPy
18-
uses: actions/checkout@v2
19-
with:
20-
submodules: true
21-
22-
- name: Setup SSH (Click me for login details)
23-
uses: ./.github/actions/setup-ssh
24-
with:
25-
github-secret: ${{ secrets.GITHUB_TOKEN }}
26-
27-
- name: Clean up previous CUDA driver installations
28-
shell: bash
29-
run: |
30-
set -x
31-
yum list installed | grep nvidia || true
32-
yum list installed | grep cuda || true
33-
sudo yum remove -y cuda || true
34-
sudo yum remove -y cuda-drivers || true
35-
sudo yum remove -y "*nvidia*" || true
36-
- name: Install nvidia driver, nvidia-docker runtime, set GPU_FLAG
37-
run: |
38-
bash .github/scripts/install_nvidia_utils_linux.sh || true
39-
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"
40-
41-
- name: Build
42-
env:
43-
DOCKER_BUILDKIT: 1
44-
run: nvidia-docker build -t multipy --progress=plain --build-arg PYTHON_3_MINOR_VERSION=${{ matrix.python3-minor-version }} --build-arg BUILD_CUDA_TESTS=1 .
45-
46-
- name: Test
47-
run: |
48-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && multipy/runtime/build/test_deploy"
49-
nvidia-docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && multipy/runtime/build/test_deploy_gpu"
50-
51-
52-
- name: Examples
53-
run: |
54-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && examples/build/hello_world_example"
55-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && python3 examples/quickstart/gen_package.py && ./examples/build/quickstart my_package.pt"
56-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && ./examples/build/movable_example"
57-
58-
- name: Benchmark
59-
run: |
60-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && ./multipy/runtime/build/deploy_benchmark 2 none jit multipy/runtime/example/generated/resnet"
61-
62-
- name: Set Python Version
63-
run: |
64-
echo "python-version=3.${{ matrix.python3-minor-version }}" >> $GITHUB_ENV
65-
66-
- name: Create Tarball
67-
run: |
68-
docker cp $(docker run -d multipy):/opt/dist/multipy .
69-
tar -czvf multipy_runtime_python${{ env.python-version }}.tar.gz multipy/
70-
71-
- name: Update nightly release
72-
uses: pyTooling/Actions/releaser@main
73-
with:
74-
tag: nightly-runtime-python${{ env.python-version }}
75-
rm: true
76-
token: ${{ secrets.GITHUB_TOKEN }}
77-
files: |
78-
multipy_runtime_python${{ env.python-version }}.tar.gz
15+
uses: ./.github/workflows/build_test_release.yaml
16+
with:
17+
python3-minor-version: ${{ matrix.python3-minor-version }}
18+
runner: ${{ matrix.platform }}
19+
release: true
20+
secrets:
21+
token: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/runtime_tests.yaml

+7-40
Original file line numberDiff line numberDiff line change
@@ -14,43 +14,10 @@ jobs:
1414
python3-minor-version: [7,8,9,10]
1515
platform: [linux.4xlarge.nvidia.gpu]
1616
fail-fast: false
17-
runs-on: ${{ matrix.platform }}
18-
steps:
19-
- name: Checkout MultiPy
20-
uses: actions/checkout@v2
21-
with:
22-
submodules: true
23-
- name: Install nvidia driver, nvidia-docker runtime, set GPU_FLAG
24-
uses: pytorch/test-infra/.github/actions/setup-nvidia@main
25-
- name: Setup SSH (Click me for login details)
26-
uses: ./.github/actions/setup-ssh
27-
with:
28-
github-secret: ${{ secrets.GITHUB_TOKEN }}
29-
30-
- name: Build
31-
env:
32-
DOCKER_BUILDKIT: 1
33-
run: nvidia-docker build -t multipy --progress=plain --build-arg PYTHON_3_MINOR_VERSION=${{ matrix.python3-minor-version }} --build-arg BUILD_CUDA_TESTS=1 .
34-
35-
- name: C++ Tests
36-
run: |
37-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && multipy/runtime/build/test_deploy"
38-
nvidia-docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && multipy/runtime/build/test_deploy_gpu"
39-
40-
- name: Pybind Tests
41-
run: |
42-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && python multipy/runtime/test_pybind.py"
43-
44-
- name: Examples
45-
run: |
46-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && examples/build/hello_world_example"
47-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && python3 examples/quickstart/gen_package.py && ./examples/build/quickstart my_package.pt"
48-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && ./examples/build/movable_example"
49-
50-
- name: Benchmark
51-
run: |
52-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -lt 8 ]]; then source ~/venvs/multipy/bin/activate; fi && ./multipy/runtime/build/deploy_benchmark 2 none jit multipy/runtime/example/generated/resnet"
53-
54-
- name: Compat Tests
55-
run: |
56-
docker run --rm multipy bash -c "if [[ ${{ matrix.python3-minor-version }} -gt 7 ]]; then pip install -r compat-requirements.txt && multipy/runtime/build/interactive_embedded_interpreter --pyscript multipy/runtime/test_compat.py; fi"
17+
uses: ./.github/workflows/build_test_release.yaml
18+
with:
19+
python3-minor-version: ${{ matrix.python3-minor-version }}
20+
runner: ${{ matrix.platform }}
21+
compat-tests: true
22+
secrets:
23+
token: ${{ secrets.GITHUB_TOKEN }}

‎build-requirements-centos7.txt

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
binutils
2+
bzip2
3+
bzip2-devel
4+
gcc
5+
gtest-devel
6+
libffi-devel
7+
make
8+
openssl-devel
9+
readline-devel
10+
sqlite
11+
sqlite-devel
12+
tk-devel
13+
xz-devel
14+
zlib-devel

‎multipy/runtime/loader.cpp

+12
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,18 @@
6767
#include <multipy/runtime/loader.h>
6868
#include <multipy/runtime/mem_file.h>
6969

70+
/* The ELF version installed with CentOS 7 comes with a trailing 64. For distro
71+
* independent use these are mapped. */
72+
73+
#ifndef R_AARCH64_TLS_DTPREL
74+
#define R_AARCH64_TLS_DTPREL R_AARCH64_TLS_DTPREL64
75+
76+
#endif
77+
#ifndef R_AARCH64_TLS_DTPMOD
78+
#define R_AARCH64_TLS_DTPMOD R_AARCH64_TLS_DTPMOD64
79+
80+
#endif
81+
7082
namespace torch {
7183
namespace deploy {
7284

0 commit comments

Comments
 (0)