Skip to content

[WIP] Windows support #753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 35 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
3b55fb9
Windows makefile
NicolasHug Jul 4, 2025
3a20ace
Fix
NicolasHug Jul 4, 2025
02a998d
Add Windows workflow
NicolasHug Jul 4, 2025
4af56ad
Fix ?
NicolasHug Jul 4, 2025
95bd9b2
tmp
NicolasHug Jul 4, 2025
9b3d6e4
Try to use custom test-infra branch
NicolasHug Jul 4, 2025
18f78ab
dont use env helper?
NicolasHug Jul 4, 2025
c814edf
Hmm?
NicolasHug Jul 4, 2025
a25882e
reorder
NicolasHug Jul 4, 2025
b46e5ac
Try something
NicolasHug Jul 4, 2025
c6c30ce
Comment-out some workflows
NicolasHug Jul 5, 2025
8930df5
Remove compilation flags
NicolasHug Jul 5, 2025
db0d342
rework flags
NicolasHug Jul 5, 2025
059c925
Remove some flags
NicolasHug Jul 5, 2025
46d4df5
Add RUNTIME destination for windows
NicolasHug Jul 5, 2025
4ebd78a
Set output dirs to CMAKE_CURRENT_BINARY_DIR for Windows
NicolasHug Jul 5, 2025
43c9e4f
Add comment
NicolasHug Jul 5, 2025
41b90f4
Disable post-build-script for Windows
NicolasHug Jul 5, 2025
8c0bc38
Fix expected wheel artifact name
NicolasHug Jul 5, 2025
cbbd34f
Update .github/workflows/windows_wheel.yaml
NicolasHug Jul 5, 2025
2ccb7ee
Update .github/workflows/windows_wheel.yaml
NicolasHug Jul 5, 2025
a06060a
fix extension finder
NicolasHug Jul 5, 2025
2791490
Merge branch 'add-windows-cmake' of github.com:NicolasHug/torchcodec …
NicolasHug Jul 5, 2025
7fb6536
maybe fix something
NicolasHug Jul 6, 2025
0be6336
try to add Library/bin to PATH??
NicolasHug Jul 6, 2025
5b98275
Add build and test from source workflow
NicolasHug Jul 6, 2025
a1ba873
Fix FFmpeg versions for source build
NicolasHug Jul 6, 2025
f61971b
Try random things
NicolasHug Jul 6, 2025
2f2a38a
Add quiet flags
NicolasHug Jul 7, 2025
c570edb
print dependencies of all libs
NicolasHug Jul 7, 2025
821982f
Crazy stuff
NicolasHug Jul 7, 2025
af7a98a
crazy cmake stuff
NicolasHug Jul 7, 2025
1413666
????
NicolasHug Jul 7, 2025
eb8b318
???
NicolasHug Jul 7, 2025
edca0c3
typo
NicolasHug Jul 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 68 additions & 68 deletions .github/workflows/cpp_tests.yaml
Original file line number Diff line number Diff line change
@@ -1,73 +1,73 @@
name: CPP tests
# name: CPP tests

on:
push:
branches: [ main ]
pull_request:
# on:
# push:
# branches: [ main ]
# pull_request:

concurrency:
group: unit-test${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_number || github.ref }}
cancel-in-progress: true
# concurrency:
# group: unit-test${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_number || github.ref }}
# cancel-in-progress: true

defaults:
run:
shell: bash -l -eo pipefail {0}
# defaults:
# run:
# shell: bash -l -eo pipefail {0}

jobs:
Cpp-tests:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
ffmpeg-version-for-tests: ['4.4.2', '5.1.2', '6.1.1', '7.0.1']
steps:
- name: Check out repo
uses: actions/checkout@v3
- name: Setup conda env
uses: conda-incubator/setup-miniconda@v3
with:
auto-update-conda: true
# Using miniforge instead of miniconda ensures that the default
# conda channel is conda-forge instead of main/default. This ensures
# ABI consistency between dependencies:
# https://conda-forge.org/docs/user/transitioning_from_defaults/
miniforge-version: latest
activate-environment: test
python-version: '3.12'
- name: Update pip
run: python -m pip install --upgrade pip
- name: Install torch dependencies
run: |
python -m pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu
- name: Install ffmpeg, pkg-config and pybind11
run: |
conda install "ffmpeg=${{ matrix.ffmpeg-version-for-tests }}" pkg-config pybind11 -c conda-forge
ffmpeg -version
- name: Build and run C++ tests
run: |
# Note: we're not setting BUILD_AGAINST_ALL_FFMPEG_FROM_S3 here, so
# we're building libtorchcodec against the installed FFmpeg version
# (from conda-forge) instead of building against our pre-built non-GPL
# FFmpeg libraries.
# The reason we need this is because the C++ tests decode x264 files.
# x264 support is not LGPL, os it is not supported by our
# pre-built non-GPL FFmpeg libraries. And if we were to build against
# those, this is also what the tests would be loading at run time,
# then failing when we try to decode x264.
# To remediate that, we build against the FFmpeg that we installed
# from conda-forge (which is able to decode x264), and that's also
# what gets loaded at run time.
# The Python tests are also decoding x264 files, and are built against
# our non-GPL FFmpeg. And yet they pass. This is because in Python
# we're able to distinguish between build-time (non-GPL FFmpeg) and
# run time (conda-forge FFmpeg).
# jobs:
# Cpp-tests:
# runs-on: ubuntu-latest
# strategy:
# fail-fast: false
# matrix:
# ffmpeg-version-for-tests: ['4.4.2', '5.1.2', '6.1.1', '7.0.1']
# steps:
# - name: Check out repo
# uses: actions/checkout@v3
# - name: Setup conda env
# uses: conda-incubator/setup-miniconda@v3
# with:
# auto-update-conda: true
# # Using miniforge instead of miniconda ensures that the default
# # conda channel is conda-forge instead of main/default. This ensures
# # ABI consistency between dependencies:
# # https://conda-forge.org/docs/user/transitioning_from_defaults/
# miniforge-version: latest
# activate-environment: test
# python-version: '3.12'
# - name: Update pip
# run: python -m pip install --upgrade pip
# - name: Install torch dependencies
# run: |
# python -m pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu
# - name: Install ffmpeg, pkg-config and pybind11
# run: |
# conda install "ffmpeg=${{ matrix.ffmpeg-version-for-tests }}" pkg-config pybind11 -c conda-forge
# ffmpeg -version
# - name: Build and run C++ tests
# run: |
# # Note: we're not setting BUILD_AGAINST_ALL_FFMPEG_FROM_S3 here, so
# # we're building libtorchcodec against the installed FFmpeg version
# # (from conda-forge) instead of building against our pre-built non-GPL
# # FFmpeg libraries.
# # The reason we need this is because the C++ tests decode x264 files.
# # x264 support is not LGPL, os it is not supported by our
# # pre-built non-GPL FFmpeg libraries. And if we were to build against
# # those, this is also what the tests would be loading at run time,
# # then failing when we try to decode x264.
# # To remediate that, we build against the FFmpeg that we installed
# # from conda-forge (which is able to decode x264), and that's also
# # what gets loaded at run time.
# # The Python tests are also decoding x264 files, and are built against
# # our non-GPL FFmpeg. And yet they pass. This is because in Python
# # we're able to distinguish between build-time (non-GPL FFmpeg) and
# # run time (conda-forge FFmpeg).

build_tests_dir="${PWD}/build_tests"
mkdir $build_tests_dir
pushd $build_tests_dir
TORCH_PATH=$(python -c "import pathlib, torch; print(pathlib.Path(torch.__path__[0]))")
Torch_DIR="${TORCH_PATH}/share/cmake/Torch"
cmake .. -DTorch_DIR=$Torch_DIR -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON -DCMAKE_VERBOSE_MAKEFILE=ON
cmake --build .
ctest --output-on-failure
popd
# build_tests_dir="${PWD}/build_tests"
# mkdir $build_tests_dir
# pushd $build_tests_dir
# TORCH_PATH=$(python -c "import pathlib, torch; print(pathlib.Path(torch.__path__[0]))")
# Torch_DIR="${TORCH_PATH}/share/cmake/Torch"
# cmake .. -DTorch_DIR=$Torch_DIR -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON -DCMAKE_VERBOSE_MAKEFILE=ON
# cmake --build .
# ctest --output-on-failure
# popd
218 changes: 109 additions & 109 deletions .github/workflows/docs.yaml
Original file line number Diff line number Diff line change
@@ -1,116 +1,116 @@
name: Docs
# name: Docs

on:
push:
branches: [ main ]
pull_request:
# on:
# push:
# branches: [ main ]
# pull_request:

permissions:
id-token: write
contents: write
# permissions:
# id-token: write
# contents: write

defaults:
run:
shell: bash -l -eo pipefail {0}
# defaults:
# run:
# shell: bash -l -eo pipefail {0}

jobs:
generate-matrix:
uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
with:
package-type: wheel
os: linux
test-infra-repository: pytorch/test-infra
test-infra-ref: main
with-cpu: disable
with-xpu: disable
with-rocm: disable
with-cuda: enable
build-python-only: "disable"
build:
needs: generate-matrix
strategy:
fail-fast: false
name: Build and Upload wheel
uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main
with:
repository: pytorch/torchcodec
ref: ""
test-infra-repository: pytorch/test-infra
test-infra-ref: main
build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
pre-script: packaging/pre_build_script.sh
post-script: packaging/post_build_script.sh
smoke-test-script: packaging/fake_smoke_test.py
package-name: torchcodec
trigger-event: ${{ github.event_name }}
build-platform: "python-build-package"
build-command: "BUILD_AGAINST_ALL_FFMPEG_FROM_S3=1 ENABLE_CUDA=1 python -m build --wheel -vvv --no-isolation"
# jobs:
# generate-matrix:
# uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
# with:
# package-type: wheel
# os: linux
# test-infra-repository: pytorch/test-infra
# test-infra-ref: main
# with-cpu: disable
# with-xpu: disable
# with-rocm: disable
# with-cuda: enable
# build-python-only: "disable"
# build:
# needs: generate-matrix
# strategy:
# fail-fast: false
# name: Build and Upload wheel
# uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main
# with:
# repository: pytorch/torchcodec
# ref: ""
# test-infra-repository: pytorch/test-infra
# test-infra-ref: main
# build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
# pre-script: packaging/pre_build_script.sh
# post-script: packaging/post_build_script.sh
# smoke-test-script: packaging/fake_smoke_test.py
# package-name: torchcodec
# trigger-event: ${{ github.event_name }}
# build-platform: "python-build-package"
# build-command: "BUILD_AGAINST_ALL_FFMPEG_FROM_S3=1 ENABLE_CUDA=1 python -m build --wheel -vvv --no-isolation"

build-docs:
runs-on: linux.4xlarge.nvidia.gpu
strategy:
fail-fast: false
matrix:
# 3.9 corresponds to the minimum python version for which we build
# the wheel unless the label cliflow/binaries/all is present in the
# PR.
python-version: ['3.9']
cuda-version: ['12.6']
ffmpeg-version-for-tests: ['7']
container:
image: "pytorch/manylinux2_28-builder:cuda${{ matrix.cuda-version }}"
options: "--gpus all -e NVIDIA_DRIVER_CAPABILITIES=video,compute,utility"
needs: build
steps:
- name: Setup env vars
run: |
cuda_version_without_periods=$(echo "${{ matrix.cuda-version }}" | sed 's/\.//g')
echo cuda_version_without_periods=${cuda_version_without_periods} >> $GITHUB_ENV
python_version_without_periods=$(echo "${{ matrix.python-version }}" | sed 's/\.//g')
echo python_version_without_periods=${python_version_without_periods} >> $GITHUB_ENV
- uses: actions/download-artifact@v4
with:
name: pytorch_torchcodec__${{ matrix.python-version }}_cu${{ env.cuda_version_without_periods }}_x86_64
path: pytorch/torchcodec/dist/
- name: Setup miniconda using test-infra
uses: pytorch/test-infra/.github/actions/setup-miniconda@main
with:
python-version: ${{ matrix.python-version }}
# We install conda packages at the start because otherwise conda may have conflicts with dependencies.
default-packages: "nvidia/label/cuda-${{ matrix.cuda-version }}.0::libnpp nvidia::cuda-nvrtc=${{ matrix.cuda-version }} nvidia::cuda-toolkit=${{ matrix.cuda-version }} nvidia::cuda-cudart=${{ matrix.cuda-version }} nvidia::cuda-driver-dev=${{ matrix.cuda-version }} conda-forge::ffmpeg=${{ matrix.ffmpeg-version-for-tests }}"
- name: Check env
run: |
${CONDA_RUN} env
${CONDA_RUN} conda info
${CONDA_RUN} nvidia-smi
${CONDA_RUN} conda list
- name: Assert ffmpeg exists
run: |
${CONDA_RUN} ffmpeg -buildconf
- name: Update pip
run: ${CONDA_RUN} python -m pip install --upgrade pip
- name: Install PyTorch
run: |
${CONDA_RUN} python -m pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu${{ env.cuda_version_without_periods }}
${CONDA_RUN} python -c 'import torch; print(f"{torch.__version__}"); print(f"{torch.__file__}"); print(f"{torch.cuda.is_available()=}")'
- name: Install torchcodec from the wheel
run: |
wheel_path=`find pytorch/torchcodec/dist -type f -name "*cu${{ env.cuda_version_without_periods }}-cp${{ env.python_version_without_periods }}*.whl"`
echo Installing $wheel_path
${CONDA_RUN} python -m pip install $wheel_path -vvv
# build-docs:
# runs-on: linux.4xlarge.nvidia.gpu
# strategy:
# fail-fast: false
# matrix:
# # 3.9 corresponds to the minimum python version for which we build
# # the wheel unless the label cliflow/binaries/all is present in the
# # PR.
# python-version: ['3.9']
# cuda-version: ['12.6']
# ffmpeg-version-for-tests: ['7']
# container:
# image: "pytorch/manylinux2_28-builder:cuda${{ matrix.cuda-version }}"
# options: "--gpus all -e NVIDIA_DRIVER_CAPABILITIES=video,compute,utility"
# needs: build
# steps:
# - name: Setup env vars
# run: |
# cuda_version_without_periods=$(echo "${{ matrix.cuda-version }}" | sed 's/\.//g')
# echo cuda_version_without_periods=${cuda_version_without_periods} >> $GITHUB_ENV
# python_version_without_periods=$(echo "${{ matrix.python-version }}" | sed 's/\.//g')
# echo python_version_without_periods=${python_version_without_periods} >> $GITHUB_ENV
# - uses: actions/download-artifact@v4
# with:
# name: pytorch_torchcodec__${{ matrix.python-version }}_cu${{ env.cuda_version_without_periods }}_x86_64
# path: pytorch/torchcodec/dist/
# - name: Setup miniconda using test-infra
# uses: pytorch/test-infra/.github/actions/setup-miniconda@main
# with:
# python-version: ${{ matrix.python-version }}
# # We install conda packages at the start because otherwise conda may have conflicts with dependencies.
# default-packages: "nvidia/label/cuda-${{ matrix.cuda-version }}.0::libnpp nvidia::cuda-nvrtc=${{ matrix.cuda-version }} nvidia::cuda-toolkit=${{ matrix.cuda-version }} nvidia::cuda-cudart=${{ matrix.cuda-version }} nvidia::cuda-driver-dev=${{ matrix.cuda-version }} conda-forge::ffmpeg=${{ matrix.ffmpeg-version-for-tests }}"
# - name: Check env
# run: |
# ${CONDA_RUN} env
# ${CONDA_RUN} conda info
# ${CONDA_RUN} nvidia-smi
# ${CONDA_RUN} conda list
# - name: Assert ffmpeg exists
# run: |
# ${CONDA_RUN} ffmpeg -buildconf
# - name: Update pip
# run: ${CONDA_RUN} python -m pip install --upgrade pip
# - name: Install PyTorch
# run: |
# ${CONDA_RUN} python -m pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu${{ env.cuda_version_without_periods }}
# ${CONDA_RUN} python -c 'import torch; print(f"{torch.__version__}"); print(f"{torch.__file__}"); print(f"{torch.cuda.is_available()=}")'
# - name: Install torchcodec from the wheel
# run: |
# wheel_path=`find pytorch/torchcodec/dist -type f -name "*cu${{ env.cuda_version_without_periods }}-cp${{ env.python_version_without_periods }}*.whl"`
# echo Installing $wheel_path
# ${CONDA_RUN} python -m pip install $wheel_path -vvv

- name: Check out repo
uses: actions/checkout@v3
# - name: Check out repo
# uses: actions/checkout@v3

- name: Install doc dependencies
run: |
cd docs
${CONDA_RUN} python -m pip install -r requirements.txt
- name: Build docs
run: |
cd docs
${CONDA_RUN} make html
- uses: actions/upload-artifact@v4
with:
name: Built-Docs
path: docs/build/html/
# - name: Install doc dependencies
# run: |
# cd docs
# ${CONDA_RUN} python -m pip install -r requirements.txt
# - name: Build docs
# run: |
# cd docs
# ${CONDA_RUN} make html
# - uses: actions/upload-artifact@v4
# with:
# name: Built-Docs
# path: docs/build/html/
Loading
Loading