Refactor python build

[rdspring1]

This PR updates the build to use a pyproject.toml and isolates the python bindings into python directory.

Install From Source:

git clone https://github.com/NVIDIA/Fuser.git
cd Fuser
pip install -r python/requirements.txt

[MAX_JOBS] python setup.py develop [args]  # DEPRECATED
pip install --no-build-isolation -e python -v

Details

• Moved csrc/python_frontend and nvfuser to python.
• Moved tools/gen_nvfuser_version.py and tools/memory.py to python.
• Created a new setup.py in python. This is the new primary setup.py.
• Updated github workflows
• Created symbolic links to support setup.py in root directory.

Changes to argument passing to root/setup.py and root/python/setup.py

• python/utils.py has the common utilities between root/setup.py and root/python/setup.py
• Updated argument parsing to use argparse to create a dataclass configuration.
• The argparse creates a default dataclass if no arguments are not provided in the command line.
• NVFUSER_BUILD_ENV_VARS then overrides the values in the dataclass.
• The root/setup.py only supports command-line arguments.

[rdspring1]

!build --dev

[xwang233]

!build --dev

[xwang233]

!build --dev

[xwang233]

!build --dev

[xwang233]

!build --dev

[xwang233]

!test --dev

[rdspring1]

!test --dev

[rdspring1]

Upgraded to fix false positive E231 error in python/nvfuser/__init__.py strings.

[rdspring1]

!test --dev

[jjsjann123]

One other question I have is, what's the reason of still keeping pieces in ./setup.py, instead of moving all the logic into ./python/setup.py? Are we planning future PRs to keep moving things, since we are going to remove root directory setup.py eventually.

[jjsjann123]

nitpick, is there any reason that we are not jumping to the new build command in CI yet?

[rdspring1]

Changed

[rdspring1]

pip install didn't work for clang-tidy, so I reverted to the old way.

  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
      main()
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 209, in prepare_metadata_for_build_editable
      return hook(metadata_directory, config_settings)
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/setuptools/build_meta.py", line 450, in prepare_metadata_for_build_editable
      return self.prepare_metadata_for_build_wheel(
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/setuptools/build_meta.py", line 379, in prepare_metadata_for_build_wheel
      self._bubble_up_info_directory(metadata_directory, ".egg-info")
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/setuptools/build_meta.py", line 350, in _bubble_up_info_directory
      info_dir = self._find_info_directory(metadata_directory, suffix)
    File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/setuptools/build_meta.py", line 361, in _find_info_directory
      assert len(candidates) == 1, f"Multiple {suffix} directories found"
  AssertionError: Multiple .egg-info directories found

[xwang233]

I think the purpose of keeping the root path /setup.py is to not breaking existing CI that builds nvfuser from source (yes, I'm saying lightning-thunder). We should add a deprecation warning to the root path /setup.py if it has not been added yet.

[jjsjann123]

LGTM.

[rdspring1]

!test --dev

[rdspring1]

!test

[crcrpar]

It seems that https://github.com/NVIDIA/Fuser/tree/5c0fb061ade125ba9e52ebef9226c06e74ff3836/python/tools is also installed as tools when building and installing nvfuser from source (see the bottom code block).
At glance, the tools directory looks specific for build. Thus I think find_packages of

Fuser/python/utils.py

Lines 548 to 553 in 5c0fb06

	setup(
	name=config.wheel_name,
	version=version_tag,
	url="https://github.com/NVIDIA/Fuser",
	description="A Fusion Code Generator for NVIDIA GPUs (commonly known as 'nvFuser')",
	packages=find_packages(),

may want to specify some args.

6m 20s mkozuki ~/ghq/github.com/crcrpar/Fuser [py3.12] (main *?)
% pip uninstall nvfuser
Found existing installation: nvfuser 0.2.27+git287ede2
Uninstalling nvfuser-0.2.27+git287ede2:
  Would remove:
    /home/mkozuki/.pyenv/versions/3.12.10/envs/py3.12/lib/python3.12/site-packages/nvfuser-0.2.27+git287ede2.dist-info/*
    /home/mkozuki/.pyenv/versions/3.12.10/envs/py3.12/lib/python3.12/site-packages/nvfuser/*
    /home/mkozuki/.pyenv/versions/3.12.10/envs/py3.12/lib/python3.12/site-packages/tools/*