Windows issue in conda-forge tests

[maresb]

Description of your problem

Hooray, a Windows issue!

On conda-forge build, the test succeeds in Linux and OSX, but fails in Windows.

Azure Pipeline logs

The relevant line is monolithic, containing lots of \n characters, but a simple print("""[paste here]""") cleans things up:

Error while running command:
['C:\\Miniconda\\conda-bld\\pyscaffoldext-markdown_1615157790774\\_test_env\\python.exe', '-m', 'tox', '-e', 'build']
{'stderr': -2, 'universal_newlines': True}

******************** Terminal ($? = 1) ********************
build create: C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build
build installdeps: build
build installed: build==0.3.0,packaging==20.9,pep517==0.9.1,pyparsing==2.4.7,toml==0.10.2
build run-test-pre: PYTHONHASHSEED='457'
build run-test: commands[0] | python -m build .
Unable to copy 'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\test_generated_extension0\\tmpex4iq1oa\\my_project\\.tox\\build\\Scripts\\venvlauncher.EXE'
Unable to copy 'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\test_generated_extension0\\tmpex4iq1oa\\my_project\\.tox\\build\\Scripts\\venvlauncher.exe'
Unable to copy 'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\test_generated_extension0\\tmpex4iq1oa\\my_project\\.tox\\build\\Scripts\\venvwlauncher.exe'
Traceback (most recent call last):
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\venv\__init__.py", line 75, in create
    self._setup_pip(context)
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\venv\__init__.py", line 303, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
ERROR: InvocationError for command 'C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\Scripts\python.EXE' -m build . (exited with code 1)
___________________________________ summary ___________________________________
ERROR:   build: commands failed

I don't know what's going on, and as a policy for my personal sanity, I don't do Windows. Any ideas?

[FlorianWilhelm]

Windows Errors! Don't we all love them so much? I don't know how many hours they cost us already in the development of PyScaffold.

The problem seems to be that venvlauncher.exe is missing and at least in 2019 there was a conda-related issue in Python itself. But it might also be related to tox where @abravalheri is an absolute expert.

[FlorianWilhelm]

Not sure I completely understand the Azure CI/CD but this run seems to have worked. Also is there any difference to PyScaffold itself, as this has even way more Windows unit tests?

[maresb]

Yay! I disabled pytest for pyscaffoldext-markdown in order to get the package to build. I figure we can diagnose this in parallel and then release a new build with tests enabled, although for rebuilds, conda-forge seems to only test against linux.

[abravalheri]

Sorry guys, just seeing it now...
I am not very familiar with the conda-forge pipelines, but there is something very weird... for example there is a bunch of warnings about undefined pytest marks that are clearly present in setup.cfg...
I wonder if the test runner is even seeing setup.cfg, that could explain some missing packages, maybe?

Instead of running pytest directly, could we run the tests as we do in the Cirrus CI via tox? It is kind of a blind guess, but it would help to isolate the problem (so we can compare directly the behaviour of Azure and Cirrus and try to find which configuration it differs).

Some other thing we should do also is update the building process, i.e. replace python -m pep517.build with python -m build. I will work on that now.

[maresb]

Ah yes! I was having a lot of trouble with tox because of some of the assumptions that the tests seem to make:

The conda-forge workflow is to install the package, copy the test files to a temp directory, and run the tests from there. At least in the pyscaffold tests, there seemed to be a lot of tests to ensure for example that putup was inside of .tox. But with this workflow, it's in site-packages instead.

It also means that I need to explicitly copy setup.cfg. Will provide references shortly...

[abravalheri]

I see... Yes in the codebase of PyScaffold we added some of those tests because at somepoint we were testing things against the wrong installation/other versions... So they work like metatests/sanity checks.

Please let me know when you find those, I think we could arrange some env vars and skip when running tests for conda-forge.

I have a silly question (sorry not familiar with conda): in order to see if the changes in the pyscaffold-dsproject solve the problem in the feedstock, do we need to keep tagging new versions?

[maresb]

Yes, I'm trying to dig up a list of those tests which were failing. (I got around them by adding not slow and not system so I wonder if we could simply add another pytest marker for skipincondaforge or something.) I will send links soon.

For reference, here's how one copies setup.cfg to the temporary test directory (I need to do this for pyscaffoldext-markdown)

[maresb]

Ok, this assertion fails because pyscaffold isn't installed in src. This function is called by these three tests.

Finally, assert ".tox" in putup

Those are all the references I wanted to find.

And your very good and not at all silly question, conda-forge renders from PyPI releases. So to first order, you'd need to publish to PyPI. But there are some mechanisms to test locally via Docker. But in that case I don't think we have access to Windows (but I could be wrong). I also think conda-forge only tests against Linux for new versions.

Let me see if I can find some feasible way to test conda-forge based on local source...

[abravalheri]

Thank you very much @maresb. Later today I will try to replicate the tests from conda in a windows machine... let's see what I can find wrong about it...

I wonder if adding python-build as a test dependency + the lastest change I did in the master a few minutes ago would solve the problem....

[abravalheri]

Regarding the tox thing, I will add this to the TODO list also.

[maresb]

I think I have things straightened out on the rc branch, so if you publish rc2 I should be able to build that on conda-forge. I'm still looking into other possibilities.

Regarding tox in general, unfortunately I'm not so familiar with it, but from what I gathered, when running tox it triggers its own build and runs pytest against that. But in conda-forge I want to run pytest against the conda-forge build, so for conda-forge I'm trying to run pytest by itself.

[abravalheri]

Hi @maresb, I just sat down to investigate these issues and saw the latest comment from Cris. Should we just do the:

- imports
- pip check
- putup --help

as suggested by him? I suppose these should speed up things and alleviate the load on their CI (PyScaffold tests are quite heavy... our Cirrus setup usually takes at least 30min).

For the extensions, I suppose we could just check if the extension flag is amount the output of putup --help, which would be very easy with grep, but then in Windows I have no idea how to make work...

[abravalheri]

If we have problems (e.g. git not being installed on the CI env or not being available), we can also try to postpone the following command (that is currently running the first time the pyscaffold.shell module is imported):

# Currently defined as a top-level var in `src/pyscaffold/shell.py`
git = get_git_cmd()

[maresb]

I just published a potential fix which includes the tests. I am now including the setup.cfg in the testing directory, so maybe the Windows problem goes away. In case it doesn't, then let's fall back to the putup --help solution.

[abravalheri]

Ok, meanwhile I will try to run the recipe locally as suggested by him. Too bad conda just takes forever for installing/creating environemtns...

[maresb]

Ya. Conda is really a PITA. Do you know about mamba, its C++ reimplementation? It's almost as smooth as pip.

[abravalheri]

This tip is 🥇! I had no idea...

[maresb]

Ya, it's truly amazing. It's quite new, which explains why nobody's heard of it. I'm really spoiled now to the point where I can't stand conda anymore. These conda-forge things are really painful for me.

So Windows failed again. Maybe because you can't set environment variables like that? Whatever, we'll do it the simple way. I'll add a comment referring to the commit on the off-chance that someone wants to resurrect the effort.

[abravalheri]

Ok, I also run in trouble when executing conda build for the yaml files....

Thank you very much @maresb for the effort anyway. Ideally Cirrus should be catching all the errors including Windows.

Let me just confirm: we are using the simple way for pyscaffold-dsproject, but it is still preferable if I change the tests in pyscaffold itself to not assume it is running inside tox, right? Or at least skip the checks if the CONDA_BUILD environment variable is defined...

[maresb]

You're welcome. As for the most preferable way to structure the tests, I honestly have no idea. I have little experience in modern software development, and half of what I learned, I actually learned from using PyScaffold on a project. My first instinct was also to do things with environment variables. I was looking through the pytest documentation, and it looks to me like markers are the correct tool for being able to turn off tests. (i.e. pytest -m "not skiponcondaforge") but I've never tried it.

I was thinking it would be clever to make the conda-forge test command putup --help | grep -- --markdown but then I realized that Windows would probably choke on that. I tried putup --pretend --markdown asdf but that still complains about git not being configured. For now I'm just going to do putup --help and call it good.

[abravalheri]

Just to log my investigation: it seems the problem identified in conda-forge CI happens every time you try to use build inside a virtualenv inside a conda env... It might sound a bit of a stretch having 2 nested environments, but it would be natural if we consider tox uses virtualenvs and tox is a very good automation tool for Python projects, including the ones running on conda...

We can follow the issue on pypa/build#253

[abravalheri]

@maresb, I added both marks and envvars checks on pyscaffold/pyscaffold#416...
Would you mind to have a look and see if that would work?