CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout #428

amotl · 2022-06-17T17:27:43Z

Edit: The outcome from this investigation has been converged into an upstream bug report at buildout/buildout#609. It will be kept alive for further ramblings. Do not merge!

At #426 and #427, we found GHA croaks when running buildout with Python 3.10 on macOS. See, for example, https://github.com/crate/crate-python/runs/6935572867.

Invoking tests with CrateDB 4.8.0 and SQLAlchemy 1.4.37
Requirement already satisfied: pip in /Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages (22.1.2)
Collecting zc.buildout==2.13.4
  Downloading zc.buildout-2.13.4-py2.py3-none-any.whl (153 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.0/154.0 kB 7.3 MB/s eta 0:00:00
Collecting setuptools<52,>=8.0
  Downloading setuptools-51.3.3-py3-none-any.whl (786 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 786.2/786.2 kB 9.8 MB/s eta 0:00:00
Installing collected packages: setuptools, zc.buildout
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
Successfully installed setuptools-51.3.3 zc.buildout-2.13.4
Creating directory '/Users/runner/work/crate-python/crate-python/eggs'.
Creating directory '/Users/runner/work/crate-python/crate-python/bin'.
Creating directory '/Users/runner/work/crate-python/crate-python/parts'.
Creating directory '/Users/runner/work/crate-python/crate-python/develop-eggs'.
Develop: '/Users/runner/work/crate-python/crate-python/.'
Getting distribution for 'zc.recipe.testrunner==2.2'.
Got zc.recipe.testrunner 2.2.
Getting distribution for 'zope.testrunner==5.2'.
Got zope.testrunner 5.2.
Getting distribution for 'zc.recipe.egg==2.0.7'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
Got zc.recipe.egg 2.0.7.
Getting distribution for 'zope.interface==5.2.0'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching 'coverage.xml' found anywhere in distribution
warning: no previously-included files matching 'appveyor.yml' found anywhere in distribution
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'benchmarks'
No eggs found in /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/easy_install-uvggdcg2/zope.interface-5.2.0/egg-dist-tmp-s7mlm8qo (setup script problem?)
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface==5.2.0'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
 Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 2174, in main
    getattr(buildout, command)(args)
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 701, in install
    [self[part]['recipe'] for part in install_parts]
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 701, in <listcomp>
    [self[part]['recipe'] for part in install_parts]
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1324, in __getitem__
    options._initialize()
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1432, in _initialize
    self.initialize()
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1438, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/buildout.py", line 1381, in _install_and_load
    zc.buildout.easy_install.install(
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 957, in install
    return installer.install(specs, working_set)
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 730, in install
    for dist in self._get_dist(req, ws):
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 574, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 1745, in _move_to_eggs_dir_and_compile
    [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: not enough values to unpack (expected 1, got 0)

P.S.: The nightly run did not catch it because it only runs ubuntu-latest - why?

amotl · 2022-06-17T17:52:13Z

I think the root cause for the execption at the glob() call is

No eggs found in /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/easy_install-1gm0jz07/zope.interface-5.2.0/egg-dist-tmp-_9jfut_a (setup script problem?)

Observations by others: setuptools 44.1.1 can't find egg in macOS Big Sur (Intel Core i5) after compilation when using buildout pypa/setuptools#2514
However, I don't see any such problems on my machine, with macOS 10.15.7 (Catalina). Tested on both Python 3.10.4 and 3.10.5, installed with pyenv.

amotl · 2022-06-17T18:04:20Z

Replicating what @idgserpro was recommending at pypa/setuptools#2514 (comment),

sw_vers -productVersion
export SYSTEM_VERSION_COMPAT=1
sw_vers -productVersion

GHA shows:

11.6.6
10.16

amotl · 2022-06-17T18:22:33Z

Adding SYSTEM_VERSION_COMPAT=1 with 64133b3 helped indeed, thanks @idgserpro!

Funny enough, Python 3.10 on macOS now succeeds with SA13, but still fails with SA14. As we can clearly see from the output, it looks like the gcc compiler is not installed or not available ¹². On the Python 3.9 run, everything worked smoothly, even compiling the optimization extensions, without further ado ³.

The reason is that while some packages are graceful on this and will use their non-optimized pure Python implementations, the SA14 build pulls in greenlet, which is not.

I am running out of ideas. There might even be something wrong with the environment provisioning operations at GHA?

Getting distribution for 'greenlet!=0.4.17'.
gcc: error: sh -c '/Applications/Xcode_13.2.1.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find gcc 2> /dev/null' failed with exit code 256: (null) (errno=Invalid argument)
xcode-select: Failed to locate 'gcc', requesting installation of command line developer tools.
error: Setup script exited with error: command '/usr/bin/gcc' failed with exit code 72

amotl · 2022-06-17T19:14:54Z

There might even be something wrong with the environment provisioning operations at GHA?

There is a difference between setting up a Python 3.9 ¹ vs. a Python 3.10 ² environment. Downgrading to 3.10.4 and 3.10.3 with c2e92d9 and a6a6c1b didn't help.

Run actions/setup-python@v3
  with:
    python-version: 3.9
    token: ***
Successfully setup CPython (3.9.13)

Run actions/setup-python@v3
Version 3.10.4 was not found in the local cache
Version 3.10.4 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.10.4-2268648287/python-3.10.4-darwin-x64.tar.gz"
Extract downloaded archive
/usr/bin/tar xz -C /Users/runner/work/_temp/55083671-ee71-47e3-a1ed-cb52fe26053e -f /Users/runner/work/_temp/5cb741ad-3c45-4967-ae28-ec2b5fe9aa37
Execute installation script
Check if Python hostedtoolcache folder exist...
Create Python 3.10.4 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
Upgrading pip...
Looking in links: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp1fxdzvoa
Requirement already satisfied: setuptools in /Users/runner/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages (58.1.0)
Requirement already satisfied: pip in /Users/runner/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages (22.0.4)
Collecting pip
Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Successfully installed pip-22.1.2
Create complete file
Successfully setup CPython (3.10.4)

On another patch, we are observing problems with Python 3.10 on macOS 11 It looks like there is no gcc compiler installed or available. Reference: crate/crate-python#428

amotl · 2022-06-18T16:31:03Z

After some further research on this matter, we

created a dedicated repository with a minimal reproduction case at test-buildout-python310 and reported about its outcome at Investigate problems with Buildout on Python 3.10 and macOS >= 11 crate-workbench/test-buildout-python310#1.
reported those observations to the people at Buildout per Problem with Buildout on Python 3.10 and macOS >= 11; with reproduction repository buildout/buildout#609.

More relevant and solid details beyond initial observations and corresponding speculations can be found within those resources.

- The sandbox setup is now based on vanilla Python and virtualenv. - A convenient bootstrap script is provided, which can be used like `source bootstrap.sh`. - The test runner has not been changed, it is still `zope.testrunner`. - The command line invocation interface `./bin/test` has not been changed. - Buildout is now only used for downloading the CrateDB tarball using `hexagonit.recipe.download`. References: - #428 - crate-workbench/test-buildout-python310#1

amotl · 2022-06-20T08:49:38Z

#429 will swiftly resolve all the matters on this.

- The sandbox setup is now based on vanilla Python and virtualenv. - A convenient bootstrap script is provided, which can be used like `source bootstrap.sh`. - The test runner has not been changed, it is still `zope.testrunner`. - The command line invocation interface `./bin/test` has not been changed. - Buildout is now only used for downloading the CrateDB tarball using `hexagonit.recipe.download`. References: - #428 - crate-workbench/test-buildout-python310#1

This reverts commit e918dd9, reversing changes made to 3e58910.

amotl · 2022-06-21T08:35:06Z

With 3e58910, we switched this to all available macOS runners in order to get the big picture.

-- https://github.com/crate/crate-python/actions/runs/2533958091

amotl · 2022-06-22T11:04:25Z

We have been continuing corresponding investigations and discussions at crate-workbench/test-buildout-python310#1 ff.

amotl · 2022-07-22T22:40:25Z

#429 removed significant portions of the Buildout-based setup and resolved the issue already. Closing this.

amotl added 2 commits June 17, 2022 19:23

CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS

d2a1403

CI: Use most recent version of zc.buildout>=2,<3

75d68c1

CI: Investigate macOS woes with sw_vers and `SYSTEM_VERSION_COMPAT

1374ab8

CI: Use SYSTEM_VERSION_COMPAT=1 to fix issue with buildout on macOS

64133b3

amotl added 2 commits June 17, 2022 20:44

CI: Explicitly use Python 3.10.4, the default is Python 3.10.5

c2e92d9

CI: Explicitly use Python 3.10.3

a6a6c1b

amotl added 2 commits June 17, 2022 21:19

CI: Update to actions/setup-python@v4

dfe22af

CI: Explicitly install the wheel package

55aac5e

amotl force-pushed the amo/fix-ci branch from fdd9b19 to 55aac5e Compare June 17, 2022 19:45

This was referenced Jun 18, 2022

Investigate problems with Buildout on Python 3.10 and macOS >= 11 crate-workbench/test-buildout-python310#1

Open

Problem with Buildout on Python 3.10 and macOS >= 11; with reproduction repository buildout/buildout#609

Closed

amotl mentioned this pull request Jun 18, 2022

Modernize sandbox: Reduce footprint of Buildout, fix macOS >= 11, improve CI #429

Merged

amotl added 2 commits June 20, 2022 10:23

CI: Increase verbosity level for buildout

6150a63

CI: Decrease verbosity level for buildout

1ebb586

amotl force-pushed the amo/fix-ci branch from 9edad7d to e23444b Compare June 20, 2022 08:25

amotl closed this Jun 20, 2022

amotl deleted the amo/fix-ci branch June 20, 2022 08:49

amotl restored the amo/fix-ci branch June 21, 2022 07:52

CI: Explicitly use Python 3.10 again

365d850

amotl reopened this Jun 21, 2022

amotl changed the title ~~CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS~~ CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout Jun 21, 2022

Explicitly use all available macOS images on GHA to get the big picture

3e58910

amotl force-pushed the amo/fix-ci branch from e23444b to 3e58910 Compare June 21, 2022 07:56

amotl added 3 commits June 21, 2022 10:01

Merge branch 'master' into amo/fix-ci

e918dd9

Revert "Merge branch 'master' into amo/fix-ci"

693639a

This reverts commit e918dd9, reversing changes made to 3e58910.

CI: Remove conditionals as this will now be focused on macOS anyway

7392663

amotl mentioned this pull request Jun 21, 2022

Possible regression with Python 3.10 on macOS >= 11; with reproduction repository actions/runner-images#5791

Closed

10 tasks

amotl closed this Jul 22, 2022

amotl deleted the amo/fix-ci branch July 22, 2022 22:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout #428

CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout #428

amotl commented Jun 17, 2022 •

edited

Loading

amotl commented Jun 17, 2022 •

edited

Loading

amotl commented Jun 17, 2022

amotl commented Jun 17, 2022 •

edited

Loading

amotl commented Jun 17, 2022

amotl commented Jun 18, 2022

amotl commented Jun 20, 2022

amotl commented Jun 21, 2022

amotl commented Jun 22, 2022

amotl commented Jul 22, 2022

CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout #428

CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout #428

Conversation

amotl commented Jun 17, 2022 • edited Loading

amotl commented Jun 17, 2022 • edited Loading

amotl commented Jun 17, 2022

amotl commented Jun 17, 2022 • edited Loading

Footnotes

amotl commented Jun 17, 2022

Footnotes

amotl commented Jun 18, 2022

amotl commented Jun 20, 2022

amotl commented Jun 21, 2022

amotl commented Jun 22, 2022

amotl commented Jul 22, 2022

amotl commented Jun 17, 2022 •

edited

Loading

amotl commented Jun 17, 2022 •

edited

Loading

amotl commented Jun 17, 2022 •

edited

Loading