MSYS2 update fails due to broken setuptools in preexisting python

[eabase]

Description / Steps to reproduce the issue

I got an MSYS update notification in winget:

# winget list --upgrade-available
No installed package found matching input criteria.

The following packages have an upgrade available, but require explicit targeting for upgrade:
Name  Id          Version  Available Source
-------------------------------------------
MSYS2 MSYS2.MSYS2 20240507 20240727  winget

So because of "explicit targeting message", I opened an MSYS shell and tried to run pacman -Suy, which then failed with:

# pacman -Suy

...
error: failed to commit transaction (conflicting files)
mingw-w64-x86_64-python-setuptools:
...

# python -V
Python 3.11.9
# pip -V
pip 24.2 from /usr/lib/python3.11/site-packages/pip (python 3.11)

---

Screenshots:

Expected behavior

That MSYS/pacman correctly updates and installs python package,
and not blocking all other packages if one fails.

Actual behavior

MSYS pacman fails to install python and subsequently fails to update anything else.

Verification

• I have verified that my MSYS2 is up-to-date before submitting the report (see https://www.msys2.org/docs/updating/)

Windows Version

MSYS_NT-10.0-22631

Are you willing to submit a PR?

no

[mmuetzel]

See: https://www.msys2.org/news/#2024-07-08-file-conflicts-when-updating-python

[eabase]

Hi Markus,

See: https://www.msys2.org/news/#2024-07-08-file-conflicts-when-updating-python

Thanks but that didn't help. I tried with pacman --overwrite "/usr/lib/python3.12/*" -Suy, with same result.

[mmuetzel]

Your files seem to be from a different version of Python in the MINGW64 environment. Try pacman --overwrite "/mingw64/lib/python3.11/*" -Suy instead

[eabase]

Yes, that seem to have worked, but with a lot of scary warnings:

# pacman --overwrite "/usr/lib/python3.12/*" -Suy

# <Waiting a very long time for completion of "checking available disk space">

# $ python -V && pip -V
Python 3.12.4
pip 24.1.2 from /usr/lib/python3.12/site-packages/pip (python 3.12)

# pip list --outdated
Package Version Latest Type
------- ------- ------ -----
pip     24.1.2  24.2   wheel

---

and not able to update the pip package itself:

# pip install -U pip

error: externally-managed-environment
... <see screenshot>

Any idea what this means?

[eabase]

This is very weird, the python in MINGW64 was not updated!

Opening MINGW64 shell and running:

# python -V && pip -V
Python 3.11.9
pip 24.1.2 from C:/msys64/mingw64/lib/python3.11/site-packages/pip (python 3.11)

# pacman -Suy
:: Synchronizing package databases...
 clangarm64 is up to date
 mingw32 is up to date
 mingw64 is up to date
 ucrt64 is up to date
 clang32 is up to date
 clang64 is up to date
 msys is up to date
:: Starting core system upgrade...
 there is nothing to do
:: Starting full system upgrade...
 there is nothing to do

I was expecting pacman to update all MSYS related environments, or is that incorrect expectation?

#  pacman -Q | grep "python"

mingw-w64-ucrt-x86_64-python 3.11.9-1
mingw-w64-x86_64-python 3.11.9-1
mingw-w64-x86_64-python-distlib 0.3.8-2
mingw-w64-x86_64-python-pip 24.1.2-2
mingw-w64-x86_64-python-setuptools 70.2.0-2
python 3.12.4-6
python-pip 24.1.2-1

[eabase]

+ Apparently the slow disk check was due to #4720

The fix is to disable pacman checking the disk space::

# Comment out "CheckSpace" in /etc/pacman.conf 
cd /etc && cp pacman.conf pacman.conf.bak && sed -i 's/^CheckSpace/#CheckSpace/' pacman.conf

[mmuetzel]

You have the latest available version of Python afaict:
https://packages.msys2.org/base/mingw-w64-python

[eabase]

I used to have, now it seem I have 2 and that you are using 3.12 in some environments while still only have 3.11 available in others.

# From MINGW64 shell
# ls -1 /usr/lib |grep python
python3.11/
python3.12/
libpython3.12.dll.a

It says a newer upstream is available, but where/how?

[mmuetzel]

It says a newer upstream is available, but where/how?

Wait until it is packaged and distributed by MSYS2.
(Or build it yourself if you know how to do that. I can't help with that.)

[lazka]

The initial error is likely from using pip outside of a venv in the MINGW64 env, which you shouldn't do on any system. Never run pip outside of a venv, unless you can handle the breakage, or you manage your own Python installation. I know it's confusing, just keep that in mind :)

We'll likely disallow this in the future, similar as with #4762 - but that will likely break various CI workflows depending on it (just a guess, no hard numbers there).

[eabase]

Hi Christoph
@lazka

The initial error is likely from using pip outside of a venv in the MINGW64 env, which you shouldn't do on any system. Never run pip outside of a venv,

(a) I don't understand this. So how should I run it? What is the venv that need to be created (or is it already created?)

(b) Do I need to run the pacman update all 6 MSYS2 environments separately and differently?

(c) How to get rid of the winget notification?

[lazka]

(a) I don't understand this. So how should I run it? What is the venv that need to be created (or is it already created?)

You have to create it. https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments

If you want to mix the system packages with the venv you can either create a venv with "--system-site-packages", or install outside of a venv with "pip install --user", but this will interfere with the system Python, so can break things.

(b) Do I need to run the pacman update all 6 MSYS2 environments separately and differently?

No, it updates everything.

(c) How to get rid of the winget notification?

I don't know, I don't have much winget experience.

[eabase]

@lazka

What is your definition of system-site-packages?
(See my comments on #4762)

(a) This doesn't make any sense. There is no info/docs about having to manually create a venv for updating MSYS2 install. MSYS2 should not interfere by anything on PC/Windows system, by default. Anything else is very problematic and probably wrong behavior.

(b) Well, that is not how it was working in my case above.

(c) "I don't know, I don't have much winget experience."

Ok, please don't get offended, but this is quite worrisome, as MSYS2 was made for cross compatibility (for *nix and Win) and for Windows usage. winget should be considered as the de-facto windows update and package management tool from now on. So any issues with it have to be taken seriously and given resolve priority, if needed.

I much appreciate your help.

---

It looks to me that updating MSYS2 with new Python version is broken and require a complete refresh install of MSYS2. This is highly inconvenient with the increased python update frequency.

[eabase]

@mmuetzel

Wait until it is packaged and distributed by MSYS2.

Given the above issues, if we have to wait until MSYS2 team can repackage new Python versions for all MSYS2 instances, you should probably not make any official releases that contain mixed python versions. (Here you mixed python 3.11 and 3.12 causing update mechanism to break.)

[mmuetzel]

Given the above issues, if we have to wait until MSYS2 team can repackage new Python versions for all MSYS2 instances, you should probably not make any official releases that contain mixed python versions. (Here you mixed python 3.11 and 3.12 causing update mechanism to break.)

You didn't show anywhere that having different versions of Python in the MSYS2 and in the MinGW environments causes the update mechanism to break. And I'm pretty sure that everything is working as expected currently.
Please, refrain from making such outlandish claims.

Like @lazka tried to explain to you multiple times, you installed packages with pip into your MSYS2 installation without using virtual environments. You should not do that. Only install Python packages that are distributed directly by MSYS2 without using a virtual environment. (E.g., pacman -S mingw-w64-x86_64-python-numpy)
You having installed Python packages with pip without using virtual environments caused issues when you tried to update the Python that is distributed by MSYS2. That has nothing to do with the fact that the Python version in the MSYS2 environment (basically a Cygwin Python) is a different version than the one in the MinGW environments.
See also: https://www.msys2.org/docs/environments/
Note the different prefixes for the different environments.

[lazka]

What is your definition of system-site-packages?

$ python -c "import site; print(site.getsitepackages())"
['/usr/lib/python3.12/site-packages']

[eabase]

I managed to pin the current installed version with:

# winget list --upgrade-available

The following packages have an upgrade available, but require explicit targeting for upgrade:
Name  Id          Version  Available Source
-------------------------------------------
MSYS2 MSYS2.MSYS2 20240507 20240727  winget


# winget pin add -q 'MSYS2.MSYS2'

Found MSYS2 Installer [MSYS2.MSYS2]
Pin added successfully

---

@mmuetzel

You didn't show anywhere that having different versions of Python in the MSYS2 and in the MinGW environments causes the update mechanism to break.

Well, I'm not going to argue with you for not looking at the screenshots. The very first one clearly show that there was a real issue with setuptools (3.11). And the remedy provided by you and @lazka fixed it.

And I'm pretty sure that everything is working as expected currently.

So "pretty sure" means you have no idea. I wouldn't come here to waste my time if I didn't think my bug report was real. So no need to make outlandish accusations.

You having installed Python packages with pip without using virtual environments caused issues when you tried to update the Python that is distributed by MSYS2.

I have not installed anything since I first installed MSYS2, neither any python packages. The python pip package that fails to update is the default one that comes with the installation.

And for the UCRT64 environment (only), we have the weird behavior of using python 3.12, but using pip with python 3.11.

$ echo -e "$MSYSTEM" && python -V && pip -V && echo && pip list && echo && pip list --outdated
UCRT64
Python 3.11.9
pip 24.1.2 from /usr/lib/python3.12/site-packages/pip (python 3.12)

Package Version
------- -------
pip     24.1.2

Package Version Latest Type
------- ------- ------ -----
pip     24.1.2  24.2   wheel

The other environments are consistent.

---

Yes, I didn't know we need to use pacman provided packages for python packages. That doesn't make sense to me, so why is that? Which packages have to be pacman provided and which does not?

[lazka]

And for the UCRT64 environment (only), we have the weird behavior of using python 3.12, but using pip with python 3.11.

That means you are missing the ucrt64 pip: pacman -S mingw-w64-ucrt-x86_64-python-pip

[eabase]

... we have the weird behavior of using python 3.12, but using pip with python 3.11.

Sorry I mixed up the versions, it should be the opposite:

... we have the weird behavior of using python 3.11, but using pip with python 3.12.