find_program does not support meson.pyz

[marinesovitch]

Describe the bug
A clear and concise description of what the bug is.

I build postgres v16 with meson.pyz and I get the following error:

Program f:\USERS\myuser\app\bin\meson.pyz found: NO

meson.build:421:12: ERROR: Program 'f:\\USERS\\myuser\\app\\bin\\meson.pyz' not found or not executable

A full log can be found at f:\USERS\myuser\src\3rdParty\postgresql-16.0\build\meson-logs\meson-log.txt

No matter I run either
python meson.pyz [...args..]
or
meson.pyz [...args..]

The whole run with more details:

f:\USERS\myuser\src\3rdParty\postgresql-16.0>"C:\Program Files\Python311\python.exe" "f:\USERS\myuser\app\bin\meson.pyz" setup build --wipe --prefix=f:\USERS\myuser\app --backend=vs2017 --buildtype=release -Dreadline=disabled -Dicu=disabled -DBISON=f:\USERS\myuser\app\bin\win_bison.exe -DFLEX=f:\USERS\myuser\app\bin\win_flex.exe
The Meson build system
Version: 1.5.1
Source dir: f:\USERS\myuser\src\3rdParty\postgresql-16.0
Build dir: f:\USERS\myuser\src\3rdParty\postgresql-16.0\build
Build type: native build
Project name: postgresql
Project version: 16.0
Activating VS 17.10.4
C compiler for the host machine: cl (msvc 19.40.33812 "Microsoft (R) C/C++ Optimizing Compiler Version 19.40.33812 for x64")
C linker for the host machine: link link 14.40.33812.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Run-time dependency threads found: YES
Library ws2_32 found: YES
Library secur32 found: YES
Program perl found: YES (C:\Strawberry\perl\bin\perl.EXE)
Program python3 found: YES (f:\USERS\myuser\app/bin\python3.EXE)
Program f:\USERS\myuser\app\bin\win_flex.exe found: YES 2.6.4 2.6.4 (f:\USERS\myuser\app\bin\win_flex.exe)
Program f:\USERS\myuser\app\bin\win_bison.exe found: YES 3.8.2 3.8.2 (f:\USERS\myuser\app\bin\win_bison.exe)
Program sed found: YES (C:\Program Files\Git\usr\bin\sed.EXE)
Program prove found: YES (C:\Strawberry\perl\bin\prove.BAT)
Program tar found: YES (C:\Windows\system32\tar.EXE)
Program gzip found: YES (C:\Program Files\Git\usr\bin\gzip.EXE)
Program lz4 found: NO
Program openssl found: YES (f:\USERS\myuser\app/bin\openssl.EXE)
Program zstd found: NO
Program dtrace skipped: feature dtrace disabled
Program config/missing found: YES (sh f:\USERS\myuser\src\3rdParty\postgresql-16.0\config/missing)
Program cp found: YES (C:\Program Files\Git\usr\bin\cp.EXE)
Program xmllint found: YES (f:\USERS\myuser\app/bin\xmllint.EXE)
Program xsltproc found: YES (f:\USERS\myuser\app/bin\xsltproc.EXE)
Program wget found: NO
Program f:\USERS\myuser\app\bin\meson.pyz found: NO

meson.build:421:12: ERROR: Program 'f:\\USERS\\myuser\\app\\bin\\meson.pyz' not found or not executable

A full log can be found at f:\USERS\myuser\src\3rdParty\postgresql-16.0\build\meson-logs\meson-log.txt

while the file exists obviously as it is used to execute the build, besides:

f:\USERS\myuser\app\bin>ls -lh meson.pyz
-rwxr-xr-x 1 myuser 1049089 3.6M Jul 31 11:04 meson.pyz

To Reproduce
Please include your meson.build files, preferably as a minimal toy example showing the issue.
You may need to create simple source code files (don't include private/proprietary code).

meson.build:
https://github.com/postgres/postgres/blob/REL_16_STABLE/meson.build

line 431
https://github.com/postgres/postgres/blob/a459e583e06b640597c2e24ab6043a0701a5a545/meson.build#L431
meson_bin = find_program(meson_binpath, native: true)

steps:

• create meson.pyz according to https://mesonbuild.com/Getting-meson.html#packing-meson-into-a-zipapp
• git clone --branch REL_16_STABLE https://github.com/postgres/postgres.git
• cd postgres
• python meson.pyz setup build --wipe --prefix=%RUNPREFIX% --backend=vs2017

Expected behavior
A clear and concise description of what you expected to happen.

find_program('f:\USERS\myuser\app\bin\meson.pyz') should recognize meson.pyz as an executable binary.

system parameters

• Is this a cross build or just a plain native build (for the same computer)?
  Native build with vs2017 on win

• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.)
  Windows Server 2019 (v1809)

• what Python version are you using e.g. 3.8.0
  python --version
  Python 3.11.2

• what meson --version
  meson.pyz --version
  1.5.1

• what ninja --version if it's a Ninja build
  msvc 19.40.33812 "Microsoft (R) C/C++ Optimizing Compiler Version 19.40.33812 for x64"
  --backend=vs2017

[eli-schwartz]

I build postgres v16 with meson.pyz and I get the following error:

Program f:\USERS\myuser\app\bin\meson.pyz found: NO

meson.build:421:12: ERROR: Program 'f:\\USERS\\myuser\\app\\bin\\meson.pyz' not found or not executable

A full log can be found at f:\USERS\myuser\src\3rdParty\postgresql-16.0\build\meson-logs\meson-log.txt

No matter I run either
python meson.pyz [...args..]
or
meson.pyz [...args..]

On Linux, I can reproduce this problem only if meson.pyz is unable to be run on its own as a command.

git clone https://github.com/mesonbuild/meson
cd meson
./packaging/create_zipapp.py
cd "test cases/common/139 mesonintrospect from scripts/"
../../../meson.pyz setup builddir-pyz --wipe
# works

python ../../../meson.pyz setup builddir-python-pyz --wipe
# works

chmod -x ../../../meson.pyz
python ../../../meson.pyz setup builddir-python-noexec-pyz --wipe
# fails

I expect this end result, since as the error message notes:

ERROR: Program '/tmp/bug13497/meson/meson.pyz' not found or not executable

[eli-schwartz]

You said that:

• you're using Windows
• you have the same problem even if you try running meson.pyz directly, as an executable

Although I am not entirely sure I understand how that works, since on Windows, a python script isn't directly able to be executed as a program?

[eli-schwartz]

/cc @anarazel as IIRC author of that code which is checking for a meson executable in postgres

[marinesovitch]

I don't know the details about how it works on win. Anyway, on the win machine (windows server 2019) I have access to in my company, I can run meson.pyz with or without python, and it seems to work:

f:\USERS\myuser\app\bin>meson.pyz --version
1.5.1
f:\USERS\myuser\app\bin>python meson.pyz --version
1.5.1
f:\USERS\myuser\app\bin>ls -lh meson.pyz
-rwxr-xr-x 1 myuser 1049089 3.6M Jul 31 11:04 meson.pyz

[anarazel]

Although I am not entirely sure I understand how that works, since on Windows, a python script isn't directly able to be executed as a program?

That's a question of configuration, I think. The gui installer asks you whether you want to associate .py with python. If that's done, you can execute .py files via cmd and powershell (although the latter possibly just invokes the former for it?). See e.g. the perl example in https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/ftype

/cc @anarazel as IIRC author of that code which is checking for a meson executable in postgres

FWIW, I have done "path/to/meson.py setup" on windows without a problem. I assume the reason that works is that meson has some logic to look at the shebang - maybe that that logic doesn't work for .pyz?

@marinesovitch

create meson.pyz according to https://mesonbuild.com/Getting-meson.html#packing-meson-into-a-zipapp

That requires specifying the interpreter - what precisely did you use for --interpreter?

[gregsskyles]

I don't know if this helps but I'm able to reproduce this bug in 'pure' Windows (i.e., not WSL, cygwin, mingw, etc.). After looking at what the Postgres meson.build does I came up with this small build file:

project('test', ['c'], version: '0.1', license: 'GPL3')
meson_binpath = 'C:\\tmp\\meson\\meson.pyz'
meson_bin = find_program(meson_binpath, native: true)

Following Eli's lead, this is what I did in a cmd prompt:

C:\Users\GregSkyles>cd \tmp
C:\tmp>git clone https://github.com/mesonbuild/meson
Cloning into 'meson'...
...
C:\tmp>cd meson
C:\tmp\meson> .\packaging\create_zipapp.py
C:\tmp\meson>meson.pyz --version
1.5.99
C:\tmp\meson>mkdir test
C:\tmp\meson>cd test
C:\tmp\meson\test>notepad++ meson.build
C:\tmp\meson\test>..\meson.pyz builddir --wipe
The Meson build system
...
Program C:\tmp\meson\meson.pyz found: NO

meson.build:4:12: ERROR: Program 'C:\\tmp\\meson\\meson.pyz' not found or not executable

Clearly the file is there and 'executable' according to cmd.exe.
The meson-log.txt file shows that find_program() tried to parse the .pyz file and failed.

'utf-8' codec can't decode byte 0xa5 in position 33: invalid start byte
Unusable script 'C:\\tmp\\meson\\meson.pyz'
'utf-8' codec can't decode byte 0xa5 in position 33: invalid start byte
Unusable script 'C:\\tmp\\meson\\meson.pyz'

I'm guessing find_program() doesn't understand the Windows environment's version of executability. You can determine if a file of a given extention is intended to be 'executable' with the assoc command, and how it's supposed to be handled with the ftype command.

C:\tmp\meson>assoc .pyz
.pyz=Python.ArchiveFile
C:\tmp\meson>ftype Python.ArchiveFile
Python.ArchiveFile="C:\Windows\py.exe" "%L" %*

In this case the Python installer from python.org has created these registry entries (and put py.exe in place) so that one can type whatever.pyz at a command prompt and it just runs.

[eli-schwartz]

Can that meson.pyz be run via subprocess.run([the_program], shell=False) and if so, how?

[gregsskyles]

Only if you 'cheat' and do the ftype substitution yourself:

C:\tmp\meson\test>python test.py
1.5.99

C:\tmp\meson\test>type test.py
import subprocess

subprocess.run(['C:\\Windows\\py.exe', 'C:\\tmp\\meson\\meson.pyz', '--version'], shell=False)

It looks like with shell=False, you take cmd.exe out of the equation, and it's what knows about all of the chicanery around file extentions.