Changes to compile on MacOS (Darwin)

[kevemueller]

Small changes needed to compile on MacOS.

Cannot test on other architectures right now.
Updated to recent improvements in configure/pkg-config of libarchive, libcurl.

Build with:

brew install pkg-config libarchive openssl
PKG_CONFIG_PATH="$(brew --prefix  libarchive)/lib/pkgconfig" ./configure --prefix=/tmp/pkg --with-libarchive.pc --with-libcurl --with-openssl.pc

[kevemueller]

Hi,

just checking in on this one - thought I would explain the use-case.

I am building pkg natively on MacOS to prepare packages and package repositories for FreeBSD targets. I have no intention to use pkg to package binaries for MacOS (there is brew).

My findings are:

• There is code inside pkg that handles MacOS and assumes binaries are Mach-O, this triggers warnings, but is not obstructing my use-case.
• pkg repo has major multi-threading issues when running natively on MacOS, crash/corrupted data.pkg, I am using WORKERS_COUNT=1 to get around this.
• pkg create has compression issues with anything except for GZ writing corrupted streams, I am using -l fast -f tgz to get around this.

With the workarounds above, I can fully support my use-case building >4000 packages.

Even if this is a borderline use-case for pkg, I would consider the advantages of clearing out above issues to overall improve the codebase (for all platforms). If there is interest I can offer a pull request with:

• GitHub Action to compile pkg on macos-latest
• Test suite to trigger the above problems.

This is hopefully sufficient to spot the error for somebody intimate with the codebase (not me).
Looking at the codebase I would change the build system to meson-build and try to externalize the embedded dependencies.
After that I would submit pkg as a brew package.
Unfortunately currently I have no resources to support this major change.

Let me know if there is interest in any of the above.
Cheers,
K

[kevemueller]

After rebasing the compression stream errors disappeared. The rebased code properly uses libarchive from pkg-config.
The pkg_repo threading related errors remained.

[emaste]

I think the changes look generally reasonable although I'm not sure what to make of the ... commits - are they intended to be folded into the first as fixups? There seems to be some leftover debugging/commented out stuff?

[kevemueller]

Hi Ed, sorry, the ... were unintentionally committed here. I am cleaning this up as we speak.
I have isolated a test harness github action in #2334 and will apply the cleaned up version on top of that.
I have discovered that the changes needed for MacOS and the changes needed for Linux with pkg-config are very much alike and am combining them here.
Cheers,

[kevemueller]

Hi Ed,

so here is the revised patch.
I have learned that the ./configure options to use pkg-config were not effective in the Makefiles (neither on Ubuntu, nor on MacOS), so I had to change the makesfiles to actually use the pkg-config set variables.

The source changes are all needed for MacOS only, they are the same as the ones in the initial patch. With this both Ubuntu as well as MacOS compile here on GitHub.
The SourceHut compilation on FreeBSD needed another push. Please review if that is is a suitable way to distinguish FreeBSD, there might be simpler ones that I am not aware of. Happy to rework.

The compilations on GitHub run kyua on 3 different OS-es, twice on MacOS to demonstrate the pkg repo multi-threading issue.
The kyua verbose listing is in the logs, but also exported in the build artefacts for local browsing.

As a summary, note that only Ubuntu 22.04/amd64 actually produces no errors.
Ubuntu 24.04/amd64 has an issue with tests/lua:print_msg.
MacOS/arm64 with single threaded repo has mutiple issues with a mismatching ABI string (arm vs. arm64) as well as other cryptical errors.
MacOS/arm64 with multi threaded repo generation fails all over the place. The generated data.pkg is a random concatenation of valid UCL fragments, rendering it impossible to parse. This error was the motivation to do all of the detailed testing, so perhaps somebody intimitate with the codebase can pick it up.
I am also running privately on MacOS/amd64 with the same multi-threading issues and some very interesting other problems. There is less noise on MacOS/amd64 as there the ABI mismatch is not occuring.

All of the artefacts are linked above in the checks, or can be looked at at https://github.com/kevemueller/pkg/actions/runs/11785029973

Hope that this can be merged to trigger overall improvement on the codebase and the build system.
Happy to rework parts that can be done better.

Cheers,
K