ZFPY osx wheels delocate dylibs

[halehawk]

ZFPY osx wheels are not linked with dylibs, we need to figure out how to delocate the dylibs correctly.

[william-silversmith]

I was going to experiment with zfpy on MacOS but it seems installing from source didn't work well either. Thanks for putting this library together!

(cv) $ pip install zfpy --no-cache-dir --no-binary :all: --force-reinstall
Collecting zfpy
  Downloading zfpy-0.5.5.tar.gz (3.0 kB)
  Preparing metadata (setup.py) ... done
Skipping wheel build for zfpy, due to binaries being disabled for it.
Installing collected packages: zfpy
  Attempting uninstall: zfpy
    Found existing installation: zfpy 0.5.5
    Uninstalling zfpy-0.5.5:
      Successfully uninstalled zfpy-0.5.5
  Running setup.py install for zfpy ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for zfpy did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running install
      running build
      running build_ext
      building 'zfpy' extension
      creating build
      creating build/temp.macosx-10.9-universal2-3.9
      creating build/temp.macosx-10.9-universal2-3.9/build
      creating build/temp.macosx-10.9-universal2-3.9/build/python
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iinclude -IDIR/.virtualenvs/cv/lib/python3.9/site-packages/numpy/core/include -IDIR/.virtualenvs/cv/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c build/python/zfpy.c -o build/temp.macosx-10.9-universal2-3.9/build/python/zfpy.o
      clang: error: no such file or directory: 'build/python/zfpy.c'
      clang: error: no input files
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  WARNING: No metadata found in DIR/.virtualenvs/cv/lib/python3.9/site-packages
  Rolling back uninstall of zfpy
  Moving to DIR/.virtualenvs/cv/lib/python3.9/site-packages/zfpy-0.5.5.dist-info/
   from DIR/.virtualenvs/cv/lib/python3.9/site-packages/~fpy-0.5.5.dist-info
  Moving to DIR/.virtualenvs/cv/lib/python3.9/site-packages/zfpy.cpython-39-darwin.so
   from /private/var/folders/w5/99_dvybs2zd1g4mrt3lhm1h40000gn/T/pip-uninstall-do5yl3iy/zfpy.cpython-39-darwin.so
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> zfpy

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
WARNING: You are using pip version 22.0.2; however, version 22.0.3 is available.
You should consider upgrading via the 'DIR/.virtualenvs/cv/bin/python -m pip install --upgrade pip' command.

[lindstro]

I'm not sure what pip install does when installing from source. If you want to build zfpy with CMake, you have to add -DBUILD_ZFPY=ON. Maybe there's a way to tell pip to use this build option?

[william-silversmith]

Hi Dr. Lindstrom! Yes, it's possible to pass it using extra_compile_args. I have some examples here from a module I'm working on:

https://github.com/seung-lab/pyspng-seunglab/blob/master/setup.py#L46-L55

[lindstro]

Great. Does pip install still fail when you add -DBUILD_ZFPY=ON?

[william-silversmith]

I took a closer look and it seems that was not sufficient to get it to build. I was able to compress and decompress a numpy array after using python setup.py develop with the following modifications:

1. run make
2. modify setup.py as follows:

    ext_modules=[
        Extension("zfpy", ["python/zfpy.pyx"],
            include_dirs=["include", np.get_include()],
            libraries=["zfp"], 
            library_dirs=["lib"],
        )
    ]

I tried to see if I could at least reproduce this using an sdist, but the zfp.pxd was not included by default using this setup, so I added a MANIFEST.in file in the same directory as setup.py with the following contents:

include python/zfpy.pxd
include python/zfpy.pyx
recursive-include include *.h
recursive-include src *.c *.h

This makes sure all the necessary c files are included in the package. However, this isn't sufficient because libzfp.a has not been built. In the pyspng-seunglab setup.py file referenced above I call out to make in a subshell for building on macos (for some reason only clang on MacOS didn't want to build for that module). For Linux and MacOS, I think the same strategy would work here. Windows is another beast, but probably a cmake solution would work for all three of them (but I'm not very good at cmake...).

I do have one cmake example I could follow which is located here: https://github.com/seung-lab/DracoPy/blob/master/setup.py

I recognize that you have a more formal build process here, so I'm sure you'd want to tweak the exact build sequence and folders.

I think others may disagree with me, but in my own libraries, I like to pre-compile the cython and build from there which eliminates the cython dependency. This might not be as future proof though I can't articulate specifically in what ways it could fail.

[halehawk]

You can try pip install zfpy today, it may be all problems gone now.

…

Sent from my iPhone

On Feb 24, 2022, at 10:18 AM, William Silversmith ***@***.***> wrote:  I took a closer look and it seems that was not sufficient to get it to build. I was able to compress and decompress a numpy array after using python setup.py develop with the following modifications: run make modify setup.py as follows: ext_modules=[ Extension("zfpy", ["python/zfpy.pyx"], include_dirs=["include", np.get_include()], libraries=["zfp"], library_dirs=["lib"], ) ] I tried to see if I could at least reproduce this using an sdist, but the zfp.pxd was not included by default using this setup, so I added a MANIFEST.in file in the same directory as setup.py with the following contents: include python/zfpy.pxd include python/zfpy.pyx recursive-include include *.h recursive-include src *.c *.h This makes sure all the necessary c files are included in the package. However, this isn't sufficient because libzfp.a has not been built. In the pyspng-seunglab setup.py file referenced above I call out to make in a subshell for building on macos (for some reason only clang on MacOS didn't want to build for that module). For Linux and MacOS, I think the same strategy would work here. Windows is another beast, but probably a cmake solution would work for all three of them (but I'm not very good at cmake...). I do have one cmake example I could follow which is located here: https://github.com/seung-lab/DracoPy/blob/master/setup.py I recognize that you have a more formal build process here, so I'm sure you'd want to tweak the exact build sequence and folders. I think others may disagree with me, but in my own libraries, I like to pre-compile the cython and build from there which eliminates the cython dependency. This might not be as future proof though I can't articulate specifically in what ways it could fail. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[william-silversmith]

Just gave it a try and still getting the same error unfortunately. T_T

(tmp) wms:~$ pip install zfpy --force-reinstall --no-cache-dir
Collecting zfpy
  Downloading zfpy-0.5.5-cp39-cp39-macosx_10_9_x86_64.whl (95 kB)
     |████████████████████████████████| 95 kB 9.3 MB/s            
Installing collected packages: zfpy
Successfully installed zfpy-0.5.5
WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
You should consider upgrading via the '/Users/wms/.virtualenvs/tmp/bin/python -m pip install --upgrade pip' command.
(tmp) wms:~$ python
Python 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:25:35) 
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import zfpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/wms/.virtualenvs/tmp/lib/python3.9/site-packages/zfpy.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/libzfp.0.dylib
  Referenced from: /Users/wms/.virtualenvs/tmp/lib/python3.9/site-packages/zfpy.cpython-39-darwin.so
  Reason: tried: '/usr/lib/libzfp.0.dylib' (no such file)

[halehawk]

Can you try zfpy-py310-macos version? I tried this morning, and have no any problem.

…

Sent from my iPhone

On Feb 24, 2022, at 11:22 AM, William Silversmith ***@***.***> wrote:  Just gave it a try and still getting the same error unfortunately. T_T (tmp) wms:~$ pip install zfpy --force-reinstall --no-cache-dir Collecting zfpy Downloading zfpy-0.5.5-cp39-cp39-macosx_10_9_x86_64.whl (95 kB) |████████████████████████████████| 95 kB 9.3 MB/s Installing collected packages: zfpy Successfully installed zfpy-0.5.5 WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available. You should consider upgrading via the '/Users/wms/.virtualenvs/tmp/bin/python -m pip install --upgrade pip' command. (tmp) wms:~$ python Python 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:25:35) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import zfpy Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: dlopen(/Users/wms/.virtualenvs/tmp/lib/python3.9/site-packages/zfpy.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/libzfp.0.dylib Referenced from: /Users/wms/.virtualenvs/tmp/lib/python3.9/site-packages/zfpy.cpython-39-darwin.so Reason: tried: '/usr/lib/libzfp.0.dylib' (no such file) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[william-silversmith]

Oh hey, that worked! I can definitely use this for experiments. Thank you!