Make pyopencl ship with the napari bundle app

[haesleinhuepf]

🚀 Feature

Make pyopencl ship with the napari bundle app installer.

Motivation

Hi all,

following some discussion on image.sc with @tlambert03 and @jni, I'm suggesting that pyopencl gets shipped together with the napari bundle app installer. At least on Windows this would resolve a couple of installation-issues. I'm sorry that I introduced pyopencl/pyclesperanto to so many plugins, but it is kind of necessary if you want to process 3D data in reasonable time.

Pitch

Add pyopencl somewhere here.

Alternatives

I think folks are working on some conda-tricks for the bundled app. I would love to know how far this project is and/or what they might say about alternatives.

Additional context

The installation issue is weakly related to issues of that kind:

• Installing some plugins fails when conda-based environments are mixed with builtin pip installer napari#3223

I think if the bundle would contain a couple of more dependencies, such as pyopencl, the likelihood of failing installations on Windows would decrease.

Btw. I'm happy to help implementing and testing this!

[AhmetCanSolak]

On installing Windows, shipping pyopencl with napari bundle can definitely take care of some installation issues. If the user missing correct .icd files(OpenCL drivers) for their system, shipping pyopencl with napari won't be enough no matter what. If we do this perhaps we should(I mean we must :) ) add OpenCL drivers as a requirement to napari.

Alternatively hosting correct windows .whl files and installing them with pip can be an alternative for plugins depending on pyopencl. Have you considered @haesleinhuepf ?

[jni]

As I mentioned in #23, I'm +1 on this.

@AhmetCanSolak can you clarify your comment a bit — are you saying .icd files for most users can be bundled in wheels?

[jni]

Or no, I think you're saying that @haesleinhuepf should create wheels for PyOpenCL, which we have discussed in community meetings but then the issue is that libraries would need to depend on roberts-pyopencl rather than pyopencl, which is weird to say the least...

[AhmetCanSolak]

@jni I am saying required .icd files vary a lot and we cannot bring with napari bundle. For intel cpus, amd cpus, amd gpus, nvidia gpus(usually comes with normal gpu driver) one needs different .icd files. Intel OpenCL driver usually not installed by default on many (especially Windows) machines in my experience so far. On OSX this is less of a problem as Metal drivers are typically there by default.

Independent from .icd issues, I suggest to @haesleinhuepf to host the wheels for PyOpenCL and plugins can depend on roberts-pyopencl yes. Might be weird to say but it would work :)

[haesleinhuepf]

Intel OpenCL driver usually not installed by default on many (especially Windows) machines in my experience so far.

I have the contrary experience: OpenCL works on Intel GPUs on Windows always out of the box.

Intel OpenCL driver usually not installed by default on many (especially Windows) machines in my experience so far.

I'm afraid therefore I would have to build pyopencl on my computer, which I haven't managed yet.

[AhmetCanSolak]

Intel OpenCL driver usually not installed by default on many (especially Windows) machines in my experience so far.

I'm afraid therefore I would have to build pyopencl on my computer, which I haven't managed yet.

I also like to drop this here: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyopencl

[haesleinhuepf]

Yeah, I know. This website is explicitly no repository and the author took actions so that you cannot specify the wheels there as dependencies in other places.

@AhmetCanSolak What speaks against bundling pyopencl with the napari-app?

[AhmetCanSolak]

Yeah, I know. This website is explicitly no repository and the author took actions so that you cannot specify the wheels there as dependencies in other places.

Yes but I do not know what is the license on those, website says they are provided as-is. If it is possible to take them and host? I don't know.

@AhmetCanSolak What speaks against bundling pyopencl with the napari-app?

I am not against it at all :) If we do bundle pyopencl with napari-app, I just want to add OpenCL drivers as a requirement to napari-app in our docs/website. As a user, it would be a bad experience to download napari-app and not being able to use a cool plugin due to a dependency I am not aware of.

Hosting PyOpenCL wheels was just an alternative idea, in case group decides to not bundle pyopencl with napari-app.

[cgohlke]

Yes but I do not know what is the license on those, website says they are provided as-is. If it is possible to take them and host? I don't know.

Feel free to redistribute the wheels. There is no extra license besides the pyopencl license.

There are two builds of pyopencl, one against OpenCL 1.2 from the CUDA SDK, the other against OpenCL 2.1 from the Intel SDK.

Since pyopencl switched from boost to pybind11, it is straightforward to build pyopencl against the CUDA SDK. I am using the following siteconf.py file to build against CUDA SDK:

import sys
CL_PRETEND_VERSION = '1.2'
CL_TRACE = False
CL_ENABLE_GL = True
CL_USE_SHIPPED_EXT = False
CL_INC_DIR = ['X:/CUDA/include']
CL_LIB_DIR =  ['X:/CUDA/lib/' + ('x64' if '64 bit' in sys.version else 'Win32')]
CL_LIBNAME = ['OpenCL']
del sys

[tlambert03]

Thank you @cgohlke!

[haesleinhuepf]

There are two builds of pyopencl, one against OpenCL 1.2 from the CUDA SDK, the other against OpenCL 2.1 from the Intel SDK.

Thanks for mentioning that @cgohlke ! Quite some GPUs/drivers around only support OpenCL 1.2 (and not 2.1). Thus, I would vote for depending on the 1.2 one...

[jaimergp]

This should be handled automatically by the conda-forge packaging machinery. Packages in CF link against openCL ICD loaders, which know how to speak to the actual implementations on the fly.

[cgohlke]

link against openCL ICD loaders, which know how to speak to the actual implementations on the fly.

Linking against CUDA or Intel SDK doesn't mean the binaries won't work on other platforms. Vendors ship compatible OpenCL ICDs for Windows these days. Anyway, I uploaded new pyopencl wheels that link to (and include) the open source Khronos' OpenCL-ICD-Loader DLL. These are meant to replace the pyopencl+cl21 builds. The pyopencl+cl12 binaries are an option for systems without proper ICD or buggy OpenCL implementations.