Cannot install from the source on macOS arm64

[fharper]

Describe the bug (mandatory)

Since there is no arm64 wheel, I want to install PyMuPDF from the source. When I run python setup.py install this is what I get: https://asciinema.org/a/RCFQwfLcdl02ou99WWx2jVtST

running install
running bdist_egg
running egg_info
writing PyMuPDF.egg-info/PKG-INFO
writing dependency_links to PyMuPDF.egg-info/dependency_links.txt
writing top-level names to PyMuPDF.egg-info/top_level.txt
adding license file 'COPYING' (matched pattern 'COPYING*')
reading manifest file 'PyMuPDF.egg-info/SOURCES.txt'
writing manifest file 'PyMuPDF.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-11.5-arm64/egg
running install_lib
running build_py
running build_ext
building 'fitz._fitz' extension
swigging fitz/fitz.i to fitz/fitz_wrap.c
swig -python -o fitz/fitz_wrap.c fitz/fitz.i
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/opt/homebrew/opt/zlib/include -I/opt/homebrew/opt/libffi/include -I/usr/local/include/mupdf -I/usr/local/include -I/usr/include/freetype2 -I/usr/local/include/freetype2 -I/usr/X11R6/include/freetype2 -I/opt/homebrew/include -I/opt/homebrew/include/mupdf -I/opt/homebrew/include/freetype2 -I/Users/fharper/.pyenv/versions/3.8.10/include/python3.8 -c fitz/fitz_wrap.c -o build/temp.macosx-11.5-arm64-3.8/fitz/fitz_wrap.o
fitz/fitz_wrap.c:1756:7: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
      0,                                    /* tp_print */
      ^
/Users/fharper/.pyenv/versions/3.8.10/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
    Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
    ^
/Users/fharper/.pyenv/versions/3.8.10/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
fitz/fitz_wrap.c:1923:7: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
      0,                                    /* tp_print */
      ^
/Users/fharper/.pyenv/versions/3.8.10/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
    Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
    ^
/Users/fharper/.pyenv/versions/3.8.10/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
fitz/fitz_wrap.c:3628:37: warning: incompatible pointer types passing 'char **' to parameter of type 'unsigned char **' [-Wincompatible-pointer-types]
        fz_buffer_storage(ctx, res, &c);
                                    ^~
/usr/local/include/mupdf/fitz/buffer.h:72:75: note: passing argument to parameter 'datap' here
size_t fz_buffer_storage(fz_context *ctx, fz_buffer *buf, unsigned char **datap);
                                                                          ^
fitz/fitz_wrap.c:4775:30: warning: expression result unused [-Wunused-value]
            if (dest->alpha) *s++;
                             ^~~~
fitz/fitz_wrap.c:4774:19: warning: unsequenced modification and access to 's' [-Wunsequenced]
                *s++ = 255 - *s;
                  ^           ~
fitz/fitz_wrap.c:4786:5: warning: code will never be executed [-Wunreachable-code]
    pdf_obj *filter;
    ^~~~~~~~~~~~~~~~
fitz/fitz_wrap.c:4814:11: warning: assigning to 'unsigned char *' from 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
        c = PyBytes_AS_STRING(imagedata);
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fitz/fitz_wrap.c:4817:11: warning: assigning to 'unsigned char *' from 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
        c = PyByteArray_AS_STRING(imagedata);
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fitz/fitz_wrap.c:5093:39: error: incomplete definition of type 'struct pdf_annot'
        pdf_dict_put_string(ctx, annot->obj, PDF_NAME(DA), (const char *) da, len);
                                 ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5114:13: error: implicit declaration of function 'pdf_load_annots' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
            pdf_load_annots(ctx, page, annots_arr);
            ^
fitz/fitz_wrap.c:5287:49: error: incomplete definition of type 'struct pdf_annot'
        pdf_obj *popup = pdf_dict_get(ctx, annot->obj, PDF_NAME(Popup));
                                           ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5290:32: error: incomplete definition of type 'struct pdf_annot'
        pdf_dict_del(ctx, annot->obj, PDF_NAME(AP));
                          ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5298:42: error: incomplete definition of type 'struct pdf_annot'
            if (!pdf_objcmp(ctx, p, annot->obj)) {
                                    ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5326:31: error: incomplete definition of type 'struct pdf_annot'
        pdf_page *page = annot->page;
                         ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5327:74: error: incomplete definition of type 'struct pdf_annot'
        for (annotptr = &page->annots; *annotptr; annotptr = &(*annotptr)->next) {
                                                              ~~~~~~~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5329:45: error: incomplete definition of type 'struct pdf_annot'
            o = pdf_dict_gets(ctx, irt_annot->obj, "IRT");
                                   ~~~~~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5331:46: error: incomplete definition of type 'struct pdf_annot'
                if (!pdf_objcmp(ctx, o, annot->obj)) {
                                        ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5410:48: error: incomplete definition of type 'struct pdf_annot'
        names = JM_get_annot_id_list(ctx, annot->page);
                                          ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5421:38: error: incomplete definition of type 'struct pdf_annot'
        pdf_dict_puts_drop(ctx, annot->obj, "NM", name);
                                ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5444:74: error: incomplete definition of type 'struct pdf_annot'
        for (annotptr = &page->annots; *annotptr; annotptr = &(*annotptr)->next) {
                                                              ~~~~~~~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5446:79: error: incomplete definition of type 'struct pdf_annot'
            const char *response = pdf_to_string(ctx, pdf_dict_gets(ctx, annot->obj, "NM"), &len);
                                                                         ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5473:74: error: incomplete definition of type 'struct pdf_annot'
        for (annotptr = &page->annots; *annotptr; annotptr = &(*annotptr)->next) {
                                                              ~~~~~~~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5475:46: error: incomplete definition of type 'struct pdf_annot'
            if (xref == pdf_to_num(ctx, annot->obj)) {
                                        ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:5565:18: warning: '&&' within '||' [-Wlogical-op-parentheses]
    if (ch >= 32 && ch <= 127 || ch == 10) {
        ~~~~~~~~~^~~~~~~~~~~~ ~~
fitz/fitz_wrap.c:5565:18: note: place parentheses around the '&&' expression to silence this warning
    if (ch >= 32 && ch <= 127 || ch == 10) {
                 ^
        (                    )
fitz/fitz_wrap.c:6506:47: warning: passing 'char *' to parameter of type 'const unsigned char *' converts between pointers to integer types with different sign [-Wpointer-sign]
    res = fz_new_buffer_from_copied_data(ctx, data, strlen(data));
                                              ^~~~
/usr/local/include/mupdf/fitz/buffer.h:105:81: note: passing argument to parameter 'data' here
fz_buffer *fz_new_buffer_from_copied_data(fz_context *ctx, const unsigned char *data, size_t size);
                                                                                ^
fitz/fitz_wrap.c:6706:36: error: incomplete definition of type 'struct pdf_annot'
                JM_set_field_type(ctx, doc, annot->obj, type);
                                            ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:6707:38: error: incomplete definition of type 'struct pdf_annot'
                pdf_dict_put_text_string(ctx, annot->obj, PDF_NAME(T), fieldname);
                                              ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:6729:34: error: incomplete definition of type 'struct pdf_annot'
                pdf_array_push(ctx, form, annot->obj); // Cleanup relies on this statement being last
                                          ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fitz/fitz_wrap.c:6748:30: error: incomplete definition of type 'struct pdf_annot'
    pdf_document *doc = annot->page->doc;
                        ~~~~~^
/usr/local/include/mupdf/pdf/document.h:32:16: note: forward declaration of 'struct pdf_annot'
typedef struct pdf_annot pdf_annot;
               ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
10 warnings and 20 errors generated.
error: command 'clang' failed with exit status 1

To Reproduce (mandatory)

Simply run python setup.py install

Expected behavior (optional)

Installing without error :)

Your configuration (mandatory)

macOS 11.5.2 build 20G95 on arm64
Python 3.9.7
pip 21.3

Additional context (optional)

I installed MuPDF with Homebrew since I'm on an M1 MacBook. I also successfully installed it by building from the source also, which was a second attempt since I wasn't sure if MuPDF was the issue. Everything from MuPDF seems fine.

I'm not sure what is going on here, any ideas?

[JorjMcKie]

Sorry, I cannot follow the video given in the link: every is scrolled away, before I see what's happening.
But anyway, best you connect to the guys doing the same thing: #875.
They seem to have found a way to install on M1.

[fharper]

I should have mentioned in my initial issue report that I followed the steps in #875, and as some other mentioned, it didn't work for me.

[JorjMcKie]

Ok, but I still cannot set the start of things going wrong. Can you not produce a normal console log?
I am no Mac user, so won't be able to reproduce anyway or give Mac-specific advice. But the error may be obvious enough ...

[fharper]

Yes, it's in the initial comment. It's probably obvious :)

[JorjMcKie]

Looks like a mismatch between PyMuPDF source version and MuPDF version:
PyMuPDF = 1.18.*, MuPDF = 1.19.0
I saw the same errors in my first compile of v1.18 source with the new MuPDF.

[JorjMcKie]

the new MuPDF no longer has such things: annot->obj or annot->page - und you have myriads of errors on those.

[fharper]

OMG, that was the issue, I needed to use the same MuPDF version, thanks a lot! Sorry if it was written somewhere and I missed it.

[fharper]

For people looking to use PyMuPDF on M1, the solution that worked for me is to

1. build & install MuPDF from source. Be sure to use the same minor version than PyMuPDF
2. build & install PyMuPDF from source.

[JorjMcKie]

You may be wondering, why there is no wheel for you.
The reason is:

1. I am forced to use a CI platform like Github Actions for generating wheels. The only exception: because I am a Windows user, and having multiple Python versions with multiple bitness (32bit & 64bit) on the same windows machine is literally a no-brainer, I generate Windows wheels on my machine flawlessly since years.
2. Linux Intel and Linux ARM can be generated on Github actions without problems, because all the required platforms and Python versions are available as Docker images. This includes even compiling MuPDF from within a Docker environment. And even for the Linux ARM platform there is an emulation that can be run on Github. Thanks to the provision of Linux Docker images, I could even do that also on my Windows machine - within WSL (Windows subsystem for Linux).
3. For Mac things are more complicated - as usual, if you forgive me the sarcastic sidekick: for Intel platforms it does work based on the great work done by the Homebrew. Within that environment, MuPDF, and then PyMuPDF can be compiled and generated. Takes much longer, than Linux, but it works.
   And for this constellation I do provide wheels.
   There also is a way to create M1 wheels in general on Github actions using cibuildwheel - but i know of no way to compile MuPDF for M1 (as it does work on your machine) on a Github Actions machine (!).

[fharper]

Thanks for everything you are doing @JorjMcKie !