[Bug]: No matching distribution found for diffq>=0.2.1

[zeddan]

Contact Details

[email protected]

What happened?

Hi!

Thanks for taking the time to work on dockerizing demucs 🙌

OS: macOS Monterey 12.0
Docker: 4.2.0

When trying to run a track separation the build fails on installation, please see the Relevant log output section.

However, if RUN python3 -m pip install -e . is replaced with manual installations like the following:

RUN python3 -m pip install dora-search
RUN python3 -m pip install diffq>=0.2.1
RUN python3 -m pip install flake8
RUN python3 -m pip install hydra-colorlog>=1.1
...

the installation of diffq completes. Then the problem instead becomes installing lameenc, with the following error message:

docker build -t facebook/demucs:latest .
[+] Building 2.4s (16/27)
 => [internal] load build definition from Dockerfile                           0.0s
 => => transferring dockerfile: 1.16kB                                         0.0s
 => [internal] load .dockerignore                                              0.0s
 => => transferring context: 2B                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.8                  1.6s
 => [auth] library/python:pull token for registry-1.docker.io                  0.0s
 => [ 1/23] FROM docker.io/library/python:3.8@sha256:68bddbf6e88c9c88d3238e13  0.0s
 => CACHED [ 2/23] RUN apt install git                                         0.0s
 => CACHED [ 3/23] RUN mkdir -p /lib/demucs                                    0.0s
 => CACHED [ 4/23] WORKDIR /lib/demucs                                         0.0s
 => CACHED [ 5/23] RUN git clone -b main --single-branch https://github.com/f  0.0s
 => CACHED [ 6/23] RUN python3 -m pip install dora-search                      0.0s
 => CACHED [ 7/23] RUN python3 -m pip install diffq>=0.2.1                     0.0s
 => CACHED [ 8/23] RUN python3 -m pip install flake8                           0.0s
 => CACHED [ 9/23] RUN python3 -m pip install hydra-colorlog>=1.1              0.0s
 => CACHED [10/23] RUN python3 -m pip install hydra-core>=1.1                  0.0s
 => CACHED [11/23] RUN python3 -m pip install julius>=0.2.3                    0.0s
 => ERROR [12/23] RUN python3 -m pip install lameenc>=1.2                      0.7s
------
 > [12/23] RUN python3 -m pip install lameenc>=1.2:
#16 0.670 ERROR: Could not find a version that satisfies the requirement lameenc (from versions: none)
#16 0.670 ERROR: No matching distribution found for lameenc
#16 0.672 WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
#16 0.672 You should consider upgrading via the '/usr/local/bin/python3 -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c python3 -m pip install lameenc>=1.2]: exit code: 1
make: *** [build] Error 1

What I've tried:

• Upgrading pip with RUN python3 -m pip install --upgrade pip
• Using a later python version by switching to FROM python:3.9 at the top

Thanks in advance!

In which operating system you found the issue?

MacOS

Relevant log output

[+] Building 5.1s (9/11)
 => [internal] load build definition from Dockerfile                           0.0s
 => => transferring dockerfile: 541B                                           0.0s
 => [internal] load .dockerignore                                              0.0s
 => => transferring context: 2B                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.8                  0.8s
 => [1/8] FROM docker.io/library/python:3.8@sha256:68bddbf6e88c9c88d3238e13f0  0.0s
 => CACHED [2/8] RUN apt install git                                           0.0s
 => CACHED [3/8] RUN mkdir -p /lib/demucs                                      0.0s
 => CACHED [4/8] WORKDIR /lib/demucs                                           0.0s
 => CACHED [5/8] RUN git clone -b main --single-branch https://github.com/fac  0.0s
 => ERROR [6/8] RUN python3 -m pip install -e .                                4.2s
------
 > [6/8] RUN python3 -m pip install -e .:
#9 0.864 Obtaining file:///lib/demucs
#9 1.246 Collecting dora-search
#9 1.481   Downloading dora_search-0.1.7.tar.gz (68 kB)
#9 1.654   Installing build dependencies: started
#9 3.874   Installing build dependencies: finished with status 'done'
#9 3.877   Getting requirements to build wheel: started
#9 3.975   Getting requirements to build wheel: finished with status 'done'
#9 3.977     Preparing wheel metadata: started
#9 4.080     Preparing wheel metadata: finished with status 'done'
#9 4.132 ERROR: Could not find a version that satisfies the requirement diffq>=0.2.1 (from demucs) (from versions: 0.0.1, 0.1.0, 0.1.1, 0.2.0)
#9 4.132 ERROR: No matching distribution found for diffq>=0.2.1
#9 4.136 WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
#9 4.136 You should consider upgrading via the '/usr/local/bin/python3 -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c python3 -m pip install -e .]: exit code: 1
make: *** [build] Error 1

[adefossez]

Hello @zeddan, sadly Github doesn't provide machines with macos 12 for the automated build of the DiffQ package. The same is true for lameenc.

It should be possible to build those packages locally but I'm not sure how to do it with Docker.

[xserrat]

Hello @zeddan , thank you so much for reporting this issue. As @adefossez commented there's no way right now to test the build step with MacOS 12 with GitHub Actions. It's true that they'll support the new version soon but now it's on beta (see: github/roadmap#240).

I'll try to install MacOS 12 on my local machine and I've created this issue #15 to add a Github Action to test the execution of the Makefile on different platforms. It's in progress but I hope I'll add a pull request soon.

[adefossez]

@zeddan thanks to this issues from Mac M1 users (facebookresearch/demucs#246) I realized I had forgotten to upload source distributions for diffq, only precompiled once. Could you give it one more go now ? I think it should work as long as a compiler is available in the docker image.
Thanks :)

[xserrat]

Hi @adefossez @zeddan , I've just upgraded my laptop to last version MacOS 12 and the process worked as expected. I've got only this error: killed due to the Docker Desktop virtual needed more RAM resources to execute the whole process. Now I've set 8gb of RAM and 2Gb for Swap.

I've created a repository in Docker Hub to store the docker image there instead of building it every time in the local environment. I hope that this change will improve the usage of the Makefile and reduce the issues related to the building process. Checkout https://hub.docker.com/r/xserrat/facebook-demucs/tags.

I'm going to create a pull request to update the docker image used in the Makefile. Feel free to suggest any improvement! :)

Thanks so much!!

[zeddan]

Hi @adefossez @xserrat

Thanks for looking into this issue!

I re-cloned the repository and tried again, and this time it succeeds on the diffq package but still seems to yield an error on the lameenc step:

docker build -t facebook/demucs:latest .
[+] Building 8.5s (10/12)
 => [internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 906B                                                                           0.0s
 => [internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.8                                                  1.9s
 => [auth] library/python:pull token for registry-1.docker.io                                                  0.0s
 => [1/8] FROM docker.io/library/python:3.8@sha256:881e0df149c29af8b29a973a9e80814dae6cddf123fe38a0bcac71864c  0.0s
 => CACHED [2/8] RUN apt install git                                                                           0.0s
 => CACHED [3/8] RUN mkdir -p /lib/demucs                                                                      0.0s
 => CACHED [4/8] WORKDIR /lib/demucs                                                                           0.0s
 => CACHED [5/8] RUN git clone -b main --single-branch https://github.com/facebookresearch/demucs /lib/demucs  0.0s
 => ERROR [6/8] RUN python3 -m pip install -e .                                                                6.5s
------
 > [6/8] RUN python3 -m pip install -e .:
#10 0.806 Obtaining file:///lib/demucs
#10 1.320 Collecting dora-search
#10 1.748   Downloading dora_search-0.1.7.tar.gz (68 kB)
#10 1.955   Installing build dependencies: started
#10 5.541   Installing build dependencies: finished with status 'done'
#10 5.543   Getting requirements to build wheel: started
#10 5.637   Getting requirements to build wheel: finished with status 'done'
#10 5.639     Preparing wheel metadata: started
#10 5.744     Preparing wheel metadata: finished with status 'done'
#10 5.848 Collecting diffq>=0.2.1
#10 5.919   Downloading diffq-0.2.2.tar.gz (44 kB)
#10 6.131 Collecting julius>=0.2.3
#10 6.220   Downloading julius-0.2.6.tar.gz (58 kB)
#10 6.463 ERROR: Could not find a version that satisfies the requirement lameenc>=1.2 (from demucs) (from versions: none)
#10 6.463 ERROR: No matching distribution found for lameenc>=1.2
#10 6.467 WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
#10 6.467 You should consider upgrading via the '/usr/local/bin/python3 -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c python3 -m pip install -e .]: exit code: 1
make: *** [build] Error 1

@xserrat you don't happen to know something else I can try, since you have got it working? I'm running on a fresh install of MacOS 12 so it's possible that I'm missing some configuration.

[xserrat]

Hi @zeddan ,

Could you try to run the following command to build the image from scratch without cache:

docker build --no-cache -t facebook/demucs:latest .

Then, execute again the make run track=xxxx.

If the error persists, I've just merged this PR #19 related to the issue #17 to prevent possible errors when building the image. You can update the main branch and execute the make run track=xxxx again and it should work.

Let me know how it was!

[adefossez]

I think the author of lameenc and forgot to include the source distribution for lameenc. I'll open a bug upstream.

[adefossez]

An issue already exist: chrisstaite/lameenc#13

[xserrat]

I didn't realise it was related to the new Mac M1 😄 🙏🏽 . Perfect @adefossez 🚀

[Reidsy]

In the meantime, as a workaround, I threw together a dockerfile that compiles lameenc while building the docker image.

https://gist.github.com/Reidsy/07db8e4d484d61f84408cab98fc24965

[xserrat]

Hi @zeddan,

Could you try to build the image again with the latest changes that support ARM64 as well as other improvements (#24)?

Just to make sure we can close this issue.

Thanks!

[radostyle]

demucs on  main [?] ➜ make run track=myfile.mp3
[+] Building 1.8s (9/10)
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 1.19kB                                                                        0.0s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:11.8.0-base-ubuntu22.04                                0.5s
 => [1/7] FROM docker.io/nvidia/cuda:11.8.0-base-ubuntu22.04@sha256:3a85383782012c87cd78f9b47c33b6105b2b6e17  0.0s
 => CACHED [4/7] WORKDIR /lib/demucs                                                                          0.0s
 => CACHED [5/7] RUN python3 -m pip install -e . --no-cache-dir                                               0.0s
 => ERROR [6/7] RUN python3 -m demucs -d cpu test.mp3                                                         1.2s
------
 > [6/7] RUN python3 -m demucs -d cpu test.mp3:
#9 0.989 Traceback (most recent call last):
#9 0.989   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
#9 0.989     return _run_code(code, main_globals, None,
#9 0.989   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
#9 0.989     exec(code, run_globals)
#9 0.989   File "/usr/lib/demucs/demucs/__main__.py", line 7, in <module>
#9 0.989     from .separate import main
#9 0.989   File "/usr/lib/demucs/demucs/separate.py", line 14, in <module>
#9 0.989     import torchaudio as ta
#9 0.989   File "/usr/local/lib/python3.10/dist-packages/torchaudio/__init__.py", line 1, in <module>
#9 0.989     from torchaudio import (  # noqa: F401
#9 0.989   File "/usr/local/lib/python3.10/dist-packages/torchaudio/_extension.py", line 135, in <module>
#9 0.989     _init_extension()
#9 0.989   File "/usr/local/lib/python3.10/dist-packages/torchaudio/_extension.py", line 105, in _init_extension
#9 0.989     _load_lib("libtorchaudio")
#9 0.989   File "/usr/local/lib/python3.10/dist-packages/torchaudio/_extension.py", line 52, in _load_lib
#9 0.989     torch.ops.load_library(path)
#9 0.989   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 643, in load_library
#9 0.990     ctypes.CDLL(path)
#9 0.990   File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
#9 0.990     self._handle = _dlopen(self._name, mode)
#9 0.990 OSError: /usr/local/lib/python3.10/dist-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK3c
104impl13OperatorEntry24assertSignatureIsCorrectENS0_12CppSignatureEb
------
executor failed running [/bin/sh -c python3 -m demucs -d cpu test.mp3]: exit code: 1
make: *** [build] Error 1

Fails like this on my Apple M1 Pro OS 12.4
Latest commit a776015

[xserrat]

@radostyle thanks for reporting this issue. It seems to be related with an incompatibility with the version of torchaudio that demucs is using. I'll investigate in more detail 🙏🏼

[radostyle]

I'm excited to try it out. Let me know if there is something to test.

[xserrat]

Hi @radostyle , sorry for my delay replying about this issue.

I've just merged the fix #37 that should solve the issue.

Could you try it again using the version v1.0.0 of this repository?

Thanks in advance.

[xserrat]

Closing issue after some time waiting for response. I assume it worked.