ERROR: Failed building wheel for fasttext

[daxida]

The commands I ran:

gh repo clone Sirozha1337/faster-auto-subtitle
cd faster-auto-subtitle/
python3 -m venv venv
source venv/bin/activate
pip install wheel
pip install -e .

Here you can see the complete result of that command:
https://pastebin.com/nqUfJhW1

Tested in Ubuntu 23.10.
Version of the venv: Python 3.11.6

I haven't tested the suggested 3.9 yet. I'm sorry if this ends up being the issue.

[Sirozha1337]

Hi, it seems the problem is with the dependency package "fasttext". Can you try installing it separately by using this command:
pip install fasttext
Then install the repo using:
pip install -e .

If it fails, try installing the fasttext package differently:
pip install fasttext-wheel

Tell me if it works for you.

[daxida]

pip install fasttext fails with the same error.

pip install fasttext-wheel succeeds, but then pip install -e . runs into the same error.

[daxida]

It might be related to a bit with easynmt: UKPLab/EasyNMT#89

Just to be sure, what version of python are you using, if you don't mind me asking?

Also, I assume there is no opting out of easynmt right?

[Sirozha1337]

Just to be sure, what version of python are you using, if you don't mind me asking?

I'm using Python 3.9.1 on Windows 11. I'll try installing the package on docker using 3.11.

Also, I assume there is no opting out of easynmt right?

Yes, EasyNMT is required for translation.

[Sirozha1337]

The problem lies in the fastText package which is a dependency of EasyNMT. It requires usage of binary library and doesn't have pre-built binaries for each Python version and OS. So it tries to build one from source but fails, because your system has a newer compiler version which works a bit differently causing errors.

1. You need to install previous gcc and g++ version using this command:
   apt install gcc-12 g++-12

2. Then you need to set this version as default one:
   update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 1

3. Then install the fasttext package:
   pip install fasttext

4. Then you can finally install the thing you wanted to install:
   pip install -e .

5. Then you can revert the version back using this command:
   update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 1

6. Remove unneeded compiler:
   apt remove gcc-12 g++-12

[Sirozha1337]

@daxida let me know if it worked for you.

[daxida]

Sorry for the late response.

Unfortunately it didn't work for me:

https://pastebin.com/KzJfvaA3

I did run a Dockerfile where it seemed to work, so I'm unsure where the error is coming from.

Edit: For context

(venv) rafa@rafapc:~/dev/faster-auto-subtitle(main)$ python3 --version
Python 3.11.6
(venv) rafa@rafapc:~/dev/faster-auto-subtitle(main)$ gcc --version
gcc (Ubuntu 12.3.0-9ubuntu2) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Edit: The update-alternatives seems to only work for gcc but not g++

Works

rafa@rafapc:/usr/bin$ ls | grep gcc
c89-gcc
c99-gcc
gcc
gcc-12
gcc-13
gcc-ar
gcc-ar-12
gcc-ar-13
gcc-nm
gcc-nm-12
gcc-nm-13
gcc-ranlib
gcc-ranlib-12
gcc-ranlib-13
x86_64-linux-gnu-gcc
x86_64-linux-gnu-gcc-12
x86_64-linux-gnu-gcc-13
x86_64-linux-gnu-gcc-ar
x86_64-linux-gnu-gcc-ar-12
x86_64-linux-gnu-gcc-ar-13
x86_64-linux-gnu-gcc-nm
x86_64-linux-gnu-gcc-nm-12
x86_64-linux-gnu-gcc-nm-13
x86_64-linux-gnu-gcc-ranlib
x86_64-linux-gnu-gcc-ranlib-12
x86_64-linux-gnu-gcc-ranlib-13
rafa@rafapc:/usr/bin$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 1
rafa@rafapc:/usr/bin$ gcc --version
gcc (Ubuntu 12.3.0-9ubuntu2) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Doesn't work (Still 13.2)

rafa@rafapc:/usr/bin$ ls | grep g++
g++
g++-12
g++-13
x86_64-linux-gnu-g++
x86_64-linux-gnu-g++-12
x86_64-linux-gnu-g++-13
rafa@rafapc:/usr/bin$ sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 1
rafa@rafapc:/usr/bin$ g++ --version
g++ (Ubuntu 13.2.0-4ubuntu3) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I'm not confident enough to identify what is happening.

[Sirozha1337]

Strange, judging by your log it calls x86_64-linux-gnu-gcc instead of gcc.

It is possible you need to do this:
update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/gcc-12 1

Then do this:
x86_64-linux-gnu-gcc --version
And you should see 12.3.0 version in the output. Then you can try reinstalling the package again, it should use the correct version now.

I don't have much experience with gcc and make, so I'm not sure how it determines which compiler to choose.

[daxida]

Now it does recognize the change but it still won't compile... because the compiler is unsupported.

The key line being (I assume) this one:

x86_64-linux-gnu-gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory

https://pastebin.com/imhHNFqL

Tough times.

[Sirozha1337]

I couldn't reproduce it using Docker containers, so I don't know how to help you.

The only fix I see is uninstalling all the gcc and g++ versions and then installing older versions (12). But I'm not sure if that would work and it might break something else.

If you don't need translation to languages other than English, you can of course modify source code to remove dependency on EasyNMT or download an older commit.

There's also the possibility of running the package inside the Docker container. I'll add the Dockerfile to the repository soon.

[daxida]

I fixed it but I'm not sure how to reproduce :D

As you can see, I have no alternatives anymore (I cleaned them manually via update-alternatives --remove-all <bin>), the version is 13 yet it compiles.

https://pastebin.com/rpk93HK3

Now what I did just before (when I first realized that it worked, although since I can't reproduce it, it may be because I changed some other symlink that I can't identify) was just:

# Clean install
sudo apt remove gcc gcc-12 gcc-13
sudo apt install gcc g++

# I don't need this anymore. Now a clean reinstall works fine for me (and that's why I can't reproduce it).

# Download the 12 version
sudo apt install gcc-12 g++-12

# Update the priorities
sudo update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/x86_64-linux-gnu-gcc-12 1
sudo update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/x86_64-linux-gnu-g++-12 1

Again, I'm unsure as to why it works now but it does.

Also, I intended to use the version without EasyNMT from the start but since I got that bug (that seems to be quite common looking at fasttext / EasyNMT issues), I figured out I might as well just open an issue here too.

[Sirozha1337]

@daxida great, I'm glad it worked out for you!

Judging by your log, it used a cache version of fasttext:
Using cached fasttext-0.9.2-cp311-cp311-linux_x86_64.whl

So, it's possible you've built it before when you had gcc-12 installed.

[daxida]

Oh that makes sense. Do you happen to know how can I get rid of a cached wheel? If I can delete I might be able to reproduce what fixed it.

[Sirozha1337]

I believe pip cache remove fasttext should help. You can also try removing your venv and creating a new one.

[daxida]

But I did remove my venv last time. It shows in the first two lines of this pastebin

As you can see, I have no alternatives anymore (I cleaned them manually via update-alternatives --remove-all ), the version is 13 yet it compiles.

https://pastebin.com/rpk93HK3

And also, are we sure that this cached version

Using cached fasttext-0.9.2-cp311-cp311-linux_x86_64.whl

doesn't just refer to the cached wheel (the uncompiled zip) that pip has stored in whatever repository it goes to fetch it?

[Sirozha1337]

I believe pip stores global cache to speed up package installation in virtual environments. So you probably need to both clear cache and delete your virtual env.

doesn't just refer to the cached wheel (the uncompiled zip) that pip has stored in whatever repository it goes to fetch it?

Yes, it does mean cached wheel. No, cached wheel is not the uncompiled zip. wheel is a pre-built binary, that was created by running gcc/g++ on your machine. Usually those are created by package maintainers, however, fasttext repository is no longer maintained. Hence no wheels and no support for newer gcc versions.