Docker internal changes

[sergiolaverde0]

Hi, I have created this to have a place where to discuss the changes I contribute to, since we have been misusing the issue tracker for that despite many of our exchanges not being proper bug reports.

For the upcoming 0.7 release, my tasks so far are:

• Finish the docker versioning changes, which should be fairly simple once I get down to it Done at feat: change the action from build on commit to main to build on release #124.
• Remove the JavaScript unneeded code from the final image Done at feat: reduce image size using a multi stage build #105.
• Remove possible unneeded Python code Not possible because of external dependencies.

Should anything more be proposed for 0.7, it goes here, and if something happens internally without affecting the end users, it goes here too and not in the issues since that would not be a bug but a part of the development process.

[simjanos-dev]

I think this is a great idea. Thank you! Should we close #59?

I may have found something. Docker desktop says, that this layer is 978.22 MB.

Command:

RUN /bin/sh -c python3 -m spacy download ja_core_news_sm && python3 -m spacy download ko_core_news_sm && python3 -m spacy download de_core_news_sm && python3 -m spacy download nb_core_news_sm && python3 -m spacy download es_core_news_sm && python3 -m spacy download zh_core_web_sm && python3 -m spacy download nl_core_news_sm && python3 -m spacy download fi_core_news_sm && python3 -m spacy download fr_core_news_sm && python3 -m spacy download it_core_news_sm && python3 -m spacy download ru_core_news_sm && python3 -m spacy download sv_core_news_sm && python3 -m spacy download uk_core_news_sm && python3 -m spacy download xx_ent_wiki_sm # buildkit

However, I looked at all the packages with dependencies, based on the spacy documentation's Dependencies section, and it does not add up.

ja_core_news_sm 11MB
ko_core_news_sm 13MB
de_core_news_sm 13MB
nb_core_news_sm 11MB
es_core_news_sm 12MB
zh_core_web_sm 46MB
nl_core_news_sm 12MB
fi_core_news_sm 13MB
fr_core_news_sm 15MB
it_core_news_sm 12MB
ru_core_news_sm 14MB
sv_core_news_sm 12MB
uk_core_news_sm 14MB
xx_ent_wiki_sm 10MB

SudachiPy ~3MB
mecab-ko ~1MB
mecab-ko-dic ~33MB
natto-py ~1MB
pymorphy3 ~1MB
pymorphy3-dicts-uk ~9MB
jieba ~20MB
spacy-pkuseg ~3MB

All added together: 279MB

Is it 3 times larger because the download size is compressed, or is there something to save? I found this, maybe it would save some space.

[simjanos-dev]

I have found a way to check if a language is installed, get a return value of 0 if it is and of 1 if it is not.
So we should have 3 functions in python: install, uninstall all and is language installed.

If it's less than 5MB, I think we can keep them in the docker image. I didn't know how large they are. Manga OCR will be large probably later on, so we will still need a system for it.

It is fine if you want to take it easier. It is Sunday and not even half of the month when our release is planned to the end of it.

Actually, I've been testing it for a few days, and found no problem. Things missing from v0.7 are:

• Merging PR from yesterday.
• Reworking readme a bit.
• Updating the overview website.
• Version numbers for docker.
• I may still add the website import if I'll feel like it.

I'll definitely slow down a bit. If I didn't make linguacafe public, It would have taken me over half a year to add all these stuff instead of a month.

[sergiolaverde0]

What setup do you use to test the backend changes? I have been thinking of doing the changes on the python container but I'm not very familiar with backend and the container does not quite expose any ports for me to route from to test it all works fine.

[simjanos-dev]

I wanted to write it into contributing.md, just haven't gotten to it yet.

I use the docker-compose-dev.yml in the dev branch. It's basically the old system that I had before you worked on it, without any copy command. I think you can create the same dev environment I use with this command:

docker compose -f ./docker-compose-dev.yml up -d --force-recreate

I forgot to remove debug mode from tokenizer.py, so the python service should reload and update if you modify the script.

I rewrote a lot of Laravel code on the current dev branch, it is not functioning 100% currently. I recommend use the main branch, or the commit on dev branch before I started working on the VocabularyController.

[sergiolaverde0]

Sorry, I didn't explain myself properly. I mean I am not sure how to test the API actually works because the Python container does not expose any ports so I don't know where to send the requests in the first place. For the Laravel one I could send GET requests to localhost:9191/path/to/be/tested and check I get the correct response, but for the Python container I'm not sure how to comunicate with it.

[simjanos-dev]

Oh, I understand it now.

When I write a python function, I just also write a laravel one, and use that to call the python function. You can find examples in TextBlock.php model's tokenizer function. You can reach other containers by using their name as domain name.

You could also expose the python port as well in the dev environment, and use some tool to make the calls directly. Or make them get calls temporarily, and open them in your browser.

For testing laravel url-s, I wrote a small tool to send requests, you can find it on the /dev url on the dev branch.

[simjanos-dev]

Maybe we could also merge the two containers together? The only benefit of having them separated is that Python does the majority of CPU intensive work, so it would be easy to have multiple python servers and one php. But I don't think it will ever have so many users that it will matter, and we can always separate them again.

[sergiolaverde0]

Yes, that is also a possibility. We could simply merge both dockerfiles into one and use the Python one as a step and then copy the binaries installed by pip into the final image (just like Node in the last PR) and then introduce changes as needed to ensure the process starts with the container (maybe adding it to entrypoint.sh?).

[simjanos-dev]

I will add these 2 libraries to python for v0.7:
pysub-parser
newspaper3k

They don't seem to take up a large space.

Edit: I was wrong, it did go up to 1.37 GB, I think it was 1.29GB before. I think it will be necessary to have an image where we allow enabling languages in a .env file, and only install missing python libraries at startup that the enabled languages need. Otherwise the python library will go over 2GB if I add other language models, other tokenizers for uncommon languages and OCR for manga reader.

It's impossible to keep the image size low, and add all the features at the same time. Will get even worse if generative AI gets better, text to speech is already seems to getting to an acceptable helpful level. I'm not sure what is the solution. I also don't think that I want to make users download and/or install these large libraries, if they only want the basic features.

(These are not important in the next few weeks/months, just some thoughts.)

[sergiolaverde0]

OCR will be a massive size increase, which reminds me of what other projects such as Stirling-PDF do by offering different images with either minimal or complete features, and sometimes points in between.

There is also the madness that is Rocker-2 which has like a dozen images built from another one, because some of them take over 4GB.

Manga OCR could be split together with the Japanese and it's 300MB of dependencies when the time comes.

None of this will be ready for 0.7, but is good to discuss it with time since is not an easy problem.

[sergiolaverde0]

On the topic of future possible features, I have been thinking if adding a RSS feed as a "book" where each entry in the feed is a "chapter" would be fitting since part of the recommended material are news feeds.

FreshRSS is less than 80MB compressed and is made in PHP so it makes for a good example.

[sergiolaverde0]

At some point some mad lad will ask to import a whole Calibre library, but I recommend against that for the foreseeable future. That would 100% need the queue, among other things, and maybe integration with calibre-web similar to the one with Anki.

[simjanos-dev]

I will try to keep the API-s and integrations somewhat limited. At a point it would get hard to maintain if I add support for every niche need.

I will add some form of mass import with the already existing import options though. I want to make myself a huge package of German or Spanish reading and listening material, and import it in LinguaCafe when it's more developed. But don't want store them imported while developing it this actively.

By the way, I will need to write a user manual. Importing large amount of text that you don't use is inadvisable, because whenever you save or delete a new phrase, it will has to be indexed in every book. I have over 700 phrases in Japanese, and over 4 books imported, and I've noticed some slowness when adding a new phrase (I've had a saving spinner for it, but removed it and do it in the background now).

[simjanos-dev]

I feel a tiny bit burned out, and changed my mind. I'll leave website and RSS import for v0.8.

[sergiolaverde0]

And that's fine. Current status on dev branch is something to be proud already, and I know people appreciate it.

[simjanos-dev]

At some point some mad lad will ask to import a whole Calibre library, but I recommend against that for the foreseeable future.

It really did happen. But I think it is a good idea, it would use a common API for book related software. I don't think I will add a mass import feature any time soon though.

[simjanos-dev]

Where should I create a "roadmap" for each update? I want to keep a list of the planned and finished changes for the next update.

I know about github projects, but there are changes that are not a separate github issue. Is it okay to do in the discussions section for each version?

[sergiolaverde0]

GitHub has something like called milestones (here an example) to keep track of the planned features for each tag however I'm not familiar with them.

I suggest to create them to discuss the details regarding the code itself and mark them as open or closed, but keep the discussion about things not directly related to code in the discussions section, such as what features to push on each version, potential release dates, distributing responsibilities between us and so on.

[simjanos-dev]

I think we should rename this and remove v0.7 from it, and just keep this discussion for any docker change in the future.

[simjanos-dev]

I may have asked this already, but I forgot and can't find it. Can I delete the docker-compose.yml file from the main branch? Someone tried to use it when v0.6 got released.

[sergiolaverde0]

Yes, it was superseded when we created the deploy branch.

[simjanos-dev]

There is an issue with the install process on Mac, it is not directly related to docker docker, but I thought I mention it here as well: #152. I'll try to find a fix for it.

I think we should also replace the Mac uncomment method with an environmental variable if possible.

I've been sick and working on a lot of refactoring in the last 2 weeks. I've just started working on features and bug fixes for the next update.

I'm planning on working on these:

• A user manual
• Word import from csv on the vocabulary page
• Website import
• RSS feed
• DeepL context
• Adding more DeepL target languages.
• Adding support for more dictionaries, including ones with non-english target languages.
• Smaller changes and bugfixes

I do not have any timeline for them. Also I may cut it into multiple updates.

@sergiolaverde0 Hope you are doing well!

[sergiolaverde0]

I've just remembered that before you started working on this issue I suggested adding a .env file, and make it auto install it at startup, and you said that it is not a good practice to install things inside the docker image after it was created and it slows down startup. But since we have decided doing it, what do you think of this solution:

We could create a bash file for installing languages, so the admin could just run it from the terminal after linguacafe is installed for the first time. It could still be stored in the storage directory, so it won't get deleted after a new image is pulled(I've realized we will also have to test this).

I somehow managed not to read this the first time. I think we can leverage our current solution by adding some environment variable(s) to store the languages that must be ensured to be installed always, then have the PHP container read it, compare it with the list returned by the Python container, and then install any missing languages one by one.

Shouldn't be too hard I think, reuses code and should be more robust than Bash.

[sergiolaverde0]

Done testing the five non-default languages.

Korean had issues the first time where the first chapter would be added but then the stats of the book would attempt to refresh seemingly forever, but after deleting the book and creating it all over again it worked perfectly fine, so I might attempt to reproduce it later to pin a possible cause. Korean was a bit underwhelming with its results since many tokens would not be available in the wiktionary file we provide and I suspect 제23조의 is not supposed to be a single token but somehow split but that is on the tokenizer and not Linguacafe.

Otherwise all five languages allowed me to create a book, import a couple of paragraphs of the Wikipedia page of the Universal Declaration of Human Rights, and check the definition and pronunciation of the tokens recognized. I also noticed I didn't know how to spell "Ukrainian" but I will amend the commit soon enough. Once that point gets addressed it will be ready for you to test.

While doing all that testing I found some small miscellaneous details about UI which you might, or might not, want to take into consideration. I will probably throw them all into a single issue despite not being really related to each other, but let me know if you prefer me doing it another way.

[simjanos-dev]

then have the PHP container read it, compare it with the list returned by the Python container, and then install any missing languages one by one.

I was suggesting this to make it simpler, and remove possible issues due to installing packages while the python script is running, or communication between many components, because I'm really worried about things going wrong. I think I would rather stick with the original UI plan than making it a background process with the php server.

Korean had issues the first time where the first chapter would be added but then the stats of the book would attempt to refresh seemingly forever, but after deleting the book and creating it all over again it worked perfectly fine

If you want to test if importing is working as intended, and you delete all books, I would suggest to also empty the unique_words table in the database for that language. It contains every unique word that has been imported, and if something fails for the first time, but works for the second, it can be a problem related to this (but unlikely).

Once that point gets addressed it will be ready for you to test.

I'll write the UI and will test it for a while, then I'll make a clean install and test it again. But my laptop is REALLY slow, so I would appreciate if you would also do a thorough final test after the UI is complete.

It is a critical part of the software, but it also won't cause large problems if it fails, because it would just return an error if someone tries to import something in a language that has not been installed yet. I don't think it can corrupt any data.

While doing all that testing I found some small miscellaneous details about UI which you might, or might not, want to take into consideration. I will probably throw them all into a single issue despite not being really related to each other, but let me know if you prefer me doing it another way.

It is fine in a single issue.

Thank you so much for working on this, I really appreciate it!

[sergiolaverde0]

I'm not sure I understood your proposal. I though you were asking for a script that would be copied inside one of the images, but:

remove possible issues due to installing packages while the python script is running

If the script is tied to the containers then it will run while the python script is running either way. Making it a separate bash script would only make it a separate process that runs parallel to the server which might actually make things worse since neither Python nor PHP would be informed of what is happening in said script.

I would rather stick with the original UI plan than making it a background process with the php server

And similarly, the bash script would be a background process all the same, so this would be as true.

So I think I misunderstood you and what you meant was a script to be shipped in the deploy branch instead of the actual images so that users could run it after cloning the branch and install the language models into the empty folders that were cloned too.

If that was indeed your proposal, then I don't think that is a good idea. If distributed this way the script would run on the host OS and any bug that happens would risk damaging the host instead of simply messing with a container that can easily repaired by restarting it; I'm certain people are tolerant to our little mishaps that erase their data because they only ever affect the software we provide and nothing else. Plus there is the issue of different systems having different versions of python/pip, and Windows support becoming even more spotty.

[simjanos-dev]

And similarly, the bash script would be a background process all the same, so this would be as true.

If the script is tied to the containers then it will run while the python script is running either way. Making it a separate bash script would only make it a separate process that runs parallel to the server which might actually make things worse since neither Python nor PHP would be informed of what is happening in said script.

I was asking for a script that would be inside the image, and your points are valid. You are right, I haven't thought this through.

[simjanos-dev]

Minor change related to docker image. Someone requested fonts for every language, so I'll write a font management system. I'll create a fonts directory inside the storage directory, and move the two default Japanese and Chinese fonts from the docker image to the deploy branch. Probably in v0.10.

[simjanos-dev]

Someone tried to update to v0.8 on MacOS without commenting the line, and they got this error on apple silicon processor:

I was surprised the git command worked. I've told them they have to uncomment the last line in docker-compose.yml. They did and run it again. After that the terminal did not answer for a while, then he got this error message:

I had no idea what it is. Someone in an other github project wrote that running it again fixed the issue. I told them to try it. After that linguacafe worked, he had his database deleted.

@sergiolaverde0 Any idea what could have happened? The only thing I can think of is that MySQL error. But I do not have any idea what exactly happened. Added an additional note to create a backup to the github release.

[sergiolaverde0]

I was surprised the git command worked

I tested it and it kept my changes after pulling, no warnings at all, since we haven't touched the file either so no conflict would happen. What I don't get is the error they got since MySQL does offer images for Arm64v8, and that same image should be pulled by all other Mac users, and be present already on their storage since it hasn't changed ever since we moved from 5.x months ago.

I had no idea what it is. Someone in an other github project wrote that running it again fixed the issue. I told them to try it. After that linguacafe worked, he had his database deleted.

My search pointed it to being an error on the git pull itself due to curl failing when reusing a connection, I assume when doing the second pull. Whatever happened to the database, this wasn't the cause.

Are we certain the database was deleted? I can understand it being corrupted, but the mounted volume shouldn't have its contents deleted. I would like to get a copy of the docker-compose the user had at the end of this odyssey, to check if there is anything wrong with it.

In any case this reminded you asked me to move away from the comment method and instead use a variable from the .env and I never did it, PR in a couple minutes.

[simjanos-dev]

Are we certain the database was deleted?

I'm not sure. He had no users in the database at least, because he was prompted to create the first user. He is not very familiar with the terminal, I didn't want to press for more details. He said he didn't have much data, so it's not a huge deal, but I still feel really bad.

I'm not sure what happened, I really don't think it was something with linguacafe, because I can't think of any significant change in the update that could cause this. 15 people updated the latest version, heard of no other problems.

In any case this reminded you asked me to move away from the comment method and instead use a variable from the .env and I never did it, PR in a couple minutes.

Thank you so much, I really appreciate it, especially making it so fast. This will certainly help some people. I also added a suggestion before I published it in large font on discord, reddit and the release notes to make a backup before updating.

(The chmod problem on mac was also solved. It was a problem with the argument order, for some reason Mac didn't support it.)

[simjanos-dev]

There was a comment thread below my reddit post about the docker process. I'm not sure what to think of it. I know the process is not conventional, but it is a simple step-by-step process to install and update.

I think it could be improved if we create the empty folders with the software after the storage directory was mounted. And copy the default files from inside the image to the mounted directory. After that it could be a simple docker-compose.yml file, and users would have to create the two directories, and it would be more similar to how other dockerized software usually works.

But I don't think we should do it. It can introduce problems, and I don't think there's anything wrong with the current install/update process itself, it's just different a little to what people are used to.

What do you think?

[sergiolaverde0]

DockerHub might provide better visibility, but not by much. I think the user last heard of the project back when you had to clone the repo and build the image yourself, hence the confusion.

I think it could be improved if we create the empty folders with the software after the storage directory was mounted

I tried my best at accomplishing this but failed to do so. The two issues we encountered are that users need root permissions to copy the dictionary files to a folder created by rootful docker (and rootless docker/podman have some rough edges) which is not ideal, and that Laravel can't write its logs to the mounted volume for the same reason since it runs as its own user instead of the user of the container. That's why we must do chmodon the whole thing.

I have had on mind asking you to (I am not sure I could do it myself) add a way to import dictionaries like we import epub books since that would solve the first issue as no files would have to be manually placed in the volume, but was waiting for development to settle down a bit more. For the second I looked at some selfhosted projects that use Laravel too and the only two I could find had similar issues with persistence.

I think achieving that would make deployment a breeze, and would enable Linguacafe (once it gets more stable) to be added to "stores" that offer single click installs of various selfhosted software like Cosmos and CasaOS since the current form is not very friendly to them.

And yet, I think there are more important points to address right now, as this smaller user base is easier to handle when we break something during development.

[simjanos-dev]

(Sorry, I'm a bit slow today, might overlook some simple stuff that is obvious.)

The two issues we encountered

I'm not sure if this is the exact same issue, but the temp folder also needs writing permissions, and it is shared between the two containers. Which also could be merged into one.

Cosmos and CasaOS

I've heard about the first, and was planning on adding linguacafe to it, but didn't have the time yet to look into it. I think it would be a good idea. It also sounds very useful for people not used to docker.

So, having a mounted storage is not an issue, the problem is that users should not copy files into it? I could replace it with a file upload. There will be also manga files(I think they can be several 100MB), other images files, audio and font files in the future which will be stored in the storage directory. Personally I prefer copying, but I think it is possible to make all these file upload based.

And yet, I think there are more important points to address right now, as this smaller user base is easier to handle when we break something during development.

I agree. I think it will take several years before linguacafe will be in a state where every feature people want will be implemented. I think I'll try focusing on the base reading experience and stability first.

By the way, speaking of stability. I recently I had a great idea: I think I should remove thousands of binded Vue events and classees (10 on every word) from the text reader, it might improve the performance a lot.

[sergiolaverde0]

I'm not sure if this is the exact same issue, but the temp folder also needs writing permissions

Ah right, this is a third issue I also forgot about the first time I touched the volumes and silently broke certain features. What a memory. Yes, it is also due to containers not being able to write into the volumes of a different one for security reasons.

So, having a mounted storage is not an issue, the problem is that users should not copy files into it?

Basically. An user that copies files needs permissions on the storage, an user that uploads files just needs permissions on the app.

There will be also manga files(I think they can be several 100MB)

I checked my own (translated) manga files and a chapter of is usually below 15Mb and almost always under 40Mb, while the one series I have compiled in tomes all are under 60Mb. Of course there can be lots of variation, but with this being selfhosted software I dare saying those sizes are reasonable.

[simjanos-dev]

When I go to the webserver package on github, I see this:

Is that third one supposed to be there?

[sergiolaverde0]

Seems to be a bug that got introduced at some point. Some of the older images were not affected, and it seems to be benign so no action from our side should be needed.

[simjanos-dev]

I was thinking about adding a backup function to the UI, since there are some users who have difficulties with the terminal, and I think even other people would be more inclined to click on a button compared to doing it manually.

I found that I can use mysqldump remotely:

mysqldump --host 127.0.0.1 --port 3307 the_database > some-host-file.sql

And then I think I could create a zip file from the .sql file and the storage folder. I think I could even write an import function for it.

This would require mysqldump command to be installed on the php container. I haven't checked yet how large is it. Do you think it is a good solution?

[sergiolaverde0]

I like the idea of dumping the database from the GUI since it does look a bit daunting, but I'm unsure about zipping together files that belong in different locations afterwards. You will need to be extremely clear on the instructions you provide for users to handle the import themselves since some might try to unzip all content in a single location and call it a day.

[simjanos-dev]

You will need to be extremely clear on the instructions you provide for users to handle the import themselves since some might try to unzip all content in a single location and call it a day.

I would make a UI for uploading the zip as well. I would unpack it, replace the storage folder, and import the .sql file. But I might just start with the database only, because it is much more important, easier, and currently the only problem with losing the storage folder is losing the book images, which can be corrected.

[sergiolaverde0]

Restoring the backup also introduces some edge cases you will need to take care of:

• If backup is imported after creating an account, handle the case where the user existing in the backup is different from the existing user; even more edge case where users share email but have different passwords. Alternatively, don't backup user details so that whatever "you" import is taken as "yours".
• You could allow a backup to be imported when you are prompted to create a new user when there are no existing users.

If the option to have multiple users is still planned to be added, we will need to implement this in a way that is compatible with them so that backwards compatibility is kept and refactoring is reduced to a minimum.

[simjanos-dev]

I was thinking about just doing a complete database export and import. So if someone restores the backup, all the current data would be lost. This would be just a UI for the process that is already described in the user manual.

I also do want to add a feature to export and import all of a single user's data in the future, but that will take a lot of work.

[sergiolaverde0]

Yes, sounds good.

[simjanos-dev]

This is so strange. There is quite a large difference between the php server image, and the python server image download counts.

I've heard no complains about anything docker related, so I don't think there's anything wrong.

[sergiolaverde0]

Oddly enough, for v0.7 it goes in one direction but for v0.8 and v0.9 it is the opposite. Even more, it also happens with the dev image.

Have there actually been changes to both images in all releases? Because if there was one where a container was identical to the previous release, docker is smart enough not to download it and just slaps the new tag on top of the old image. Then this creates a cascade effect.

[simjanos-dev]

I don't remember honestly, but I think in most of the releases they were both changed.

[simjanos-dev]

I've experimented a little with SQLite, but could not get it working. All the migrations are broken. SQLite uses different collations, which I will have to understand and test.

NativePHP seem to be progressing a bit. I won't have time to do it in the next year, but if it will work well, It could be used to package linguacafe as a standalone software by dropping languages that does not have space separated words, and a few features that use the python service.

[simjanos-dev]

It seems like the docker image download count is tripled compared to the usual after v0.12, which seemed to attract less new users and github visitors than usual. Any idea what could have happened? I don't think these are separate users who actually use linguacafe.

[sergiolaverde0]

Now that the install was reduced to a single compose file it is far easier for people to just try it for a couple of minutes even if they don't have any intention of using it long term. Specially the homelabbers that use portainer and couldn't use git before, a few of which were probably laying low waiting for this moment.

[simjanos-dev]

I'm not sure, it seems like a lot of people. But v0.12.4 also had 272 downloads, and v0.12.5 already has 75. If they really are actual users, then it is thanks to you, for fixing my terrible docker setup. :)

[simjanos-dev]

Do you think it would be a good idea to create a beta docker image for people who are proficient with docker enough? I usually manually test features, then I'll start using the new version myself for around a week, before I release it. But since updates will slow down a bit, and with the increased amount of users I want to test things for a longer time, I think it would be useful to have a beta image that would be a new version that I already use myself in my own prod environment.

This would also help with an other issue: When I finished a new version, I cannot work on new stuff that will go into the next version, because I do not have a branch for it. Using a beta branch for fixing bugs and problems in the version that is coming out soon, and using dev branch for developing the next version would be really useful. This could also be done without having a beta docker image.

[sergiolaverde0]

Yes I had thought of using an extra testing branch to merge dev into before each release so that there is more separation between what goes in the next release and what needs a bit more time; the image can be built with the same action you use to build the dev image and will be automatically tagged with the branch name you decide to use.

I'm not so certain how many actual users would use it, but it could be a good tool for the contributors that have gathered here.

[simjanos-dev]

Can you please make a copy of "Build test image" github action, that also works for silicon mac users? (Please take your time with it if you do it, it's not important or urgent.)

I'm not so certain how many actual users would use i

A few people on discord said that they would.

[sergiolaverde0]

Sure, let me know the exact name of the branch to target (or create it) and I will create an action that is identical to the one that builds the releases from main but with a different tag.

Edit: for more clarity, it will actually work like the ones that builds the dev images but will build for ARM too.

[simjanos-dev]

Thank you! I've created a beta branch.

Do you think I should I create beta releases somehow? I think I should just add a description of the beta process, and tell people to check the packages page to see if there's a new beta image, and I'll just leave a comment on discord.

[sergiolaverde0]

Hi, I have created the PR, the action tags images with their build date so that would be your "releases", the day they were made.

[simjanos-dev]

The latest test image build has failed. Do you have any idea what could have happened? It's just the basic language install/download command. I already had a successful build after the problematic changes were reverted the other day, and the dockerfile have not changed since that.

[simjanos-dev]

I'm a bit dumb, I just found the actual error message. The best thing I could find is that some people said that downgrading to an older python version fixed their issue.

[simjanos-dev]

I replaced the python image with python:3.12.3-slim. It is the version in the last dev image that worked, it works again. Is it good solution, or could it have problems? I think it's okay if we just update the image yearly.

[sergiolaverde0]

Pinning it to that version won't cause any issues, being able to do that is one of the main benefits of docker. The only downside is not getting the benefits of the most recent updates, but that might be fine given the alternative is having broken software.

[mateuszmrw]

Hello @sergiolaverde0,

From what I understood from discord discussion with the @simjanos-dev is that the python image currently does not provides arm image is due to japanese spacy model failing to built on the arm machines. The issue is due to package called sudachipy, sudachipy requires Rust and Cargo to present on the arm machines to be installed.

There are some extra steps required as RUN pip3 install --upgrade pip setuptools wheel, without upgrade the sudachipy package will fail with the different reason.

This is docker image that works for me, I removed the addition of user because of issues with cargo and path, but I believe it can be added back after some playing around with it. It can install japanese on the arm machines. I am pretty sure that users that are on arm and currently have japanese or other additional languages installed would need to re-install them. However, they could just stay on the linux/amd64 version if it's something they are not comfortable with.

FROM ubuntu:22.04

WORKDIR /app

RUN apt-get update -y \
    && apt-get install -y --no-install-recommends \
        python3 \
        python3-pip \
        build-essential \
        gfortran \
        libopenblas-dev \
        liblapack-dev \
        libxml2-dev \
        libxslt1-dev \
        zlib1g-dev \
        curl \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="/root/.cargo/bin:$PATH"

RUN pip3 install --upgrade pip setuptools wheel

RUN pip3 install -U --no-cache-dir \
        lxml[html_clean] \
        youtube_transcript_api \
        ebooklib \
        pykakasi \
        bottle \
        spacy \
        pinyin \
        pysub-parser \
        newspaper3k \
        numpy

RUN python3 -m spacy download de_core_news_sm \
        && python3 -m spacy download nb_core_news_sm \
        && python3 -m spacy download es_core_news_sm \
        && python3 -m spacy download nl_core_news_sm \
        && python3 -m spacy download fi_core_news_sm \
        && python3 -m spacy download fr_core_news_sm \
        && python3 -m spacy download it_core_news_sm \
        && python3 -m spacy download sv_core_news_sm \
        && python3 -m spacy download en_core_web_sm \
        && python3 -m spacy download el_core_news_sm \
        && python3 -m spacy download ca_core_news_sm \
        && python3 -m spacy download hr_core_news_sm \
        && python3 -m spacy download da_core_news_sm \
        && python3 -m spacy download lt_core_news_sm \
        && python3 -m spacy download mk_core_news_sm \
        && python3 -m spacy download pl_core_news_sm \
        && python3 -m spacy download pt_core_news_sm \
        && python3 -m spacy download ro_core_news_sm \
        && python3 -m spacy download sl_core_news_sm \
        && python3 -m spacy download xx_ent_wiki_sm

COPY ./tools /app

ENV PYTHONPATH="${HOME}/.local/bin:${PYTHONPATH}"

To test only japanese install instead of installing all spacy languages you can run only:

RUN python3 -m spacy download ja_core_news_sm

[sergiolaverde0]

This looks good!

I think either Korean or Chinese have some problematic package too, but they might be worked around with a similar approach.

I will be looking at it and at adding the user back later to prevent it from running as root, hopefully we just have to set CARGOHOME for that.

[sergiolaverde0]

Done at #315 but yet untested.

[simjanos-dev]

Laravel will need 2 additional processes to run for v0.14: one for websockets, and one for job queues. Laravel horizon is supposed to be set up with supervisor, and the websockets can be started with the php artisan websockets:serve command.

Is there anything important I should know? Supervisor seems straightforward although I got a permission error. I'll try to make it work next week. However I'm not sure where to start the php artisan websockets:serve command. I think it could be started in the background at the end of phpdockerfile with a RUN command, or inside the entrypoint (I'll have to look up what is the difference).

It's on the features/websockets-vuex branch.

And seriously, I really appreciate you debugging the arch linux issue, that was a lot of work.

[sergiolaverde0]

The way docker recommends to handle this is to have only one process per container, which makes sense for an enterprise deployment but not so much for home users.

We can probably pack all those processes in a single container, at the expense of reliability (having a process crash can get messy) and clarity on the logs.

I think s6-overlay (sorry, it's clunky to add links on the phone) makes handling this easier, as a docker-centered alternative for supervisor but I lack experience with it.

However, won't we need a Redis container too? I see you already added it.

And seriously, I really appreciate you debugging the arch linux issue, that was a lot of work.

Don't sweat it.

[simjanos-dev]

I think s6-overlay (sorry, it's clunky to add links on the phone) makes handling this easier, as a docker-centered alternative for supervisor but I lack experience with it.

Which one should I use? I wanted to do supervisor, because it is in the laravel docs, but it doesn't matter for me which one we use.

and clarity on the logs.

I didn't even think about that. The websocket service would produce constant unnecessary messages in the docker logs. However the horizon logs are very useful to see in the logs.

This docker part might get more difficult than I thought.

[sergiolaverde0]

Which one should I use?

Looking deeper, supervisor seems to have more resources available, and I think we can use it for the websocket too.

For the logs, we can probably prevent those from ever reaching stdout/stderr but still put them in a log file in case we need them, and ask users to attach it to their bug reports.

[simjanos-dev]

Could you please help me with starting supervisor?

I added the config files and modified the docker-compose-dev.yml file in the feature/websockets-vuex branch, and mounted them to the image. (latest commit)

A little help, because I think you don't use the dev environment a lot. This is the command I use for starting my server: docker compose -f ./docker-compose-dev.yml up -d --force-recreate. If you have it already, you will have to build it again. This will start npm: docker exec -ti linguacafe-webserver-dev npm run watch-poll, and you can reach the server from :3000 port.

And this is the command that I managed to start the servers with: docker exec -u root -ti linguacafe-webserver-dev service supervisor start. This will start both the job queue and the websockets processes. I am unsure where to put it in the docker files.

You can see if horizon is active or not on the /horizon url, and you can test websockets by importing a dictionary. If you get a working progress bar, the websockets server works.

If you make it start on the dev image, I can modify the main version of the files. Since it worked surprisingly quickly, I think I can manage to finish the last 2 small queue/websockets task by friday, and I can test/work on stanza on the weekend. But I may need a small break, so possibly next week.

[sergiolaverde0]

Just so that you get the news before going to bed: I got all the process to run but requires some degree of manual intervention. Horizon seems like it work like a charm to my eyes that never seen the dashboard before, but the dictionary import had no progress bar, it stayed at zero and then eventually was marked as successful; the log just had two lines repeated a different amount of times:

Starting the WebSocket server on port 6001
Saving statistics...

I will draft the PR once I tidy the commits a bit.

[simjanos-dev]

@sergiolaverde0

Could you please help setting up a cron job for laravel? I tried to do it with supervisor but could not get it working yet. I tried to do it based on this, however I got a permission error for writing into /etc/cron.d/my-cron while building the image.

* * * * * cd /var/www/html && php artisan schedule:run >> /dev/null 2>&1

If it works, it should create a backup every 4 minutes (it runs laravel scheduler every minute) in the storage/backup directory, but obviously I could not test it.

I'll try to do it later on the weekend again if you have no time for it.

[sergiolaverde0]

Hi, I have been looking into it, but I have struggled a bit because cron is made to be run as root and the process inside the container runs unprivileged since a few versions ago. I'm currently looking for alternatives.

[simjanos-dev]

Thank you!

That's the issue I had as well. I think anything would work, if it calls the Laravel command every minute.

Maybe a simple bash script that calls the Laravel command, does nothing for a minute, then exists, and then the supervisor would restart it? I know it's bad practice, but if cron is not possible, it would work as well.

[sergiolaverde0]

I didn't want to go down that route but StackOverflow offered no other option either, so I made it a bash script that waits for 60 seconds, runs the command, then exits.

However, nothing appeared at storage/backup and after making a change so that I get proper logs I saw the message INFO No scheduled commands are ready to run so I'm feeling a bit dumb since I think I forked off the wrong branch, which one should I have used instead of dev?

And as a side comment, I wonder if would be a good idea to have a single timer in the bash script itself, as the current model has two different timers, one of one minute on the script and the four minute one in the command it runs.

[simjanos-dev]

I probably just messed something up, and it is not on your side. If you open a PR ill fix it in a few days.

That is a good point. One timer would be better in this case as long as we have no plans on using other timed commands.

4 minutes was for testing only. I dont know what would be a good interval, probably daily.

[sergiolaverde0]

Done at #353. Side comment, Horizon currently expects the redis server to be end with -dev and fails to find it in my "production" mock up, so give that a look sometime before we start the serious testing.

[simjanos-dev]

Hi!

Can I, and should I create a release for beta on github? If yes, should I turn on the pre-release option, and can I set it as the latest release, or will it cause an issue with images?

How stable do you think a beta version should be? So far I planned to release it after using it myself for a few days. However nowadays I don't really read at all, and currently this is the only thing holding me back from releasing a beta. Do you think it's okay to just release it after manually testing every change a bit?

I haven't worked a lot on linguacafe lately and we didn't communicate much, I hope you are doing well nowadays.

[sergiolaverde0]

Hi,

The action for beta images is triggered manually so you shouldn't use the pre-release option that GitHub uses since I didn't plan around it and honestly am not completely sure how it works.

How stable do you think a beta version should be?

Some places just build one off whatever the last commit to dev was at the end of the day. But I think it should be stable enough that new issues won't be found in less than 15 minutes of use, or in other words, the more obvious and glaring issues should have been at least identified and have an open issue. So the beta image is released to locate the much less obvious ones, or the ones that your own knowledge as a dev is stopping you from noticing.

I'm doing well, thank you, hope you are too and that you still find time to do things that make you happy.

[simjanos-dev]

The action for beta images is triggered manually

I know. The reason I want to make a release for a beta is visibility.

[sergiolaverde0]

Announcing via Discord and the website is probably enough, I don't think there is many people following the GH releases exclusively.

[simjanos-dev]

I don't think there is many people following the GH releases exclusively.

I think most users don't follow neither, but it seems easier to find on GitHub. I've heard from multiple people who do not want to use Discord. There are also only 108 people on discord, and the latest hotfix update have thousands of downloads (which I'm still very skeptical about being true).

But I think it should be stable enough that new issues won't be found in less than 15 minutes of use, or in other words, the more obvious and glaring issues should have been at least identified and have an open issue. So the beta image is released to locate the much less obvious ones, or the ones that your own knowledge as a dev is stopping you from noticing.

I think that kind of fits. I did test everything, but I could only possibly find more if I started using it for long periods of time.