Numerous errors during Spades running

[DenisGoryunov]

Description of bug

Hello,
There are two errors and one warning in the attached Spades log file.

1. ```mimalloc: error: thread 0x7f294b7af640: unable to allocate OS memory (67108864 bytes, error code: 12 [Cannot allocate memory], address: (nil), large only: 0, allow large: 1)
mimalloc: process [info:](url)
   elapsed:  160563.809 s
   process: user: 1224599.957 s, system: 14503.077 s, faults: 26306211
       rss: current: 579801792512, peak: 480573063168
    commit: current: 579801792512, peak: 579801792512

The above error is clearly related to the lack of required RAM. Is there a way from the log file i already have to figure out an approximate RAM amount i need for the successful Spades run? Besides, i'm wondering what is the actual peak of memory 480573063168 or 579801792512? Can you recommend a tool to check RAM consumption during Spades run or the information which is already present in the log file is sufficient for that?

2. Why i can not to save the check point and got the warning:
   * 0:00:00.028     1M / 18M   WARN   StageManager             (stage.cpp                 : 162)   No saved checkpoint

3.  * 1:35:39.164  9053M / 9053M ERROR   General                 (kmer_splitter.hpp         : 151)   I/O error! Incomplete write! Reason: No space left on device. Error code: 28

There is approximately 1.5 Tb of free space on the mounted file storage i use on my virtual machine. What is the possible reason for that error message and how to fix it?

Many thanks,
Denis

spades.log

spades_out-5.txt

params.txt

params-3.txt

SPAdes version

4.0.0

Operating System

Ubuntu 22.04.3 LTS

Python Version

3.13.1

Method of SPAdes installation

conda

No errors reported in spades.log

• Yes

[asl]

Hello

Is there a way from the log file i already have to figure out an approximate RAM amount i need for the successful Spades run?

The amount of RAM required depends on many factors including (but not limited to) genome size, repeat content & complexity, ploidy & strain contents, error rate, etc. It is not possible to estimate this in advance. And it is not possible to estimate the amount of RAM required in advance as well.

Besides, i'm wondering what is the actual peak of memory

The peak resident set size was approx 440 before memory allocation failed. And your OS failed to fulfil SPAdes request to allocate 64M more.

Can you recommend a tool to check RAM consumption during Spades run

There are multiple tools available around for measuring memory consumption including e.g. top, htop, atop, etc. You may always consult with your system administrator what is available on a particular system. SPAdes-internal monitoring provides SPAdes-internal way, but certainly does not account for e.g. OS-imposed overheads, etc.

Why i can not to save the check point and got the warning:

Checkpoints are disabled by default. Refer to manual for description of checkpoint-related options.

I/O error! Incomplete write! Reason: No space left on device. Error code: 28
There is approximately 1.5 Tb of free space on the mounted file storage i use on my virtual machine. What is the possible reason for that error message and how to fix it?

Well, the error is clear: write failed and this was the error message provided by your system. If you're suspecting some system misconfiguration, you might want to ask your system administrator.

[ronald50928]

I want to highlight how difficult this issue is for Sys Admins and Cloud Engineers to diagnose and resolve for our genomics experts. When SPAdes fails due to a supposed memory issue, the OS does not register a RAM problem- meaning tools like htop, top, and atop provide no useful insight. This makes it nearly impossible to troubleshoot effectively.

Telling scientists to “check with your sysadmin” isn’t a viable solution when the OS, even Hypervisor sees no RAM issue, and yet, SPAdes dies even when there’s 1.1TiB of available memory left. On our 2TiB cloud instances(x2idn.32xlarge), SPAdes is failing at 800GiB usage, with only a RAM OS error and no clear debugging path as discussed in this discussion and this issue: ##1449

As developers, there should be a way to expose more meaningful error messages or diagnostics to help guide resolution. Without this, we’re left in the dark, unable to differentiate between a true system-level issue and a SPAdes-specific memory handling problem.

I’m eager to open another ticket or discussion, but the behavior and previous responses are very similar and haven’t provided a practical way forward for us. Right now, debugging SPAdes memory issues feels like trying to run a game in 4K without knowing the actual system requirements for SPAdes. Imagine launching a game, and instead of telling you that your GPU isn’t powerful enough, just crashes with a vague error message without even making the GPU do any significant work. That’s exactly what’s happening here... SPAdes fails, but the OS still reports plenty of available RAM or no errors, leaving us with no clear way to diagnose the issue.

Would really appreciate a more actionable approach to troubleshooting this. Looking forward to any insights.

[asl]

#1449 Contains no error and there is no issue. As practice shows, the majority of cases are caused by sine external problems ("disk quota exceeded", "i/o problem on the used disk system resulted in incomplete write", user set up low memory limit, user failed to read the error message). Note that from the developer perspective there is not much that could be done or diagnosed when e.g. memory allocation failed – it is the error returned by the operating system without any further information. So the only viable solution is to report the error back to the user. As there is variety of different system and different things that might go wrong, there is no universal way how the things might be troubleshooted. In addition, practice shown that many systems are not configured properly wrt large memory consumption and as a result there is no way to utilize all available memory at all (see #871 as an example).

And indeed, one cannot predict the required memory consumption ("system requirements") from the input data alone. As this depends on many factors unknown before the assembly. This is the specifics of the problem being solved and one needs to accept this. This is in contract with games where the input is effectively fixed ("games assets"), different settings are limited and one can definitely tune & estimate everything with respect to this input.

Finally, note that the developers are doing all this including the support in their spare time. This is in contrast with local system administrators that are paid to maintain the corresponding HPC system.

[ronald50928]

Thanks, we tried that configuration #871, and also did THP(transparent huge pages), Linux overcommit mode to 2 (vm.overcommit_memory=2) and configurations I used to do for other workloads of higher memory intensive use. None of these have worked for us. We will try re-deploying from scratch all of our components and installs of Spades. I do hope is our error, OS, or anything else and not SPAdes.

I appreciate you and all the team's contributions to SPAdes and I understand that open-source work is majority if not all done in personal time. Regarding the discouraging comment from your part This is in contrast with local system administrators that are paid to maintain the corresponding HPC system.- Being part of an open-source community is a choice, and if doing this for free is a frustration or time consuming for you, it may be worth seeking opportunities where your skills are compensated. You should not throw this as a comment to close an argument or in any way to someone you have no idea what their work stand is. Feels a personal comment, unprofessional and not related at all to the issue at hand.

The goal here is to improve SPAdes, and my feedback comes from real-world workloads that could benefit the project you decided to maintain.

[asl]

Indeed this is an open-source project and patches are always welcome. Source code is open, feel free to submit pull requests with the changes and we will do our best to review them.

[DenisGoryunov]

Hi,
I appreciate your detailed response. May i clarify a few points, please?

The peak resident set size was approx 440 before memory allocation failed. And your OS failed to fulfil SPAdes request to allocate 64M more.

1. Could you indicate, please, where in my log file i can find this number? I need that for future debugging.

Well, the error is clear: write failed and this was the error message provided by your system. If you're suspecting some system misconfiguration, you might want to ask your system administrator.

2. Can i get information regarding both the exact free disc space which Spades was looking for during the run and the maximal size of Spades temporal files? I will need that information for discussing with the system administrator.

Regards,
Denis

[bernt-matthias]

I also ran into I/O error! Incomplete write! Reason: No space left on device.. I think the missing information is where (e.g. a path) this problem occurred.

[asl]

@bernt-matthias The only path is your output directory (and scratch if you specified it separately). SPAdes does not write to any other paths except ones explicitly provided by a user.

[bernt-matthias]

Thanks for the prompt answer. Indeed I have --tmp-dir specified. In this case the output of the path where the problem occurred might be helpful.