Comet "segfault-like" error when running test pipeline under WSL2 #270
Comments
|
@radusuciu How much memory did you allocate in Docker? |
|
I did not explicitly configure a system-wide memory limit for docker and to my knowledge there shouldn't be a limit (other than system memory) by default -- I did not approach the system memory limit, at least as reported by Task Manager. I did try adding a I should also note that I've verified that comet itself runs fine on WSL2. I have not however tested CometAdapter by itself (ie. not as a part of quantms) |
|
Sorry to bother you. Can you try conda? |
|
Or did you try running comet from docker inside WSL2 e.g. by pulling and starting its biocontainer or openms-thirdparty biocontainer? |
Not a bother at all - happy to provide more information! Just a few notes on the conda installation process starting from a fresh install of conda:
After those steps, I was able to run the following command:
I do this all the time but not with the biocontainer or openms-thirdparty container -- I will try this as well. |
|
Thanks for trying. Regarding conda, openms should come with it as well (in the bioconda channel). There probably was something wrong during installation of the packages. To clarify on docker: If you use the docker profile, it will pull the openms-thirdparty biocontainer, which is based on the bioconda builds for every step that involves openms or an openms adapter. It includes both openms and most third-party tools. |
|
So I went on a quick debugging journey which you can follow below. My conclusion is that My debugging journeyAlright, now we're getting closer. I can reproduce the issue when running the $ docker run -it --rm -v $PWD:/data quay.io/biocontainers/openms-thirdparty:2.9.1--h9ee0642_1 comet -P/data/comet.params.new -D/data/18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta /data/BSA1_F1.mzMLand in the host Let's do some sanity checks and try some stuff 1. Download latest comet.linux.exe binary from the comet github page, rename to comet and run in WSL:$ ./comet -Pcomet.params.new -D18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta BSA1_F1.mzML
Comet version "2023.01 rev. 2 (7c9150d)"
Search start: 05/11/2023, 02:44:48 PM
- Input file: BSA1_F1.mzML
- Load spectra: 481
- Search progress: 100%
- Post analysis: done
Search end: 05/11/2023, 02:44:51 PM, 0m:3sOkay, that works well enough. 2. Let's do the same, but in a Ubuntu container$ docker run -it --rm -v $PWD:/data ubuntu /data/comet -P/data/comet.params.new -D/data/18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta /data/BSA1_F1.mzML
Comet version "2023.01 rev. 2 (7c9150d)"
Search start: 05/11/2023, 09:48:40 PM
- Input file: /data/BSA1_F1.mzML
- Load spectra: 481
- Search progress: 100%
- Post analysis: done
Search end: 05/11/2023, 09:48:43 PM, 0m:3sOkay, so that works too.. I also tried different versions of Ubuntu and those all worked too. 3. Alright, let's swap in the OpenMS comet binary from the OpenMS/THIRDPARTY repoLet's confirm it's based on the same comet release. $ wget -q https://github.com/OpenMS/THIRDPARTY/raw/master/Linux/64bit/Comet/comet.exe -O comet_openms && ./comet_openms
Comet version 2023.01 rev. 2 (7c9150d)okay good, now let's run it locally: $ ./comet_openms -Pcomet.params.new -D18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta BSA1_F1.mzML
Comet version "2023.01 rev. 2 (7c9150d)"
Search start: 05/11/2023, 03:00:57 PM
- Input file: BSA1_F1.mzML
- Load spectra: 481
- Search progress: 100%
- Post analysis: done
Search end: 05/11/2023, 03:00:59 PM, 0m:2sokay, that works, now lets run the openms binary inside an ubuntu container as we did before: $ docker run -it --rm -v $PWD:/data ubuntu /data/comet_openms -P/data/comet.params.new -D/data/18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta /data/BSA1_F1.mzML
Comet version "2023.01 rev. 2 (7c9150d)"
Search start: 05/11/2023, 10:01:49 PM
- Input file: /data/BSA1_F1.mzML
- Load spectra: 481
- Search progress: 100%
- Post analysis: done
Search end: 05/11/2023, 10:01:52 PM, 0m:3sso that works just fine... however. I noticed that it's not the same binary, so let's copy it from the container directly: $ docker cp fe2b26f4027e:/usr/local/bin/comet comet_openms_container
Successfully copied 11.1MB to /mnt/d/processing/quantms/comet_openms_containerand then run it in WSL2 directly: $ ./comet_openms_container -Pcomet.params.new -D18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta BSA1_F1.mzML
Segmentation faultand in an ubuntu container: $ docker run -it --rm -v $PWD:/data ubuntu /data/comet_openms_container -P/data/comet.params.new -D/data/18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta /data/BSA1_F1.mzMLno output, and the same failed 4. Let's try a bunch of different comet binaries, forgetting about docker since it doesn't seem to be relevant.They all run fine (once we use a indexed BSA_F1.mzML for versions that require it) so it seems that the issue is with the binary being distributed with the |
|
Cool. That is helpful. Specifically with the comment You could try that. Would be better to fix this during the bioconda build but not sure if it requires changes in the bioconda buildsystem/pinned versions of GCC/glibc or in the comet recipe or the comet code itself. |
|
Yep, the comment you linked makes it work, specifically, adding: to $ docker run -it --rm -v $PWD:/data quay.io/biocontainers/openms-thirdparty:2.9.1--h9ee0642_1 comet -P/data/comet.params.new -D/data/18Protein_SoCe_Tr_detergents_trace_target_decoy.fasta /data/BSA1_F1_indexed.mzML
Comet version "2023.01 rev. 0"
Search start: 05/11/2023, 11:04:47 PM
- Input file: /data/BSA1_F1_indexed.mzML
- Load spectra: 481
- Search progress: 100%
- Post analysis: done
Search end: 05/11/2023, 11:04:49 PM, 0m:2sand |
|
Very nice. I always wanted to try myself as well on WSL. Thanks a lot. We could add a note in the "Run" section of the docs or a new "Known issues" section of the docs/changelog. We could also forward to bioconda (@ypriverol often does updates of conda recipes). From the short look in the linked issues, I have the feeling this could be avoided by not statically linking to glibc in the comet bioconda recipe, but I am not sure. |
|
Also to the readthedocs section known issues, we should update it. |
|
Stale issue, close for now, please reopen if needed. |
Description of the bug
I hit this error after installing the latest version of nextflow and running the
test_lfq,dockerprofile:Standard error: Process '/usr/local/bin/comet.exe' crashed hard (segfault-like). Please check the log.The log is not really all that more informative unfortunately.I suspect this error may be related to running under WSL2, as @lazear has experienced something similar with the mhcquant pipeline. I haven't had time to test in a different environment, but was wondering if others have experienced similar issues and if there are any workarounds.
Command used and terminal output
Log
System information
Running on a Windows 10 workstation in a reasonably fresh Ubuntu 22.04 WSL2 environment.
Nextflow 23.04.1 build 5866
Container engine: Docker 23.0.5, build bc4487
WSL version: 1.1.3.0
Kernel version: 5.15.90.1
The text was updated successfully, but these errors were encountered: