Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OCI index found, but accept header does not support OCI indexes #4

Closed
liuronghyw opened this issue Apr 13, 2023 · 17 comments
Closed

OCI index found, but accept header does not support OCI indexes #4

liuronghyw opened this issue Apr 13, 2023 · 17 comments
Labels
bug Something isn't working

Comments

@liuronghyw
Copy link

Description of the bug

Dear developers:
I can't pull the viroprofilter in cluster with UNIX operate system, the command as follows:

export NXF_SINGULARITY_CACHEDIR="./temp"

nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r v0.2 --db "/public/home/**/db/viroprofiler" -without-conda

The error is as follows:
image

Would you please give me some suggestions on how to solve this problem.

Command used and terminal output

nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r v0.2 --db "/public/home/hpc4011032/db/viroprofiler"

Relevant files

No response

System information

N E X T F L O W ~ version 22.10.6

HPC
local
Signalarity
CentOS Linux

viroprofiler v0.2
@liuronghyw liuronghyw added the bug Something isn't working label Apr 13, 2023
@rujinlong
Copy link
Collaborator

Hi @liuronghyw, could you try the following command and then report the output?

singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2

@liuronghyw
Copy link
Author

liuronghyw commented Apr 13, 2023 •
edited
Loading

Hi, Dr. long. I can pull the image with your command. Since I am not familiar with Nextflow. Would you please tell me how to bulid the pipeline with local image (denglan-viroprofilter-base-v0.2)？ Thank you very much! @rujinlong @deng-lab

@rujinlong
Copy link
Collaborator

I guess the original error occurred because I released the new version 10 hours ago, and you executed the Nextflow command simultaneously while the image was not yet successfully created. If you run the Nextflow command now, the error should not occur. Alternatively, you can pull images manually using the following commands, and then execute the Nextflow command:

singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
singularity pull denglab-viroprofiler-vibrant-v0.2.img docker://denglab/viroprofiler-vibrant:v0.2
singularity pull denglab-viroprofiler-binning-v0.2.img docker://denglab/viroprofiler-binning:v0.2
singularity pull denglab-viroprofiler-host-v0.2.img docker://denglab/viroprofiler-host:v0.2
singularity pull denglab-viroprofiler-geneannot-v0.2.img docker://denglab/viroprofiler-geneannot:v0.2
singularity pull denglab-viroprofiler-taxa-v0.2.img docker://denglab/viroprofiler-taxa:v0.2
singularity pull denglab-viroprofiler-abundance-v0.2.img docker://denglab/viroprofiler-abundance:v0.2
singularity pull denglab-viroprofiler-bracken-v0.2.img docker://denglab/viroprofiler-bracken:v0.2
singularity pull denglab-viroprofiler-replicyc-v0.2.img docker://denglab/viroprofiler-replicyc:v0.2

@olgais93
Copy link

I'm having the same issue:


Error executing process > 'VIROPROFILER:SETUP:DB_VREFSEQ'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name denglab-viroprofiler-base-v0.2.img.pulling.1681391469512 docker://denglab/viroprofiler-base:v0.2 > /dev/null
  status : 1
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
    WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
    ERROR MANIFEST_UNKNOWN: OCI index found, but accept header does not support OCI indexes
    ERROR: pulling container failed!

@rujinlong
Copy link
Collaborator

rujinlong commented Apr 13, 2023 •
edited
Loading

What is the version of singularity? Could you try the following command to update the pipeline,

nextflow pull deng-lab/viroprofiler

and then run with the -r main parameter,

nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main

@olgais93
Copy link

Thank you! I did as your said and obtained this error:

Error executing process > 'VIROPROFILER:SETUP:DB_CHECKV'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name denglab-viroprofiler-base.img.pulling.1681393187226 docker://denglab/viroprofiler-base > /dev/null
  status : 1
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
    WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
    ERROR MANIFEST_UNKNOWN: OCI index found, but accept header does not support OCI indexes
    ERROR: pulling container failed!

@rujinlong
Copy link
Collaborator

Could you please paste the first two lines of the output here? it should look like the following,

N E X T F L O W  ~  version 22.04.0
Launching `https://github.com/deng-lab/viroprofiler` [soggy_elion] DSL2 - revision: 471c32e0ba [main]

@olgais93
Copy link

Sure!

$ nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
N E X T F L O W  ~  version 22.10.6
Launching `https://github.com/deng-lab/viroprofiler` [hopeful_borg] DSL2 - revision: 471c32e0ba [main]

@rujinlong
Copy link
Collaborator

It's quite strange. The error seems to be caused by singularity. I cannot reproduce this error, but if you can provide additional information, it may help me identify the bug. Could you try as the following,

  1. Pull the image manually using this command.
  2. If it fails, check the singularity version and path using
singularity --version
which singularity

and paste the output here.

@olgais93
Copy link

The first thing failed and gave this error:

$ singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
ERROR: pull is only supported for shub URIs

Singularity version and path:

$ singularity --version
2.4.2-dist
$ which singularity
/home/olga/miniconda3/bin/singularity

@rujinlong
Copy link
Collaborator

Ok, this might be the reason. Your singularity version is quite old. Could you update to the newest version? The latest on conda is singularity v3.8.7 and singularityce v3.10.0. If possible, ask your sys-admin to install the newest singularity as root, as explained in another issue darcyabjones/pante#4 (comment)

@olgais93
Copy link

I installed a newer version of Singularity.

$ singularity --version
singularity-ce version 3.11.1-jammy

It is now giving a different error:

$ nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main

WorkflowError:
Failed to solve the job scheduling problem with pulp. Please report a bug and use --scheduler greedy as a workaround:
Pulp: Error while trying to execute, use msg=True for more detailscbc
[2023-04-13 19:49 CRITICAL] Command 'snakemake --snakefile /opt/conda/envs/viroprofiler-virsorter2/lib/python3.10/site-packages/virsorter/rules/setup-retry.smk --directory /home/olga/viroprofiler/virsorter2 --quiet --config Skip_deps_install=False --jobs 1 --rerun-incomplete --latency-wait 600 --nolock  --use-conda --conda-prefix /home/olga/viroprofiler/virsorter2/conda_envs --conda-frontend mamba ' returned non-zero exit status 1.

Work dir:
/home/olga/work/b9/26db2d6609386b7de7b34fa9ee8433

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Thank you very much

@rujinlong
Copy link
Collaborator

rujinlong commented Apr 13, 2023 •
edited
Loading

It seems this is caused by snakemake in virsorter2 jiarong/VirSorter2#51 (also see this). I updated the code, please try the following commands,

# update the pipeline
nextflow pull deng-lab/viroprofiler

# delete the old virsorter2 database (in your case, /home/olga/viroprofiler/virsorter2)
rm -rf /path/to/viroprofiler/db/virsorter2

# run the pipeline
nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main

@olgais93
Copy link

Thank you, I did as you said and running nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main worked:

executor >  local (3)
[05/27b9fd] process > VIROPROFILER:SETUP:DB_CHECKV     [100%] 1 of 1, cached: 1 ✔
[ec/7f85af] process > VIROPROFILER:SETUP:DB_VIBRANT    [100%] 1 of 1, cached: 1 ✔
[fe/c41c20] process > VIROPROFILER:SETUP:DB_VIRSORTER2 [100%] 1 of 1 ✔
[ef/8e9ed3] process > VIROPROFILER:SETUP:DB_DRAM       [100%] 1 of 1 ✔
[df/78f962] process > VIROPROFILER:SETUP:DB_IPHOP      [100%] 1 of 1 ✔
[cb/451b85] process > VIROPROFILER:SETUP:DB_VREFSEQ    [100%] 1 of 1, cached: 1 ✔
-[ViroProfiler] Pipeline completed successfully-

But then running the test failed:
$ nextflow run deng-lab/viroprofiler -r main -profile singularity,test

This is the error I got:

Error executing process > 'VIROPROFILER:VIRSORTER2'

Caused by:
  Process `VIROPROFILER:VIRSORTER2` terminated with an error exit status (1)

Command executed:

  virsorter config --set HMMSEARCH_THREADS=2
  virsorter run --seqname-suffix-off --viral-gene-enrich-off --prep-for-dramv -i putative_vcontigs_pref1.fasta -w out_vs2 --include-groups dsDNAphage --min-length 3000 --min-score 0.5 -j 2 --provirus-off -d /home/olga/viroprofiler/virsorter2 all
  grep '^>' out_vs2/final-viral-combined.fa | sed 's/>//' | sed 's/||.*//' > virus_virsorter2.list
  ln -s out_vs2/for-dramv/final-viral-combined-for-dramv.fa .
  ln -s out_vs2/for-dramv/viral-affi-contigs-for-dramv.tab .
  seqkit fx2tab -n final-viral-combined-for-dramv.fa | sed 's/-cat_/,/g' | csvtk add-header -n Contig,vs2_category > vs2_category.csv
  
  cat <<-END_VERSIONS > versions.yml
  "VIROPROFILER:VIRSORTER2":
executor >  local (29)
[db/8c0fa0] process > VIROPROFILER:FASTQC (UC30)               [100%] 15 of 15, cached: 15 ✔
[d3/91c5e0] process > VIROPROFILER:FASTP (UC30)                [100%] 15 of 15, cached: 15 ✔
[87/47afb0] process > VIROPROFILER:SPADES (HT24)               [100%] 15 of 15, cached: 14 ✔
[d7/86e681] process > VIROPROFILER:CONTIGLIB                   [100%] 1 of 1 ✔
[ce/a24c4f] process > VIROPROFILER:CHECKV                      [100%] 1 of 1 ✔
[3e/545e1b] process > VIROPROFILER:CONTIGLIB_CLUSTER           [100%] 1 of 1 ✔
[c4/607316] process > VIROPROFILER:GENEPRED4CTG                [100%] 1 of 1 ✔
[46/f2fbf7] process > VIROPROFILER:NRPROT                      [100%] 1 of 1 ✔
[98/6f1211] process > VIROPROFILER:NRGENE                      [100%] 1 of 1 ✔
[02/8154e6] process > VIROPROFILER:CONTIGINDEX                 [100%] 1 of 1 ✔
[6d/32e55f] process > VIROPROFILER:MAPPING2CONTIGS2 (HT24)     [100%] 15 of 15 ✔
[28/e6bfaf] process > VIROPROFILER:ABUNDANCE                   [100%] 1 of 1 ✔
[03/627fa9] process > VIROPROFILER:VIBRANT                     [100%] 1 of 1 ✔
[1c/6f9093] process > VIROPROFILER:DVF                         [100%] 1 of 1 ✔
[84/3d432f] process > VIROPROFILER:VIRCONTIGS_PRE              [100%] 1 of 1 ✔
[42/6be750] process > VIROPROFILER:VIRSORTER2                  [100%] 1 of 1, failed: 1 ✘
[-        ] process > VIROPROFILER:DRAMV                       -
[-        ] process > VIROPROFILER:TAXONOMY_VCONTACT           -
[-        ] process > VIROPROFILER:TAXONOMY_MMSEQS             -
[-        ] process > VIROPROFILER:TAXONOMY_MERGE              -
[f5/24a9a1] process > VIROPROFILER:VIRALHOST_IPHOP             [100%] 1 of 1, failed: 1 ✘
[-        ] process > VIROPROFILER:BACPHLIP                    -
[-        ] process > VIROPROFILER:RESULTS_TSE                 -
[-        ] process > VIROPROFILER:CUSTOM_DUMPSOFTWAREVERSIONS -
[-        ] process > VIROPROFILER:MULTIQC                     -
Pulling Singularity image docker://denglab/viroprofiler-taxa:v0.2 [cache /home/olga/work/singularity/denglab-viroprofiler-taxa-v0.2.img]
Pulling Singularity image docker://denglab/viroprofiler-replicyc:v0.2 [cache /home/olga/work/singularity/denglab-viroprofiler-replicyc-v0.2.img]
Execution cancelled -- Finishing pending tasks before exit

Any ideas?

@rujinlong
Copy link
Collaborator

rujinlong commented Apr 14, 2023 •
edited
Loading

There is a log file in work/42/6be750*/.command.log, could show me the content of that file?

@olgais93
Copy link

In /home/olga/work/f5/24a9a1c9f030cba84ff25d0ee074cc/.command.log, there was this:

Pblm, I could not find directory /home/olga/viroprofiler/iphop/Sept_2021_pub. Maybe the host database directory was not specified with the --db_dir and is not the default "db/" directory ?

@rujinlong
Copy link
Collaborator

Your previous setup of the iphop database is not complete. The complete iphop database is ~200G. Please delete the iphop folder, and then run the setup step again, or you can use the --use_iphop false parameter to skip the host prediction step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rujinlong @olgais93 @liuronghyw