Failed to create user namespace: user namespace disabled

[arpan-52]

Let me first describe the setup.

1. I have made a python 3.8 virtual environment with Anaconda.
2. Installed singularity with conda, version 3.8.6
3. Installed Stimela version 1.7.9 with Conda.
4. I have also specified the path for the singularity cache and temp directories.
5. After pulling all the images with singularity I tried to run my .yml file.
6. Whatever I am trying to do I am getting an error..

The error.....

INFO: Converting SIF file to temporary sandbox...

FATAL: while extracting /data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled

: exit status 1

2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17060054084448693 --containall --userns returns error code 255
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: job failed at 2024-01-23 15:53:31.281294 after 0:00:01.924652
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: Traceback (most recent call last):
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: job.run_job()
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: self.job.run(output_wrangler=self.apply_output_wranglers)
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/singularity.py", line 128, in run
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: utils.xrun(f"{BINARY} run --workdir {self.execdir} --containall {extras}",
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: stimela.utils.StimelaCabRuntimeError: /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17060054084448693 --containall --userns returns error code 255
2024-01-23 15:53:

From the initial inspection, it looks to me as a singularity problem. Is not conda installed singularity sufficient? Do we need a singularity with root permissions?

[KshitijT]

This seems similar to #1532. @arpan-52 do you permissions to pull the singularity images on the machine you are using?

[arpan-52]

Yes, the images are pulled successfully.

-rw-rw-r-- 1 yanda yanda 207 Dec 15 15:08 log-stimela.txt
-rw-rw-r-- 1 yanda yanda 3.1K Dec 15 15:08 stimela.config.deps
drwxrwxr-x 2 yanda yanda 4.0K Dec 15 16:17 STIMELA_IMAGES
-rw-rw-r-- 1 yanda yanda 765 Dec 21 23:06 start.sh
drwxrwxr-x 3 yanda yanda 27 Jan 23 15:48 singularity_cache
drwxrwxr-x 10 yanda yanda 242 Jan 24 20:13 singularity_tmp
drwxrwxr-x 5 yanda yanda 247 Jan 24 20:13 test_caracal

STIMELA_IMAGES are the folder with all the images. yanda is my user name.

[KshitijT]

Could you please specify your caracal run command and also append the full log?

[arpan-52]

I have used

stimela run ddf_stimela_hogbom_mask.py

This .py file contains all the recipes to do the imaging.

[arpan-52]

log:

2025-02-05 22:39:23 STIMELA.DE calibration INFO: ---------------------------------
2024-02-05 22:39:23 STIMELA.DE calibration INFO: Stimela version 1.7.9
2024-02-05 22:39:23 STIMELA.DE calibration INFO: Running: DE calibration
2024-02-05 22:39:23 STIMELA.DE calibration INFO: ---------------------------------
2024-02-05 22:39:23 STIMELA.DE calibration INFO: Config directory cannot be found. Will create /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17071529638081439/stimela_parameter_files
2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif' (x.x.x) to recipe, container name 'image_target_1'
2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_breizorro_1.7.4.sif' (0.1.0) to recipe, container name 'mask1'
2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif' (x.x.x) to recipe, container name 'image_target_2'
2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_cubical_1.7.8-dev.sif' (1.6.3) to recipe, container name 'calibrate_dd'
2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_breizorro_1.7.4.sif' (0.1.0) to recipe, container name 'mask2'
2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif' (x.x.x) to recipe, container name 'image_target_3'
2024-02-05 22:39:24 STIMELA.DE calibration INFO: STEP 1 :: image_target_1::ddfacet
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Completed jobs : []
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Remaining jobs : ['mask1', 'image_target_2', 'calibrate_dd', 'mask2', 'image_target_3']
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: mask1:: Make initial mask
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: image_target_2::ddfacet
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: calibrate_dd::cubical
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: mask2:: Make initial mask
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: image_target_3::ddfacet
2024-02-05 22:39:28 STIMELA.DE calibration INFO: Saving pipeline information in .last_DE_calibration.json

FATAL: while extracting /data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled
: exit status 1
/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17071529638081439 --containall --userns returns error code 255
job failed at 2024-02-05 22:39:28.804534 after 0:00:03.951822
Traceback (most recent call last):
File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
job.run_job()
File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
self.job.run(output_wrangler=self.apply_output_wranglers)
File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/singularity.py", line 128, in run
utils.xrun(f"{BINARY} run --workdir {self.execdir} --containall {extras}",
File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17071529638081439 --containall --userns returns error code 255

[KshitijT]

@arpan-52 this is a stimela issue, not a CARACal one, could you please close this issue and open it on the stimela repository?

[arpan-52]

Hi, I finally managed to tackle the issue with a system-wide updated version of singularity installation. The issue is now gone and the whole code works well. Thanks a lot.