Installation with conda: environment is inconsistent

[Dani-Lindsay]

Hi there,

I'm trying to install isce and mintpy on our hpc system where I don't have administrative permissions. Using conda has worked in the past however this time around (after conda mysteriously broke last Thursday) I am getting the warning below.

The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - conda-forge/linux-64::isce2==2.6.0=py38h95a6a49_0
  - conda-forge/linux-64::gdal==3.4.3=py38h51ade5a_0
  - conda-forge/noarch::mintpy==1.3.3=pyhd8ed1ab_0
  - conda-forge/linux-64::libgdal==3.4.3=h3886835- 
done

Here is the terminal output of my installation
installation.log

I have checked all the paths in my .bashrc are correct. I now get an error with stripmapApp.py -h but smallbaselineApp.py -h seems to work ok.

(insar) [daniellelindsay@ln002 ~]$ stripmapApp.py -h
Traceback (most recent call last):
  File "/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/applications/stripmapApp.py", line 39, in <module>
    from isce import logging
ModuleNotFoundError: No module named 'isce'

Do you think this might be an issue with isce or MintPy or the way I have created the environment maybe? Feeling a bit lost so any advice is helpful! Thank you!

[jhkennedy]

It looks like you ran these commands:

conda create -n insar python=3.8
conda activate insar
conda install -c conda-forge isce2
conda install -c conda-forge mintpy

I was able to walk through those commands without getting an inconsistent environment.

However, for my environment, none of the ISCE applications are on my path and would instead need to be called like:

${ISCE_HOME}/applications/stripmapApp.py -h

The above does work for me without error. I suspect it's something to do with what you've added to your .bashrc.

Can you:

1. Activate your insar environment
2. confirm that running stripmapApp.py -h results in the ModuleNotFoundError
3. post the output of all of these commands:
   • echo $ISCE_HOME
   • echo $PATH
   • echo $PYTHONPATH
   • which stripmapApp.py
   • which python3

[jhkennedy]

What is likely happening is that when you run the ISCE2 stripmapApp.py script, which is invoked like /usr/bin/env python3 strimapApp.py, you're not using your environment's python.

Whereas, because MintPy installs it's scripts like smallbaselineApp.py as console script entry points, smallbaselineApp.py will be invoked via the environment's python directly with a #! that looks like this on your system:

#!/global/home/users/daniellelindsay/.conda/envs/insar/bin/python3.8

which you can confirm with head -1 $(which smallbaselineApp.py).

[Dani-Lindsay]

Thanks for the quick response! I think what you are saying is exactly what is happening. Below is the output from the commands you suggested.

(base) [daniellelindsay@ln001 ~]$ module load python
(base) [daniellelindsay@ln001 ~]$ python --version
Python 3.6.8 :: Anaconda custom (64-bit)
(base) [daniellelindsay@ln001 ~]$ conda activate insar
(insar) [daniellelindsay@ln001 ~]$ python --version 
Python 3.6.8 :: Anaconda custom (64-bit)
(insar) [daniellelindsay@ln001 ~]$ stripmapApp.py -h
Traceback (most recent call last):
  File "/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/applications/stripmapApp.py", line 39, in <module>
    from isce import logging
ModuleNotFoundError: No module named 'isce'
(insar) [daniellelindsay@ln001 ~]$ echo $ISCE_HOME
/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce
(insar) [daniellelindsay@ln001 ~]$ echo $PATH
/global/software/sl-7.x86_64/modules/langs/python/3.6/bin:/global/home/users/daniellelindsay/.conda/envs/insar/bin:/global/software/sl-7.x86_64/modules/langs/python/3.7/condabin:/global/home/users/daniellelindsay/sources/SSARA:/global/home/users/daniellelindsay/.conda/envs/insar/share/isce2/alosStack:/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/applications:/global/software/sl-7.x86_64/modules/tools/sq/0.1.0/bin:/global/software/sl-7.x86_64/modules/tools/emacs/25.1/bin:/global/software/sl-7.x86_64/modules/tools/vim/7.4/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/global/home/groups/allhands/bin:/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/bin:/global/home/users/daniellelindsay/.conda/envs/insar/bin/mintpy:/global/home/users/daniellelindsay/sources/s1_scripts:/global/home/users/daniellelindsay/bin
(insar) [daniellelindsay@ln001 ~]$ echo $PYTHONPATH
:/global/home/users/daniellelindsay/.conda/envs/insar/bin/
(insar) [daniellelindsay@ln001 ~]$ which stripmapApp.py
~/.conda/envs/insar/lib/python3.8/site-packages/isce/applications/stripmapApp.py
(insar) [daniellelindsay@ln001 ~]$ python3 -c 'import isce; print(isce.__file__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'isce'
(insar) [daniellelindsay@ln001 ~]$ 
(insar) [daniellelindsay@ln001 ~]$ head -1 $(which smallbaselineApp.py)
#!/global/home/users/daniellelindsay/.conda/envs/insar/bin/python3.8
(insar) [daniellelindsay@ln001 ~]$ 
(insar) [daniellelindsay@ln001 ~]$ head -1 $(which stripmapApp.py)
#!/usr/bin/env python3

Definitely seems like I am having troubles with using different versions of python. Is there a way to make isce2 use the environmental python? When I login to the hpc and specify the version of python I want to load I get the following:

(base) [daniellelindsay@ln003 ~]$ module load python/3.8.2
(base) [daniellelindsay@ln003 ~]$ python --version
Python 3.8.2
(base) [daniellelindsay@ln003 ~]$ conda activate insar
(insar) [daniellelindsay@ln003 ~]$ python --version
Python 3.8.2
(insar) [daniellelindsay@ln003 ~]$ stripmapApp.py -h
Traceback (most recent call last):
  File "/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/applications/stripmapApp.py", line 39, in <module>
    from isce import logging
ModuleNotFoundError: No module named 'isce'
(insar) [daniellelindsay@ln003 ~]$ echo $ISCE_HOME
/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce
(insar) [daniellelindsay@ln003 ~]$ echo $PATH
/global/software/sl-7.x86_64/modules/langs/python/3.8.2/bin:/global/home/users/daniellelindsay/.conda/envs/insar/bin:/global/software/sl-7.x86_64/modules/langs/python/3.7/condabin:/global/home/users/daniellelindsay/sources/SSARA:/global/home/users/daniellelindsay/.conda/envs/insar/share/isce2/alosStack:/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/applications:/global/software/sl-7.x86_64/modules/tools/sq/0.1.0/bin:/global/software/sl-7.x86_64/modules/tools/emacs/25.1/bin:/global/software/sl-7.x86_64/modules/tools/vim/7.4/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/global/home/groups/allhands/bin:/global/home/users/daniellelindsay/.conda/envs/insar/lib/python3.8/site-packages/isce/bin:/global/home/users/daniellelindsay/.conda/envs/insar/bin/mintpy:/global/home/users/daniellelindsay/sources/s1_scripts:/global/home/users/daniellelindsay/bin
(insar) [daniellelindsay@ln003 ~]$ echo $PYTHONPATH
:/global/home/users/daniellelindsay/.conda/envs/insar/bin/
(insar) [daniellelindsay@ln003 ~]$ which stripmapApp.py
~/.conda/envs/insar/lib/python3.8/site-packages/isce/applications/stripmapApp.py
(insar) [daniellelindsay@ln003 ~]$ which python3
/global/software/sl-7.x86_64/modules/langs/python/3.8.2/bin/python3
(insar) [daniellelindsay@ln003 ~]$ 

This at least means everything is trying to use python 3.8 but not the solution either.

Attached is a copy of my bashrc
savio_bashrc.txt

[jhkennedy]

@Dani-Lindsay first thing I notice is you have the base environment loaded before you do a module load python or module load python/3.8.2, so I think you likely do not want to load any python modules -- that ends up putting that module's python right at the front of your path, and is what is getting picked up. That alone may fix the issue.

After that, I think we can adjust your .bashrc and leverage conda environment configuration to make everything a little cleaner and ensure you're using the right environment.

For the .bashrc:

• you shouldn't need any of the MintPy or PyAPS exports when installing via conda-forge
• we can move all the ISCE exports to the conda environment config

So your .bashrc would end up looking like:

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi

# User specific aliases and functions

##--------- SSARA ------------------##
#Add SSARA to path
export SSARA_HOME=/global/home/users/daniellelindsay/sources/SSARA
export PATH=$SSARA_HOME:$PATH

##-------- S1_scripts ----------#
export S1_scripts=/global/home/users/daniellelindsay/sources/s1_scripts
export PATH=${PATH}:${S1_scripts}

#>>> conda initialize >>>

# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/global/software/sl-7.x86_64/modules/langs/python/3.7/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/global/software/sl-7.x86_64/modules/langs/python/3.7/etc/profile.d/conda.sh" ]; then
        . "/global/software/sl-7.x86_64/modules/langs/python/3.7/etc/profile.d/conda.sh"
    else
        export PATH="/global/software/sl-7.x86_64/modules/langs/python/3.7/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

---

Importantly, do the above, log out and log in again, and then do the next section!

---

And for your insar conda environment, I'd run these commands (only need to do once!):

conda activate insar

conda env config vars set PATH=${PATH}:${ISCE_HOME}/bin:${ISCE_HOME}/applications:${CONDA_PREFIX}/share/isce2/alosStack:${CONDA_PREFIX}/share/isce2/stripmapStack:${CONDA_PREFIX}/share/isce2/topsStack

which will put the ISCE executables, applications, and contributed stack processors on your path every time you activate your conda environment. And when you deactivate, $PATH will be restored to what it was before activation.

Note: If you don't need the stack processors, you can also drop the ${CONDA_PREFIX}/share/isce2/*Stacks from the conda environment config command above.

---

Alternatively, instead of using conda env config (but still adjust your .bashrc!), you could call the executables directly like:

${ISCE_HOME}/bin/mdx
${ISCE_HOME}/applications/stripmapApp.py -h
${CONDA_PREFIX}/share/isce2/alosStack/compute_baseline.py -h

but you'd have to adjust any current scripts/workflows for that.

[Dani-Lindsay]

@jhkennedy you were totally right! If I login in without module load python isce appears to work! Before everything magically broke it was first thing I called every time. Not sure why it worked before but not now. If I find other problems I will implement your suggestions for bashrc and look and moving the path variables to the InSAR conda environment.

Thank you so much for your help with this non-MintPy specific problem.

(base) [daniellelindsay@ln003 ~]$ python --version
Python 3.7.4
(base) [daniellelindsay@ln003 ~]$ conda activate insar
(insar) [daniellelindsay@ln003 ~]$ stripmapApp.py -h
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce

[yunjunz]

I concur with @jhkennedy.

You may also find this installation note (https://github.com/yunjunz/conda_envs) helpful: it installs the development version of mintpy, and the released / development version of isce2, with the path setup included.

[Dani-Lindsay]

@yunjunz I found your instructions yesterday, thanks for sharing! Really helpful! Just wondering what the benefit is to running the development versions rather than stable? Is it easier to update? Is this just a best practice thing I should be doing?

[yunjunz]

I guess it depends on your need for the new features: The released version is more stable, with roughly 3-6 months per new release currently; the development version has the latest feature and bug fixes (and potential newly-introduced bugs) and can be easily updated using git pull.

[Dani-Lindsay]

OK, interesting. I think at the moment I will stick with the stable releases (because I have them working right at this moment) but for the next time I need to re-build I'll take another look at the instructions you shared. Thanks for all the work you do to maintain this great tool!