building binutils from eb inside singularity. Initial work #3
Conversation
done by Paulo Souza. Few modifications done by Shahzeb Siddiqui. Attaching log file for output just in case there are issues building container on different architectures
|
@shahzebsiddiqui why do you want to merge a log file? |
|
I can remove the log file if you want, I just put it in the commit to capture the output of the build. For instance the container might build on some architectures and OS but not others. |
binutils/binutils-2.26.def
Outdated
| # https://gcc.gnu.org/wiki/InstallingGCC | ||
| yum install -y libmpc-devel mpfr-devel gmp-devel | ||
| yum install -y zlib-devel* | ||
| yum install -y curl |
There was a problem hiding this comment.
none of this is actually needed, I would let dependencies be handled by EasyBuild?
binutils/binutils-2.26.def
Outdated
| # scratch space to build and download source files from eb | ||
| export EASYBUILD_PREFIX=/scratch | ||
| mkdir -p $EASYBUILD_PREFIX | ||
| chmod 777 $EASYBUILD_PREFIX |
binutils/binutils-2.26.def
Outdated
| mkdir -p $EASYBUILD_PREFIX | ||
| chmod 777 $EASYBUILD_PREFIX | ||
|
|
||
| mkdir -p /app |
binutils/binutils-2.26.def
Outdated
| pip install -U setuptools | ||
| pip install -U easybuild | ||
| yum install -y git | ||
| chmod 777 /app |
There was a problem hiding this comment.
use $EASYBUILD_INSTALLPATH, but there's no need for the 777?
There was a problem hiding this comment.
I removed 777 for now. Also removed git since we are installing easybuild via pip. chmod 777 was set so that singularity user can write to /scratch and /app. By default root will create the directories with 755 privilege and we want to install eb apps with a non-root account. That is why 777 was set.
binutils/binutils-2.26.def
Outdated
| chmod 777 /app | ||
| useradd singularity | ||
| . /etc/profile | ||
| module load lmod |
There was a problem hiding this comment.
this is a bit strange, chicken-or-egg problem?
There was a problem hiding this comment.
I agree, I don't think we need this, also no need to load lmod module. I think runuser singularity -c will run in Interactive Non-login mode. With bash we need the singularity user to have ~/.bashrc which I confirmed did exist using useradd command. There is a known issue with Interactive Non-login for Bash user in Lmod see page 26 of https://media.readthedocs.org/pdf/lmod/latest/lmod.pdf. The solution seems to be in implemented already so singularity should pick up module command with this shell invocation.
binutils/binutils-2.26.def
Outdated
|
|
||
| runuser singularity -c 'eb b/binutils/binutils-2.26.eb --robot' | ||
| rm -rf $EASYBUILD_PREFIX | ||
| chmod 755 /app |
There was a problem hiding this comment.
use $EASYBUILD_INSTALLPATH, but no need for this?
There was a problem hiding this comment.
I removed the line
binutils/binutils-2.26.def
Outdated
|
|
||
| cd $EASYBUILD_PREFIX | ||
| runuser singularity -c 'git clone https://github.com/hpcugent/easybuild-easyconfigs.git' | ||
| cd easybuild-easyconfigs/easybuild/easyconfigs/ |
There was a problem hiding this comment.
this is not needed, easyconfigs are shipped with EasyBuild?
binutils/binutils-2.26.def
Outdated
| runuser singularity -c 'git clone https://github.com/hpcugent/easybuild-easyconfigs.git' | ||
| cd easybuild-easyconfigs/easybuild/easyconfigs/ | ||
|
|
||
| export EASYBUILD_ROBOT_PATHS=$PWD |
There was a problem hiding this comment.
I agree, this should be setup within easybuild when installing with pip
binutils/binutils-2.26.def
Outdated
| export EASYBUILD_ROBOT_PATHS=$PWD | ||
|
|
||
| # mkdir $PWD/tmp | ||
| # chmod 777 $PWD/tmp |
binutils/binutils-2.26.def
Outdated
| yum install -y epel-release | ||
| yum install -y Lmod | ||
| # vi | ||
| yum install -y vim-enhanced |
There was a problem hiding this comment.
We still need epel-release and Lmod, epel-release is also needed to install pip I have removed vim and Development Tools. I have commented out tcsh, not sure if we want to support csh.
| yum install -y Lmod | ||
|
|
||
| # eb dependencies | ||
| yum install python-devel -y |
|
|
||
| # eb dependencies | ||
| yum install python-devel -y | ||
| yum install python-setuptools -y |
There was a problem hiding this comment.
being installed with pip below, so either drop this, or drop the pip install setuptools?
| . /etc/bashrc | ||
| fi | ||
| module use /app/modules/all | ||
| module load binutils |
There was a problem hiding this comment.
Can we somehow use a variable for both the software name and (long) version?
So you can do:
%post
...
runuser singularity -c "eb ${SOFTWARE_NAME}-${SOFTWARE_VERSION}.eb --robot"
%environment
...
module load ${SOFTWARE_NAME}/${SOFTWARE_VERSION}
Not sure if that's possible though...
| # scratch space to build and download source files from eb | ||
| export EASYBUILD_PREFIX=/scratch | ||
|
|
||
| mkdir -p $EASYBUILD_PREFIX |
| # module load lmod | ||
|
|
||
|
|
||
| runuser singularity -c 'echo $PATH; env | grep EASYBUILD; eb --version; eb --show-config' |
There was a problem hiding this comment.
the env | grep EASYBUILD is kind of overkill, since eb --show-config will show via which mechanism each setting was defined (E for environment)
There was a problem hiding this comment.
i was testing this out, because eb was not working due to PATH issue. Dont worry will remove this command
done by Paulo Souza. Few modifications done by Shahzeb Siddiqui.
Attaching log file for output just in case there are issues building
container on different architectures