diff --git a/.github/workflows/build_installation_package.yaml b/.github/workflows/build_installation_package.yaml new file mode 100644 index 0000000..240a82c --- /dev/null +++ b/.github/workflows/build_installation_package.yaml @@ -0,0 +1,42 @@ +name: Build DQMGUI Installation Package + +on: [push] + +jobs: + build_and_upload: + runs-on: ubuntu-20.04 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v4 + - name: Download and compress dependencies + run: | + sudo apt update && sudo apt install -y libcurl4-gnutls-dev + $(which python3) --version + bash download_dependencies.sh + rm -rf .git + mkdir -p /tmp/dqmgui + tar -cf /tmp/dqmgui/dqmgui_installation_package.tar.gz . -I "gzip --best" + - name: Archive package artifact + uses: actions/upload-artifact@v3 + with: + name: dqmgui-installation-package + path: | + /tmp/dqmgui/dqmgui_installation_package.tar.gz + # Create a dummy release, so that the artifact is downloadable + # without a GitHub account + # This will automatically update the artifacts of the "latest" release, + # even though the "latest" tag does match with the actually latest commit. + - name: Release + uses: ncipollo/release-action@v1 + with: + artifacts: "/tmp/dqmgui/dqmgui_installation_package.tar.gz" + body: + "Automatically generated dummy release for making the installation + package publicly available. The `latest` tag is + put on a random commit, and does not reflect the current state of the release, + while the artifact of this release is updated with the actually latest files, + on each push." + tag: "latest" + allowUpdates: "true" diff --git a/README.md b/README.md index 35edbfc..2a81547 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,13 @@ # [Legacy] DQMGUI Deployment -This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.6 for the needs of moving DQM production to new machines. +[![Build DQMGUI Installation Package](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml/badge.svg)](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml) -It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while this script downloads and builds all requirements from source (hence takes longer to deploy). +This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.8 for the needs of moving DQM production to new machines. + +It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while the method we implement in this repository depends on two steps: + +1. Download all the external resources needed (e.g. python packages from PyPI, github repositories) and compress them. This is done automatically with GitHub actions, and you can download a package ready to install [here](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml). +2. Copy the archives to the P5 machine and extract, then build from source (hence takes longer to deploy). > **Warning** > This deployment script should *not* be run as a sudo user. @@ -12,15 +17,12 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt - RHEL8 (Tested with version 8.8) - `sudo` permissions: - To install the system-wide packages (listed below). - -- Python 3.6 -- Git -- Access to PyPI for downloading python packages. -- Access to GitHub for cloning several external dependencies. +- Python 3.8 - Several system packages, installed via `yum`:
- Package list + Package list + - unzip - bzip2 - libglvnd-opengl - libX11-devel @@ -42,12 +44,10 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt - perl-Digest-MD5 - tcsh - zsh - - root - - python3-root - epel-release - - python3-pip - libcurl-devel - - python36-devel + - python38 + - python38-devel - boost-python3-devel - protobuf-devel - jemalloc-devel @@ -56,60 +56,77 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt - lzo-devel - cmake - xz-devel - - python3-sphinx - openssl-devel - libjpeg-turbo-devel - libpng-devel + - gcc-c++ + - gcc + - binutils + - gcc-gfortran + - mesa-libGL-devel + - mesa-libGLU-devel + - glew-devel + - ftgl-devel + - fftw-devel + - cfitsio-devel + - graphviz-devel + - libuuid-devel + - avahi-compat-libdns_sd-devel + - openldap-devel + - python3-numpy + - libxml2-devel + - gsl-devel + - readline-devel + - R-devel + - R-Rcpp-devel + - R-RInside-devel + - xrootd-client
## Deploying [Legacy] DQMGUI -This procedure has been tested on a RHEL8 Openstack VM. +This procedure has been tested on a RHEL8 Openstack VM. Instructions below are primarily for a personal VM. -1. Install the system packages: +0. Download the latest build artifact and copy it to the machine you want to install it to: ```bash - sudo yum install -y tmux git bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel libpng-devel libjpeg-turbo-devel - ``` - 2. Add a non-privileged user, create and give access to necessary directories and switch to it: - + ```bash adduser dqm + + # Installation directory sudo mkdir -p /data/srv sudo chown -R dqm /data/srv + + # Data directory sudo mkdir -p /dqmdata/dqm sudo chown -R dqm /dqmdata + + # Installation package + sudo chown dqm /tmp/dqmgui_installation_package.tar.gz + sudo su dqm ``` 3. Start the deployment (`dev` flavor): - + ```bash cd ~ - # Clone this script and all necessary files, like patches - git clone https://github.com/cms-DQM/dqmgui_prod_deployment - + tar -xf dqmgui_installation_package.tar.gz -C dqmgui_deployment + # Start the deployment script, it will take some time to finish - bash /home/dqm/dqmgui_prod_deployment/deploy_dqmgui.sh + bash /home/dqm/dqmgui_deployment/deploy_dqmgui.sh # Start all the services /data/srv/current/config/dqmgui/manage -f dev start "I did read documentation" @@ -126,7 +143,7 @@ This procedure has been tested on a RHEL8 Openstack VM. firewall-cmd --zone=public --add-port=8081/tcp # relval ``` -## Command-line arguments +## Deployment command-line arguments This script uses some internal variables to specify options such as the main installation directory, or the git tags/refs to use when cloning repositories. It's not recommended that you edit those, as most of them are hand-picked so that the project compiles (we're mostly referring to the git refs). @@ -137,12 +154,10 @@ Below is a list of the values that you will most probably need to change to cust | Variable name | Description |Default value | |---------------|-------------|--------------| | `INSTALLATION_DIR` | The directory to install the GUI into. It should be writable by the user running the script | `/data/srv` | -| `DMWM_GIT_TAG` | The git tag/ref to checkout `dmwm/deployment` to. See [here](https://github.com/dmwm/deployment/tags) for a list. | `HG2903c` | -| `DQMGUI_GIT_TAG` | The git tag/ref to checkout `cms-DQM/dqmgui_prod` to. See [here](https://github.com/cms-DQM/dqmgui_prod) for a list. | `9.8.0` | -## [Debug] Selectively run parts of the installation script +### [Debug] Selectively run parts of the installation script -The script is split into steps (see: `installation_steps`), which can all be toggled off or on by arguments when running the script. +The script is split into steps (see the `installation_steps` array declared in the `deploy_dqmgui.sh` script), which can all be toggled off or on by arguments when running the script. The flags are named by concatenating `do_` with the name of the step, so, for example, `do_check_dependencies` or `do_install_rotoglup`. @@ -152,8 +167,16 @@ The are all set to `1` by default, and you can override them when running the sc bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 ``` -A useful combination that can be used when you've already downloaded all external dependencies once, and you want just to re-compile the DQMGUI part for testing: +A useful combination that can be used when you've already installed and built all steps once, but you only want to re-compile the DQMGUI part for testing: ```bash -bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0 +bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_root=0 do_compile_root=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0 ``` + +## Download script + +The installation package is created by running `download_dependencies.sh`. This is done automatically by github actions (see `.github/workflows/build_installation_package` in this repository). The versions of the packages downloaded are specified in `config.sh`. + +## Notes + +- We're not using the RHEL8 `root` package, due to the fact that they are built for python3.6, hence we need to build it with python3.8. diff --git a/config.sh b/config.sh new file mode 100644 index 0000000..a9e25cc --- /dev/null +++ b/config.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +# Configuration file, which is sourced by both download_dependencies.sh and deploy_dqmgui.sh +# Here we specify both the URLs and tags/branches/refs of the repositories that DQMGUI +# depends on. + +# Tag to use for getting the layouts and manage/deploy scripts +# See: https://github.com/dmwm/deployment/tags +#DMWM_GIT_URL=https://github.com/dmwm/deployment +DMWM_GIT_TAG=debug +DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment + +# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags +#DQMGUI_GIT_TAG=9.8.0 +DQMGUI_GIT_TAG=python3_backup +DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod + +# Boost.GIL. At most version 1.67!! The API changed radically after that. +BOOST_GIL_GIT_URL=https://github.com/boostorg/gil +BOOST_GIL_GIT_TAG=boost-1.66.0 + +# OLD rotoglup code. Commit was found with lots of pain, so that the patch +# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch +ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494 +ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad + +# Yahoo!(TM) UI +YUI_GIT_URL=https://github.com/yui/yui2 +YUI_GIT_TAG=master + +# Extjs +EXTJS_GIT_URL=https://github.com/probonogeek/extjs +EXTJS_GIT_TAG=3.1.1 + +# D3 +D3_GIT_URL=https://github.com/d3/d3 +D3_GIT_TAG=v2.7.4 + +# JSROOT +JSROOT_GIT_URL=https://github.com/root-project/jsroot +JSROOT_GIT_TAG=5.1.0 + +# ROOT +ROOT_GIT_URL=https://github.com/root-project/root/ +ROOT_GIT_TAG=v6-28-08 diff --git a/deploy_dqmgui.sh b/deploy_dqmgui.sh index 586eb72..c54650a 100755 --- a/deploy_dqmgui.sh +++ b/deploy_dqmgui.sh @@ -3,9 +3,10 @@ # Script intended for installing [legacy] DQMGUI on online HLT machines (dqmsrv-...). # It tries to imitate the behavior of the Deploy script without installing external RPMs. # -# It clones DQMGUI, dmwm/deployment code and classlib, creates a virtual env...TODO +# The installation depends on the steps defined in the installation_steps array. +# They are executed in sequence and can be skipped with the appropriate flag. # -# Only targeting RHEL8 + Python3.6 for now(!) +# Only targeting RHEL8 + Python3.8 for now(!) # # If the target (installation+dmqm tag) directory exists (e.g. /data/srv/$DMWM_GIT_TAG), it will # be *DELETED* by the script, before re-installing. The "state" dir is left alone. @@ -25,47 +26,16 @@ EUID_USER_DQM=40454 # Main directory we're installing into. INSTALLATION_DIR=/data/srv +# Where ROOT will be installed +ROOT_INSTALLATION_DIR=/data/srv/root + # This scipt's directory SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) -# Tag to use for getting the layouts and manage/deploy scripts -# See: https://github.com/dmwm/deployment/tags -#DMWM_GIT_TAG=HG2307b -#DMWM_GIT_URL=https://github.com/dmwm/deployment -DMWM_GIT_TAG=debug -DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment - -# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags -#DQMGUI_GIT_TAG=9.8.0 -DQMGUI_GIT_TAG=python3_backup -DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod - -# Boost.GIL configuration -BOOST_GIL_GIT_URL=https://github.com/boostorg/gil - -# At most 1.67 for boost/gil/extension/io/jpeg_io.hpp -BOOST_GIL_GIT_TAG=boost-1.66.0 - -# OLD rotoglup code. Commit was found with lots of pain, so that the patch -# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch -ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494 -ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad - -# Yahoo!(TM) UI -YUI_GIT_URL=https://github.com/yui/yui2 -YUI_GIT_TAG=master - -# Extjs -EXTJS_GIT_URL=https://github.com/probonogeek/extjs -EXTJS_GIT_TAG=3.1.1 - -# D3 -D3_GIT_URL=https://github.com/d3/d3 -D3_GIT_TAG=v2.7.4 - -# JSROOT -JSROOT_GIT_URL=https://github.com/root-project/jsroot -JSROOT_GIT_TAG=5.1.0 +# Get constants from config file +# This is needed to name some directories whose name is +# based on the version of the package (DMWM and DQMGUI). +source $SCRIPT_DIR/config.sh # Preliminary checks to do before installing the GUI preliminary_checks() { @@ -102,13 +72,16 @@ preliminary_checks() { # Check for needed OS-wide dependencies check_dependencies() { pkgs_installed=1 - declare -a required_packages=(bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel + declare -a required_packages=(unzip bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor - libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release - python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel - pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel - libjpeg-turbo-devel libpng-devel) + libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh epel-release + libcurl-devel python38 python38-devel boost-python3-devel protobuf-devel jemalloc-devel + pcre-devel boost-devel lzo-devel cmake xz-devel openssl-devel + libjpeg-turbo-devel libpng-devel gcc-c++ gcc binutils gcc-gfortran mesa-libGL-devel mesa-libGLU-devel + glew-devel ftgl-devel fftw-devel cfitsio-devel graphviz-devel libuuid-devel avahi-compat-libdns_sd-devel + openldap-devel python3-numpy libxml2-devel gsl-devel readline-devel R-devel R-Rcpp-devel R-RInside-devel + xrootd-client) # Instead of doing a 'yum list' per package, it may be faster to just # ask all of them at once, and dump to file. Then grep the file. @@ -168,9 +141,9 @@ create_directories() { mkdir -p "$dirname" done - if [ ! -L $INSTALLATION_DIR/$DMWM_GIT_TAG/apps ]; then + if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps" ]; then echo "DEBUG: Creating link $INSTALLATION_DIR/$DMWM_GIT_TAG/apps.sw <-- $INSTALLATION_DIR/$DMWM_GIT_TAG/apps" - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG/apps.sw $INSTALLATION_DIR/$DMWM_GIT_TAG/apps + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps.sw" "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps" fi # Create a "current" link to the DMWM version we're using, like how it was done @@ -179,57 +152,22 @@ create_directories() { rm $INSTALLATION_DIR/current fi echo "DEBUG: Creating link $INSTALLATION_DIR/$DMWM_GIT_TAG <-- $INSTALLATION_DIR/current" - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG $INSTALLATION_DIR/current + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG" "$INSTALLATION_DIR/current" # Directories for external source and lib files (e.g. classlib) - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src" + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib" # DQMGUI dirs echo "DEBUG: Creating subdirectory $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" } -# Helper function to clone a repo and checkout to specific tag -# Takes the following positional arguments: -# - URL of the repo, -# - A reference (tag, branch, commit) to checkout to and -# - The directory to clone the code to. -_clone_from_git() { - cd $INSTALLATION_DIR/ - # repo url - url=$1 - - # Which branch/tag to checkout to - ref=$2 - - # The path to the directory to clone to - dir=$3 - - # If tmp dir exists, remove it - if [[ -d $dir ]]; then - rm -rf "$dir" - fi - - # Shallow clone - git clone "$url" "$dir" --depth 1 - - # Checkout to ref - cd "$dir" - # Shallow clone misses remote branches, so - # fix that. - git remote set-branches origin '*' - git fetch --all --tags - git fetch --depth=1 - # DIRTY HACK - git checkout "$ref" || (git fetch --all --depth=50 && git checkout "$ref") - cd $INSTALLATION_DIR/ -} - install_rotoglup() { ROTOGLUP_TMP_DIR=/tmp/rotoglup - _clone_from_git $ROTOGLUP_GIT_URL $ROTOGLUP_GIT_TAG $ROTOGLUP_TMP_DIR + mkdir -p $ROTOGLUP_TMP_DIR + tar -xzf "$SCRIPT_DIR/rotoglup/rotoglup.tar.gz" -C /tmp cd $ROTOGLUP_TMP_DIR #patch -p1 < $SCRIPT_DIR/rotoglup/patches/01.patch @@ -241,25 +179,25 @@ install_rotoglup() { # Compilation step for classlib compile_classlib() { - cd $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3 + cd "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3" #INCLUDE_DIRS="$INCLUDE_DIRS:/usr/include/lzo" make -j `nproc` make -j "$(nproc)" CXXFLAGS="-Wno-error=extra -ansi -pedantic -W -Wall -Wno-long-long -Werror" # Move the compiled library in the libs dir - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/.libs/libclasslib.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libclasslib.so + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/.libs/libclasslib.so" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libclasslib.so" } # Classlib is needed both as a shared object and for its header files for DQMGUI compilation. install_classlib() { # Temporary directory to extract to - CLASSLIB_TMP_DIR=/tmp/classlib + CLASSLIB_TMP_DIR=/tmp/classlib-3.1.3 mkdir -p $CLASSLIB_TMP_DIR - tar -xf "$SCRIPT_DIR/classlib/classlib-3.1.3.tar.bz2" -C $CLASSLIB_TMP_DIR + tar -xf "$SCRIPT_DIR/classlib/classlib-3.1.3.tar.bz2" -C /tmp # Apply code patches I found on cmsdist. The 7th one is ours, and has some extra needed fixes. - cd $CLASSLIB_TMP_DIR/classlib-3.1.3 + cd $CLASSLIB_TMP_DIR for i in 1 2 3 4 5 6 7 8; do patch -p1 <"$SCRIPT_DIR/classlib/patches/0${i}.patch" done @@ -275,55 +213,52 @@ install_classlib() { !/^\S+: / && s{\S+LZO((C|Dec)ompressor|Constants|Error)\S+}{}g' \ $CLASSLIB_TMP_DIR/classlib-3.1.3/Makefile - if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3 ]; then - rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3 + if [ -d "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3" ]; then + rm -rf "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3" fi # Move the classlib files inside the installation dir, needed for compiling the GUI - mv $CLASSLIB_TMP_DIR/classlib-3.1.3 $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/ + mv "$CLASSLIB_TMP_DIR" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/" # Make a link so that DQMGUI compilation can find the classlib headers easily - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/classlib $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/classlib" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib" rm -rf $CLASSLIB_TMP_DIR } install_boost_gil() { - BOOST_GIL_TMP_DIR=/tmp/boost_gil - _clone_from_git $BOOST_GIL_GIT_URL $BOOST_GIL_GIT_TAG $BOOST_GIL_TMP_DIR - mv $BOOST_GIL_TMP_DIR/include/boost $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost - rm -rf $BOOST_GIL_TMP_DIR + mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/ + tar -xzf "$SCRIPT_DIR/boost_gil/boost_gil.tar.gz" -C /tmp + mv /tmp/boost_gil/include/boost "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost" } install_gil_numeric() { NUMERIC_TMP_DIR=/tmp/numeric - mkdir -p $NUMERIC_TMP_DIR - tar -xf "$SCRIPT_DIR/numeric/numeric.tar.gz" -C $NUMERIC_TMP_DIR - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/ - mv $NUMERIC_TMP_DIR/numeric $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/numeric - rm -rf $NUMERIC_TMP_DIR + tar -xzf "$SCRIPT_DIR/numeric/numeric.tar.gz" -C /tmp + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/" + mv "$NUMERIC_TMP_DIR" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/numeric" } install_dmwm() { # Temporary directory to clone DMWM deployment scripts into - DMWM_TMP_DIR=/tmp/deployment - _clone_from_git $DMWM_GIT_URL $DMWM_GIT_TAG $DMWM_TMP_DIR - + DMWM_TMP_DIR=/tmp/dmwm + mkdir -p $DMWM_TMP_DIR + tar -xzf "$SCRIPT_DIR/dmwm/dmwm.tar.gz" -C /tmp # Move dqmgui-related scripts from DMWM to the config folder - mv $DMWM_TMP_DIR/dqmgui $INSTALLATION_DIR/$DMWM_GIT_TAG/config/ + mv "$DMWM_TMP_DIR/dqmgui" "$INSTALLATION_DIR/$DMWM_GIT_TAG/config/" rm -rf $DMWM_TMP_DIR } # env.sh and init.sh file creation. They're needed by other scripts (e.g. manage). _create_env_and_init_sh() { - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/ + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/" # init.sh contents. This is sourced by env.sh echo "export PATH=$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/bin:$PATH export PYTHONPATH=$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib/python${PYTHON_VERSION}/site-packages:$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib64/python${PYTHON_VERSION}/site-packages . $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/activate -" >$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/init.sh +" >"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/init.sh" # env.sh contents. This is sourced by the manage script echo ". $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/init.sh @@ -336,7 +271,8 @@ export DQMGUI_VERSION='$DQMGUI_GIT_TAG'; # For pointing to the custom built libraries export LD_PRELOAD=\"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/libDQMGUI.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libclasslib.so\" export LD_LIBRARY_PATH=\"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/:$LD_LIBRARY_PATH\" -" >$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/env.sh +source $ROOT_INSTALLATION_DIR/bin/thisroot.sh +" >"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/env.sh" } # Crete the Python3 virtual environment for the GUI @@ -345,31 +281,48 @@ _create_python_venv() { python_venv_dir=$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv mkdir -p "$python_venv_dir" - eval "$python_exe" -m pip install --upgrade pip --user - eval "$python_exe" -m pip install virtualenv --user + + # Extract the downloaded python packages + tar -xzf $SCRIPT_DIR/pypi/pypi.tar.gz -C /tmp echo -n "INFO: Creating virtual environment at $python_venv_dir" - virtualenv $python_venv_dir + $python_exe -m venv $python_venv_dir # Now use the new venv's python python_venv_exe=$python_venv_dir/bin/python - cd $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/ - eval $python_venv_exe -m pip install -r $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/requirements.txt - echo "Done" # Needed for specifying the PYTHONPATH later - export PYTHON_VERSION=$($python_exe --version | cut -d ' ' -f 2 | cut -d '.' -f 1,2) - export PYTHON_LIB_DIR_NAME=lib/python$PYTHON_VERSION/site-packages + PYTHON_VERSION=$($python_exe --version | cut -d ' ' -f 2 | cut -d '.' -f 1,2) + export PYTHON_VERSION + PYTHON_LIB_DIR_NAME=lib/python$PYTHON_VERSION/site-packages + export PYTHON_LIB_DIR_NAME + + # Install pip + unzip -u /tmp/pip/pip*whl -d /tmp/pip/pip + if [ -d "$python_venv_dir/$PYTHON_LIB_DIR_NAME/pip" ]; then + rm -rf "$python_venv_dir/$PYTHON_LIB_DIR_NAME/pip" + fi + + # pipipipi + mv /tmp/pip/pip/pip "$python_venv_dir/$PYTHON_LIB_DIR_NAME/pip" + rm -rf /tmp/pip/pip + + # Install wheels + eval "${python_venv_exe} -m pip install --no-index --find-links /tmp/pip /tmp/pip/*" + eval "${python_venv_exe} -m pip install $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128" + + cd "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/" + rm -rf /tmp/pip + echo "Done" } # External requirements for building the GUI # Must be run after the venv is created _create_makefile_ext() { - echo "INCLUDE_DIRS = . $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp /usr/include/root /usr/include/libpng16 /usr/include/jemalloc $(python3-config --includes | sed -e 's/-I//g') $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src /usr/include/google/protobuf /usr/include/boost - -LIBRARY_DIRS = . /usr/lib64/root $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib /usr/lib /usr/lib64 $(python3-config --ldflags | sed -e 's/-L//g' | sed -e '[[:space:]]-l[a-zA-Z0-9\.]+') $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/ $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/ -" >$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/makefile.ext + echo "INCLUDE_DIRS = . $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp /usr/include/libpng16 /usr/include/jemalloc $(root-config --incdir) $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src /usr/include/google/protobuf /usr/include/boost $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/root +LIBRARY_DIRS = . $ROOT_INSTALLATION_DIR/lib $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib /usr/lib /usr/lib64 $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/ $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/ +" >"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/makefile.ext" } # Compile and setup all the required stuff that DQMGUI needs. @@ -377,18 +330,26 @@ LIBRARY_DIRS = . /usr/lib64/root $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib # Then runs the actual compilation, which is the part that takes the longest # in this script. compile_dqmgui() { - cd $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/ + cd "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/" # Links to python libraries so that the build command can find them - if [ ! -L $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so ]; then - ln -s /usr/lib64/libboost_python3.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so + if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so" ]; then + ln -s /usr/lib64/libboost_python3.so "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so" fi if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libpython${PYTHON_VERSION}.so" ]; then ln -s /usr/lib64/libpython3.so "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libpython${PYTHON_VERSION}.so" fi + # python3-config is not always in a predictable place + python_config_cmd=$(which python3-config) || python_config_cmd=$(find /usr/bin -name "python3*-config" | head -1) + + if [ -z "$python_config_cmd" ]; then + echo "ERROR: Could not find python3-config" + exit 1 + fi # The actual build command. Uses the makefile in the DQMGUI's repo. - PYTHONPATH="$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib/python${PYTHON_VERSION}/site-packages:$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib64/python${PYTHON_VERSION}/site-packages" CPLUS_INCLUDE_PATH="$(python3-config --includes | sed -e 's/-I//g')" $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/python $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/setup.py -v build_system -s DQM -d + source "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/activate" + PYTHONPATH="$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib/python${PYTHON_VERSION}/site-packages:$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib64/python${PYTHON_VERSION}/site-packages" CPLUS_INCLUDE_PATH="$(${python_config_cmd} --includes | sed -e 's/-I//g')" $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/python $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/setup.py -v build_system -s DQM -d # Stuff that I found being done in the dqmgui spec file. I kind of blindy copy paste it # here because reasons. @@ -396,12 +357,12 @@ compile_dqmgui() { # Move executables to expected place for exe in DQMCollector visDQMIndex visDQMRender; do - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/$exe $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/bin/$exe + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/$exe" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/bin/$exe" done # Move libs to expected place - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/ - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/libDQMGUI.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/libDQMGUI.so + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/" + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/libDQMGUI.so" $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/libDQMGUI.so # Move the custom Boost.Python interface library to libs. mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/Accelerator.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/build/lib/Monitoring/DQM/Accelerator.so @@ -412,29 +373,31 @@ compile_dqmgui() { # The resulting directory structure and compiled binaries is a mess, but that's the best # we can do right now, considering the existing mess. install_dqmgui() { + # Activate ROOT, we need it to be available so that we can run root-config later + source "$INSTALLATION_DIR/root/bin/thisroot.sh" + # Temporary directory to clone GUI into - DQMGUI_TMP_DIR=/tmp/128 - _clone_from_git $DQMGUI_GIT_URL $DQMGUI_GIT_TAG $DQMGUI_TMP_DIR + DQMGUI_TMP_DIR=/tmp/dqmgui + tar -xzf "$SCRIPT_DIR/dqmgui/dqmgui.tar.gz" -C /tmp # Move dqmgui source and bin files to appropriate directory - if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG ]; then - rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG + if [ -d "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" ]; then + rm -rf "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" fi - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128 - - mv $DQMGUI_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG + mkdir -p "$DQMGUI_TMP_DIR" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/" + mv $DQMGUI_TMP_DIR "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128" - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/data - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/ - if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates ]; then - rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/data" + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/" # Needed for DQMGUI templates + if [ -d "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates" ]; then + rm -rf "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates" fi - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/templates $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/templates" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates" - if [ ! -L $INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui ]; then + if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui" ]; then echo "DEBUG: Creating link $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG <-- $INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui" - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG $INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui" fi # Create python venv for all python "binaries" and webserver @@ -448,28 +411,58 @@ install_dqmgui() { _create_makefile_ext } +# Javascript library install_yui() { - YUI_TMP_DIR=/tmp/yui - _clone_from_git $YUI_GIT_URL $YUI_GIT_TAG $YUI_TMP_DIR - mv $YUI_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/yui + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/yui" + tar -xzf "$SCRIPT_DIR/yui/yui.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" } +# Javascript library install_extjs() { - EXTJS_TMP_DIR=/tmp/extjs - _clone_from_git $EXTJS_GIT_URL $EXTJS_GIT_TAG $EXTJS_TMP_DIR - mv $EXTJS_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/extjs + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/extjs" + tar -xzf "$SCRIPT_DIR/extjs/extjs.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" } +# Javascript library install_d3() { - D3_TMP_DIR=/tmp/d3 - _clone_from_git $D3_GIT_URL $D3_GIT_TAG $D3_TMP_DIR - mv $D3_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/d3 + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/d3" + tar -xzf "$SCRIPT_DIR/d3/d3.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" } +# Javascript library install_jsroot() { - JSROOT_TMP_DIR=/tmp/jsroot - _clone_from_git $JSROOT_GIT_URL $JSROOT_GIT_TAG $JSROOT_TMP_DIR - mv $JSROOT_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/jsroot + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/jsroot" + tar -xzf "$SCRIPT_DIR/jsroot/jsroot.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" +} + +# Extract the ROOT tar to a tmp folder for compilation +install_root() { + tar -xzf "$SCRIPT_DIR/root/root.tar.gz" -C /tmp + #if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/root ]; then + # rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/root + #fi + # + #cp -r /tmp/root $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/root +} + +compile_root() { + ROOT_TMP_DIR=/tmp/root + ROOT_TMP_BUILD_DIR=/tmp/root_build + if [ ! -d $ROOT_TMP_DIR ]; then + echo "ERROR: ROOT source was not found in $ROOT_TMP_DIR" + exit 1 + fi + + if source "$ROOT_INSTALLATION_DIR/bin/thisroot.sh"; then + echo "INFO: ROOT installation found, not re-compiling ROOT" + return + fi + mkdir -p $ROOT_TMP_BUILD_DIR + cd $ROOT_TMP_BUILD_DIR + cmake -DCMAKE_INSTALL_PREFIX=$ROOT_INSTALLATION_DIR $ROOT_TMP_DIR -DPython3_ROOT_DIR=$(which python3) + cmake --build . --target install -j $(nproc) + cd $INSTALLATION_DIR + rm -rf $ROOT_TMP_DIR $ROOT_TMP_BUILD_DIR } ### Main script ### @@ -485,6 +478,8 @@ declare -a installation_steps=(preliminary_checks install_classlib compile_classlib install_dmwm + install_root + compile_root install_dqmgui compile_dqmgui install_yui diff --git a/download_dependencies.sh b/download_dependencies.sh new file mode 100644 index 0000000..f3253cd --- /dev/null +++ b/download_dependencies.sh @@ -0,0 +1,119 @@ +#!/bin/bash + +# Script that downloads all online dependencies for [Legacy] DQMGUI +# in order to install them at P5, where there is no access to neither GitHub, +# nor PyPI. This script is to be called whenever those packages are to be recreated, e.g +# + +set -ex + +# This scipt's directory +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) + +# Get constants from config file +source "$SCRIPT_DIR/config.sh" + +# Helper function to clone a repo and checkout to specific tag +# Takes the following positional arguments: +# - URL of the repo, +# - A reference (tag, branch, commit) to checkout to and +# - The directory to clone the code to. +_clone_from_git() { + # repo url + url=$1 + + # Which branch/tag to checkout to + ref=$2 + + # The path to the directory to clone to + dir=$3 + + # If tmp dir exists, remove it + if [[ -d $dir ]]; then + rm -rf "$dir" + fi + + # Shallow clone. We don't use --branch as we may need this to work + # for a specific commit, too. + git clone "$url" "$dir" --depth 1 + + # Checkout to ref. + cd "$dir" + # Shallow clone misses remote branches, so + # fix that. + git remote set-branches origin '*' + git fetch --all --tags + git fetch --depth=1 + # DIRTY HACK + git checkout "$ref" || (git fetch --all --depth=50 && git checkout "$ref") + cd - +} + +download_repos() { + for repo in "${repos_to_download[@]}"; do + download_repo_flag_name=do_download_$repo + if [ "${!download_repo_flag_name}" -eq 0 ]; then + echo "INFO: Skipping downloading $repo" + continue + fi + + echo "INFO: Downloading $repo" + + prefix="$(echo $repo | tr '[:lower:]' '[:upper:]')" + git_url=${prefix}_GIT_URL + git_tag=${prefix}_GIT_TAG + if [ -z "${!git_url}" ] || [ -z "${!git_tag}" ]; then + echo "WARNING: git url or tag not configured for repository $repo, skipping" + continue + fi + + temp_dir=/tmp/$repo + mkdir -p "$temp_dir" + _clone_from_git ${!git_url} ${!git_tag} $temp_dir + rm -rf "$temp_dir/.git" # Not needed, and will slow things down + mkdir -p "$SCRIPT_DIR/$repo" + if [ -f "$SCRIPT_DIR/$repo/$repo.tar.gz" ]; then + rm "$SCRIPT_DIR/$repo/$repo.tar.gz" + fi + tar -cf "$SCRIPT_DIR/$repo/$repo.tar.gz" --directory=/tmp "$repo" -I "gzip --best" + rm -rf "$temp_dir" + done +} + +download_python_packages() { + declare -a pypi_requirements=(pip sphinx wheel cherrypy jsmin pycurl numpy matplotlib pillow Cheetah3 virtualenv) + PIP_TEMP_DIR=/tmp/pip + mkdir -p $PIP_TEMP_DIR + mkdir -p "$SCRIPT_DIR/pypi" + python_exe=$(which python3) + eval "$python_exe -m pip download ${pypi_requirements[*]} --destination-directory $PIP_TEMP_DIR" + tar -cf "$SCRIPT_DIR/pypi/pypi.tar.gz" --directory=/tmp pip -I "gzip --best" + rm -rf $PIP_TEMP_DIR +} + +### Main script + +# For GitHub repos, clone the source on a specific branch/tag/ref, then make a tar. +declare -a repos_to_download=(rotoglup boost_gil dmwm dqmgui yui extjs d3 jsroot root) + +# Create dynamic flags to selectively disable/enable steps of the download procedure +# Those flags are named "do_download" with the name of the repo, e.g. "do_download_root" for +# the "root" repo. +# We set those flags to 1 by default. +for repo in "${repos_to_download[@]}"; do + eval "do_download_${repo}=1" +done + +# Parse command line arguments -- use = to override the flags mentioned above. +# e.g. DQMGUI_GIT_TAG=9.9.0 +for ARGUMENT in "$@"; do + KEY=$(echo "$ARGUMENT" | cut -f1 -d=) + KEY_LENGTH=${#KEY} + VALUE="${ARGUMENT:$KEY_LENGTH+1}" + eval "$KEY=$VALUE" +done + +download_repos +download_python_packages + +echo "INFO: Done!" diff --git a/numeric/numeric.tar.gz b/numeric/numeric.tar.gz deleted file mode 100644 index 063b861..0000000 Binary files a/numeric/numeric.tar.gz and /dev/null differ