From 20ed6f9e5d79ff89414a3b4a159645cbe858cb37 Mon Sep 17 00:00:00 2001 From: Dimitris Papagiannis Date: Mon, 6 Nov 2023 17:18:13 +0100 Subject: [PATCH] Making it work offline --- .../workflows/build_installation_package.yaml | 42 +++ README.md | 109 +++--- config.sh | 45 +++ deploy_dqmgui.sh | 311 +++++++++--------- download_dependencies.sh | 119 +++++++ numeric/numeric.tar.gz | Bin 8488 -> 0 bytes 6 files changed, 425 insertions(+), 201 deletions(-) create mode 100644 .github/workflows/build_installation_package.yaml create mode 100644 config.sh create mode 100644 download_dependencies.sh delete mode 100644 numeric/numeric.tar.gz diff --git a/.github/workflows/build_installation_package.yaml b/.github/workflows/build_installation_package.yaml new file mode 100644 index 0000000..240a82c --- /dev/null +++ b/.github/workflows/build_installation_package.yaml @@ -0,0 +1,42 @@ +name: Build DQMGUI Installation Package + +on: [push] + +jobs: + build_and_upload: + runs-on: ubuntu-20.04 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v4 + - name: Download and compress dependencies + run: | + sudo apt update && sudo apt install -y libcurl4-gnutls-dev + $(which python3) --version + bash download_dependencies.sh + rm -rf .git + mkdir -p /tmp/dqmgui + tar -cf /tmp/dqmgui/dqmgui_installation_package.tar.gz . -I "gzip --best" + - name: Archive package artifact + uses: actions/upload-artifact@v3 + with: + name: dqmgui-installation-package + path: | + /tmp/dqmgui/dqmgui_installation_package.tar.gz + # Create a dummy release, so that the artifact is downloadable + # without a GitHub account + # This will automatically update the artifacts of the "latest" release, + # even though the "latest" tag does match with the actually latest commit. + - name: Release + uses: ncipollo/release-action@v1 + with: + artifacts: "/tmp/dqmgui/dqmgui_installation_package.tar.gz" + body: + "Automatically generated dummy release for making the installation + package publicly available. The `latest` tag is + put on a random commit, and does not reflect the current state of the release, + while the artifact of this release is updated with the actually latest files, + on each push." + tag: "latest" + allowUpdates: "true" diff --git a/README.md b/README.md index 35edbfc..2a81547 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,13 @@ # [Legacy] DQMGUI Deployment -This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.6 for the needs of moving DQM production to new machines. +[![Build DQMGUI Installation Package](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml/badge.svg)](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml) -It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while this script downloads and builds all requirements from source (hence takes longer to deploy). +This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.8 for the needs of moving DQM production to new machines. + +It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while the method we implement in this repository depends on two steps: + +1. Download all the external resources needed (e.g. python packages from PyPI, github repositories) and compress them. This is done automatically with GitHub actions, and you can download a package ready to install [here](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml). +2. Copy the archives to the P5 machine and extract, then build from source (hence takes longer to deploy). > **Warning** > This deployment script should *not* be run as a sudo user. @@ -12,15 +17,12 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt - RHEL8 (Tested with version 8.8) - `sudo` permissions: - To install the system-wide packages (listed below). - -- Python 3.6 -- Git -- Access to PyPI for downloading python packages. -- Access to GitHub for cloning several external dependencies. +- Python 3.8 - Several system packages, installed via `yum`:
- Package list + Package list + - unzip - bzip2 - libglvnd-opengl - libX11-devel @@ -42,12 +44,10 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt - perl-Digest-MD5 - tcsh - zsh - - root - - python3-root - epel-release - - python3-pip - libcurl-devel - - python36-devel + - python38 + - python38-devel - boost-python3-devel - protobuf-devel - jemalloc-devel @@ -56,60 +56,77 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt - lzo-devel - cmake - xz-devel - - python3-sphinx - openssl-devel - libjpeg-turbo-devel - libpng-devel + - gcc-c++ + - gcc + - binutils + - gcc-gfortran + - mesa-libGL-devel + - mesa-libGLU-devel + - glew-devel + - ftgl-devel + - fftw-devel + - cfitsio-devel + - graphviz-devel + - libuuid-devel + - avahi-compat-libdns_sd-devel + - openldap-devel + - python3-numpy + - libxml2-devel + - gsl-devel + - readline-devel + - R-devel + - R-Rcpp-devel + - R-RInside-devel + - xrootd-client
## Deploying [Legacy] DQMGUI -This procedure has been tested on a RHEL8 Openstack VM. +This procedure has been tested on a RHEL8 Openstack VM. Instructions below are primarily for a personal VM. -1. Install the system packages: +0. Download the latest build artifact and copy it to the machine you want to install it to: ```bash - sudo yum install -y tmux git bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel libpng-devel libjpeg-turbo-devel - ``` - 2. Add a non-privileged user, create and give access to necessary directories and switch to it: - + ```bash adduser dqm + + # Installation directory sudo mkdir -p /data/srv sudo chown -R dqm /data/srv + + # Data directory sudo mkdir -p /dqmdata/dqm sudo chown -R dqm /dqmdata + + # Installation package + sudo chown dqm /tmp/dqmgui_installation_package.tar.gz + sudo su dqm ``` 3. Start the deployment (`dev` flavor): - + ```bash cd ~ - # Clone this script and all necessary files, like patches - git clone https://github.com/cms-DQM/dqmgui_prod_deployment - + tar -xf dqmgui_installation_package.tar.gz -C dqmgui_deployment + # Start the deployment script, it will take some time to finish - bash /home/dqm/dqmgui_prod_deployment/deploy_dqmgui.sh + bash /home/dqm/dqmgui_deployment/deploy_dqmgui.sh # Start all the services /data/srv/current/config/dqmgui/manage -f dev start "I did read documentation" @@ -126,7 +143,7 @@ This procedure has been tested on a RHEL8 Openstack VM. firewall-cmd --zone=public --add-port=8081/tcp # relval ``` -## Command-line arguments +## Deployment command-line arguments This script uses some internal variables to specify options such as the main installation directory, or the git tags/refs to use when cloning repositories. It's not recommended that you edit those, as most of them are hand-picked so that the project compiles (we're mostly referring to the git refs). @@ -137,12 +154,10 @@ Below is a list of the values that you will most probably need to change to cust | Variable name | Description |Default value | |---------------|-------------|--------------| | `INSTALLATION_DIR` | The directory to install the GUI into. It should be writable by the user running the script | `/data/srv` | -| `DMWM_GIT_TAG` | The git tag/ref to checkout `dmwm/deployment` to. See [here](https://github.com/dmwm/deployment/tags) for a list. | `HG2903c` | -| `DQMGUI_GIT_TAG` | The git tag/ref to checkout `cms-DQM/dqmgui_prod` to. See [here](https://github.com/cms-DQM/dqmgui_prod) for a list. | `9.8.0` | -## [Debug] Selectively run parts of the installation script +### [Debug] Selectively run parts of the installation script -The script is split into steps (see: `installation_steps`), which can all be toggled off or on by arguments when running the script. +The script is split into steps (see the `installation_steps` array declared in the `deploy_dqmgui.sh` script), which can all be toggled off or on by arguments when running the script. The flags are named by concatenating `do_` with the name of the step, so, for example, `do_check_dependencies` or `do_install_rotoglup`. @@ -152,8 +167,16 @@ The are all set to `1` by default, and you can override them when running the sc bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 ``` -A useful combination that can be used when you've already downloaded all external dependencies once, and you want just to re-compile the DQMGUI part for testing: +A useful combination that can be used when you've already installed and built all steps once, but you only want to re-compile the DQMGUI part for testing: ```bash -bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0 +bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_root=0 do_compile_root=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0 ``` + +## Download script + +The installation package is created by running `download_dependencies.sh`. This is done automatically by github actions (see `.github/workflows/build_installation_package` in this repository). The versions of the packages downloaded are specified in `config.sh`. + +## Notes + +- We're not using the RHEL8 `root` package, due to the fact that they are built for python3.6, hence we need to build it with python3.8. diff --git a/config.sh b/config.sh new file mode 100644 index 0000000..a9e25cc --- /dev/null +++ b/config.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +# Configuration file, which is sourced by both download_dependencies.sh and deploy_dqmgui.sh +# Here we specify both the URLs and tags/branches/refs of the repositories that DQMGUI +# depends on. + +# Tag to use for getting the layouts and manage/deploy scripts +# See: https://github.com/dmwm/deployment/tags +#DMWM_GIT_URL=https://github.com/dmwm/deployment +DMWM_GIT_TAG=debug +DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment + +# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags +#DQMGUI_GIT_TAG=9.8.0 +DQMGUI_GIT_TAG=python3_backup +DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod + +# Boost.GIL. At most version 1.67!! The API changed radically after that. +BOOST_GIL_GIT_URL=https://github.com/boostorg/gil +BOOST_GIL_GIT_TAG=boost-1.66.0 + +# OLD rotoglup code. Commit was found with lots of pain, so that the patch +# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch +ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494 +ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad + +# Yahoo!(TM) UI +YUI_GIT_URL=https://github.com/yui/yui2 +YUI_GIT_TAG=master + +# Extjs +EXTJS_GIT_URL=https://github.com/probonogeek/extjs +EXTJS_GIT_TAG=3.1.1 + +# D3 +D3_GIT_URL=https://github.com/d3/d3 +D3_GIT_TAG=v2.7.4 + +# JSROOT +JSROOT_GIT_URL=https://github.com/root-project/jsroot +JSROOT_GIT_TAG=5.1.0 + +# ROOT +ROOT_GIT_URL=https://github.com/root-project/root/ +ROOT_GIT_TAG=v6-28-08 diff --git a/deploy_dqmgui.sh b/deploy_dqmgui.sh index 586eb72..c54650a 100755 --- a/deploy_dqmgui.sh +++ b/deploy_dqmgui.sh @@ -3,9 +3,10 @@ # Script intended for installing [legacy] DQMGUI on online HLT machines (dqmsrv-...). # It tries to imitate the behavior of the Deploy script without installing external RPMs. # -# It clones DQMGUI, dmwm/deployment code and classlib, creates a virtual env...TODO +# The installation depends on the steps defined in the installation_steps array. +# They are executed in sequence and can be skipped with the appropriate flag. # -# Only targeting RHEL8 + Python3.6 for now(!) +# Only targeting RHEL8 + Python3.8 for now(!) # # If the target (installation+dmqm tag) directory exists (e.g. /data/srv/$DMWM_GIT_TAG), it will # be *DELETED* by the script, before re-installing. The "state" dir is left alone. @@ -25,47 +26,16 @@ EUID_USER_DQM=40454 # Main directory we're installing into. INSTALLATION_DIR=/data/srv +# Where ROOT will be installed +ROOT_INSTALLATION_DIR=/data/srv/root + # This scipt's directory SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) -# Tag to use for getting the layouts and manage/deploy scripts -# See: https://github.com/dmwm/deployment/tags -#DMWM_GIT_TAG=HG2307b -#DMWM_GIT_URL=https://github.com/dmwm/deployment -DMWM_GIT_TAG=debug -DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment - -# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags -#DQMGUI_GIT_TAG=9.8.0 -DQMGUI_GIT_TAG=python3_backup -DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod - -# Boost.GIL configuration -BOOST_GIL_GIT_URL=https://github.com/boostorg/gil - -# At most 1.67 for boost/gil/extension/io/jpeg_io.hpp -BOOST_GIL_GIT_TAG=boost-1.66.0 - -# OLD rotoglup code. Commit was found with lots of pain, so that the patch -# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch -ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494 -ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad - -# Yahoo!(TM) UI -YUI_GIT_URL=https://github.com/yui/yui2 -YUI_GIT_TAG=master - -# Extjs -EXTJS_GIT_URL=https://github.com/probonogeek/extjs -EXTJS_GIT_TAG=3.1.1 - -# D3 -D3_GIT_URL=https://github.com/d3/d3 -D3_GIT_TAG=v2.7.4 - -# JSROOT -JSROOT_GIT_URL=https://github.com/root-project/jsroot -JSROOT_GIT_TAG=5.1.0 +# Get constants from config file +# This is needed to name some directories whose name is +# based on the version of the package (DMWM and DQMGUI). +source $SCRIPT_DIR/config.sh # Preliminary checks to do before installing the GUI preliminary_checks() { @@ -102,13 +72,16 @@ preliminary_checks() { # Check for needed OS-wide dependencies check_dependencies() { pkgs_installed=1 - declare -a required_packages=(bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel + declare -a required_packages=(unzip bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor - libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release - python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel - pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel - libjpeg-turbo-devel libpng-devel) + libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh epel-release + libcurl-devel python38 python38-devel boost-python3-devel protobuf-devel jemalloc-devel + pcre-devel boost-devel lzo-devel cmake xz-devel openssl-devel + libjpeg-turbo-devel libpng-devel gcc-c++ gcc binutils gcc-gfortran mesa-libGL-devel mesa-libGLU-devel + glew-devel ftgl-devel fftw-devel cfitsio-devel graphviz-devel libuuid-devel avahi-compat-libdns_sd-devel + openldap-devel python3-numpy libxml2-devel gsl-devel readline-devel R-devel R-Rcpp-devel R-RInside-devel + xrootd-client) # Instead of doing a 'yum list' per package, it may be faster to just # ask all of them at once, and dump to file. Then grep the file. @@ -168,9 +141,9 @@ create_directories() { mkdir -p "$dirname" done - if [ ! -L $INSTALLATION_DIR/$DMWM_GIT_TAG/apps ]; then + if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps" ]; then echo "DEBUG: Creating link $INSTALLATION_DIR/$DMWM_GIT_TAG/apps.sw <-- $INSTALLATION_DIR/$DMWM_GIT_TAG/apps" - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG/apps.sw $INSTALLATION_DIR/$DMWM_GIT_TAG/apps + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps.sw" "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps" fi # Create a "current" link to the DMWM version we're using, like how it was done @@ -179,57 +152,22 @@ create_directories() { rm $INSTALLATION_DIR/current fi echo "DEBUG: Creating link $INSTALLATION_DIR/$DMWM_GIT_TAG <-- $INSTALLATION_DIR/current" - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG $INSTALLATION_DIR/current + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG" "$INSTALLATION_DIR/current" # Directories for external source and lib files (e.g. classlib) - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src" + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib" # DQMGUI dirs echo "DEBUG: Creating subdirectory $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" } -# Helper function to clone a repo and checkout to specific tag -# Takes the following positional arguments: -# - URL of the repo, -# - A reference (tag, branch, commit) to checkout to and -# - The directory to clone the code to. -_clone_from_git() { - cd $INSTALLATION_DIR/ - # repo url - url=$1 - - # Which branch/tag to checkout to - ref=$2 - - # The path to the directory to clone to - dir=$3 - - # If tmp dir exists, remove it - if [[ -d $dir ]]; then - rm -rf "$dir" - fi - - # Shallow clone - git clone "$url" "$dir" --depth 1 - - # Checkout to ref - cd "$dir" - # Shallow clone misses remote branches, so - # fix that. - git remote set-branches origin '*' - git fetch --all --tags - git fetch --depth=1 - # DIRTY HACK - git checkout "$ref" || (git fetch --all --depth=50 && git checkout "$ref") - cd $INSTALLATION_DIR/ -} - install_rotoglup() { ROTOGLUP_TMP_DIR=/tmp/rotoglup - _clone_from_git $ROTOGLUP_GIT_URL $ROTOGLUP_GIT_TAG $ROTOGLUP_TMP_DIR + mkdir -p $ROTOGLUP_TMP_DIR + tar -xzf "$SCRIPT_DIR/rotoglup/rotoglup.tar.gz" -C /tmp cd $ROTOGLUP_TMP_DIR #patch -p1 < $SCRIPT_DIR/rotoglup/patches/01.patch @@ -241,25 +179,25 @@ install_rotoglup() { # Compilation step for classlib compile_classlib() { - cd $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3 + cd "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3" #INCLUDE_DIRS="$INCLUDE_DIRS:/usr/include/lzo" make -j `nproc` make -j "$(nproc)" CXXFLAGS="-Wno-error=extra -ansi -pedantic -W -Wall -Wno-long-long -Werror" # Move the compiled library in the libs dir - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/.libs/libclasslib.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libclasslib.so + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/.libs/libclasslib.so" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libclasslib.so" } # Classlib is needed both as a shared object and for its header files for DQMGUI compilation. install_classlib() { # Temporary directory to extract to - CLASSLIB_TMP_DIR=/tmp/classlib + CLASSLIB_TMP_DIR=/tmp/classlib-3.1.3 mkdir -p $CLASSLIB_TMP_DIR - tar -xf "$SCRIPT_DIR/classlib/classlib-3.1.3.tar.bz2" -C $CLASSLIB_TMP_DIR + tar -xf "$SCRIPT_DIR/classlib/classlib-3.1.3.tar.bz2" -C /tmp # Apply code patches I found on cmsdist. The 7th one is ours, and has some extra needed fixes. - cd $CLASSLIB_TMP_DIR/classlib-3.1.3 + cd $CLASSLIB_TMP_DIR for i in 1 2 3 4 5 6 7 8; do patch -p1 <"$SCRIPT_DIR/classlib/patches/0${i}.patch" done @@ -275,55 +213,52 @@ install_classlib() { !/^\S+: / && s{\S+LZO((C|Dec)ompressor|Constants|Error)\S+}{}g' \ $CLASSLIB_TMP_DIR/classlib-3.1.3/Makefile - if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3 ]; then - rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3 + if [ -d "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3" ]; then + rm -rf "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3" fi # Move the classlib files inside the installation dir, needed for compiling the GUI - mv $CLASSLIB_TMP_DIR/classlib-3.1.3 $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/ + mv "$CLASSLIB_TMP_DIR" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/" # Make a link so that DQMGUI compilation can find the classlib headers easily - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/classlib $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib-3.1.3/classlib" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/classlib" rm -rf $CLASSLIB_TMP_DIR } install_boost_gil() { - BOOST_GIL_TMP_DIR=/tmp/boost_gil - _clone_from_git $BOOST_GIL_GIT_URL $BOOST_GIL_GIT_TAG $BOOST_GIL_TMP_DIR - mv $BOOST_GIL_TMP_DIR/include/boost $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost - rm -rf $BOOST_GIL_TMP_DIR + mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/ + tar -xzf "$SCRIPT_DIR/boost_gil/boost_gil.tar.gz" -C /tmp + mv /tmp/boost_gil/include/boost "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost" } install_gil_numeric() { NUMERIC_TMP_DIR=/tmp/numeric - mkdir -p $NUMERIC_TMP_DIR - tar -xf "$SCRIPT_DIR/numeric/numeric.tar.gz" -C $NUMERIC_TMP_DIR - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/ - mv $NUMERIC_TMP_DIR/numeric $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/numeric - rm -rf $NUMERIC_TMP_DIR + tar -xzf "$SCRIPT_DIR/numeric/numeric.tar.gz" -C /tmp + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/" + mv "$NUMERIC_TMP_DIR" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/boost/gil/extension/numeric" } install_dmwm() { # Temporary directory to clone DMWM deployment scripts into - DMWM_TMP_DIR=/tmp/deployment - _clone_from_git $DMWM_GIT_URL $DMWM_GIT_TAG $DMWM_TMP_DIR - + DMWM_TMP_DIR=/tmp/dmwm + mkdir -p $DMWM_TMP_DIR + tar -xzf "$SCRIPT_DIR/dmwm/dmwm.tar.gz" -C /tmp # Move dqmgui-related scripts from DMWM to the config folder - mv $DMWM_TMP_DIR/dqmgui $INSTALLATION_DIR/$DMWM_GIT_TAG/config/ + mv "$DMWM_TMP_DIR/dqmgui" "$INSTALLATION_DIR/$DMWM_GIT_TAG/config/" rm -rf $DMWM_TMP_DIR } # env.sh and init.sh file creation. They're needed by other scripts (e.g. manage). _create_env_and_init_sh() { - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/ + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/" # init.sh contents. This is sourced by env.sh echo "export PATH=$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/bin:$PATH export PYTHONPATH=$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib/python${PYTHON_VERSION}/site-packages:$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib64/python${PYTHON_VERSION}/site-packages . $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/activate -" >$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/init.sh +" >"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/init.sh" # env.sh contents. This is sourced by the manage script echo ". $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/init.sh @@ -336,7 +271,8 @@ export DQMGUI_VERSION='$DQMGUI_GIT_TAG'; # For pointing to the custom built libraries export LD_PRELOAD=\"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/libDQMGUI.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libclasslib.so\" export LD_LIBRARY_PATH=\"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/:$LD_LIBRARY_PATH\" -" >$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/env.sh +source $ROOT_INSTALLATION_DIR/bin/thisroot.sh +" >"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/profile.d/env.sh" } # Crete the Python3 virtual environment for the GUI @@ -345,31 +281,48 @@ _create_python_venv() { python_venv_dir=$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv mkdir -p "$python_venv_dir" - eval "$python_exe" -m pip install --upgrade pip --user - eval "$python_exe" -m pip install virtualenv --user + + # Extract the downloaded python packages + tar -xzf $SCRIPT_DIR/pypi/pypi.tar.gz -C /tmp echo -n "INFO: Creating virtual environment at $python_venv_dir" - virtualenv $python_venv_dir + $python_exe -m venv $python_venv_dir # Now use the new venv's python python_venv_exe=$python_venv_dir/bin/python - cd $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/ - eval $python_venv_exe -m pip install -r $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/requirements.txt - echo "Done" # Needed for specifying the PYTHONPATH later - export PYTHON_VERSION=$($python_exe --version | cut -d ' ' -f 2 | cut -d '.' -f 1,2) - export PYTHON_LIB_DIR_NAME=lib/python$PYTHON_VERSION/site-packages + PYTHON_VERSION=$($python_exe --version | cut -d ' ' -f 2 | cut -d '.' -f 1,2) + export PYTHON_VERSION + PYTHON_LIB_DIR_NAME=lib/python$PYTHON_VERSION/site-packages + export PYTHON_LIB_DIR_NAME + + # Install pip + unzip -u /tmp/pip/pip*whl -d /tmp/pip/pip + if [ -d "$python_venv_dir/$PYTHON_LIB_DIR_NAME/pip" ]; then + rm -rf "$python_venv_dir/$PYTHON_LIB_DIR_NAME/pip" + fi + + # pipipipi + mv /tmp/pip/pip/pip "$python_venv_dir/$PYTHON_LIB_DIR_NAME/pip" + rm -rf /tmp/pip/pip + + # Install wheels + eval "${python_venv_exe} -m pip install --no-index --find-links /tmp/pip /tmp/pip/*" + eval "${python_venv_exe} -m pip install $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128" + + cd "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/" + rm -rf /tmp/pip + echo "Done" } # External requirements for building the GUI # Must be run after the venv is created _create_makefile_ext() { - echo "INCLUDE_DIRS = . $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp /usr/include/root /usr/include/libpng16 /usr/include/jemalloc $(python3-config --includes | sed -e 's/-I//g') $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src /usr/include/google/protobuf /usr/include/boost - -LIBRARY_DIRS = . /usr/lib64/root $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib /usr/lib /usr/lib64 $(python3-config --ldflags | sed -e 's/-L//g' | sed -e '[[:space:]]-l[a-zA-Z0-9\.]+') $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/ $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/ -" >$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/makefile.ext + echo "INCLUDE_DIRS = . $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp /usr/include/libpng16 /usr/include/jemalloc $(root-config --incdir) $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src /usr/include/google/protobuf /usr/include/boost $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/root +LIBRARY_DIRS = . $ROOT_INSTALLATION_DIR/lib $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib /usr/lib /usr/lib64 $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/ $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/ +" >"$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/etc/makefile.ext" } # Compile and setup all the required stuff that DQMGUI needs. @@ -377,18 +330,26 @@ LIBRARY_DIRS = . /usr/lib64/root $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib # Then runs the actual compilation, which is the part that takes the longest # in this script. compile_dqmgui() { - cd $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/ + cd "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/" # Links to python libraries so that the build command can find them - if [ ! -L $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so ]; then - ln -s /usr/lib64/libboost_python3.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so + if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so" ]; then + ln -s /usr/lib64/libboost_python3.so "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libboost_python.so" fi if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libpython${PYTHON_VERSION}.so" ]; then ln -s /usr/lib64/libpython3.so "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/lib/libpython${PYTHON_VERSION}.so" fi + # python3-config is not always in a predictable place + python_config_cmd=$(which python3-config) || python_config_cmd=$(find /usr/bin -name "python3*-config" | head -1) + + if [ -z "$python_config_cmd" ]; then + echo "ERROR: Could not find python3-config" + exit 1 + fi # The actual build command. Uses the makefile in the DQMGUI's repo. - PYTHONPATH="$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib/python${PYTHON_VERSION}/site-packages:$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib64/python${PYTHON_VERSION}/site-packages" CPLUS_INCLUDE_PATH="$(python3-config --includes | sed -e 's/-I//g')" $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/python $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/setup.py -v build_system -s DQM -d + source "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/activate" + PYTHONPATH="$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib/python${PYTHON_VERSION}/site-packages:$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/lib64/python${PYTHON_VERSION}/site-packages" CPLUS_INCLUDE_PATH="$(${python_config_cmd} --includes | sed -e 's/-I//g')" $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/bin/python $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/setup.py -v build_system -s DQM -d # Stuff that I found being done in the dqmgui spec file. I kind of blindy copy paste it # here because reasons. @@ -396,12 +357,12 @@ compile_dqmgui() { # Move executables to expected place for exe in DQMCollector visDQMIndex visDQMRender; do - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/$exe $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/bin/$exe + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/$exe" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/bin/$exe" done # Move libs to expected place - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/ - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/libDQMGUI.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/libDQMGUI.so + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/" + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/libDQMGUI.so" $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/lib/libDQMGUI.so # Move the custom Boost.Python interface library to libs. mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/cpp/Accelerator.so $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/build/lib/Monitoring/DQM/Accelerator.so @@ -412,29 +373,31 @@ compile_dqmgui() { # The resulting directory structure and compiled binaries is a mess, but that's the best # we can do right now, considering the existing mess. install_dqmgui() { + # Activate ROOT, we need it to be available so that we can run root-config later + source "$INSTALLATION_DIR/root/bin/thisroot.sh" + # Temporary directory to clone GUI into - DQMGUI_TMP_DIR=/tmp/128 - _clone_from_git $DQMGUI_GIT_URL $DQMGUI_GIT_TAG $DQMGUI_TMP_DIR + DQMGUI_TMP_DIR=/tmp/dqmgui + tar -xzf "$SCRIPT_DIR/dqmgui/dqmgui.tar.gz" -C /tmp # Move dqmgui source and bin files to appropriate directory - if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG ]; then - rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG + if [ -d "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" ]; then + rm -rf "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" fi - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128 - - mv $DQMGUI_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG + mkdir -p "$DQMGUI_TMP_DIR" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/" + mv $DQMGUI_TMP_DIR "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128" - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/data - mkdir -p $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/ - if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates ]; then - rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/data" + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/" # Needed for DQMGUI templates + if [ -d "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates" ]; then + rm -rf "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates" fi - mv $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/templates $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates + mv "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG/128/src/templates" "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/venv/data/templates" - if [ ! -L $INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui ]; then + if [ ! -L "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui" ]; then echo "DEBUG: Creating link $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG <-- $INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui" - ln -s $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG $INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui + ln -s "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/cms/dqmgui/$DQMGUI_GIT_TAG" "$INSTALLATION_DIR/$DMWM_GIT_TAG/apps/dqmgui" fi # Create python venv for all python "binaries" and webserver @@ -448,28 +411,58 @@ install_dqmgui() { _create_makefile_ext } +# Javascript library install_yui() { - YUI_TMP_DIR=/tmp/yui - _clone_from_git $YUI_GIT_URL $YUI_GIT_TAG $YUI_TMP_DIR - mv $YUI_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/yui + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/yui" + tar -xzf "$SCRIPT_DIR/yui/yui.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" } +# Javascript library install_extjs() { - EXTJS_TMP_DIR=/tmp/extjs - _clone_from_git $EXTJS_GIT_URL $EXTJS_GIT_TAG $EXTJS_TMP_DIR - mv $EXTJS_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/extjs + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/extjs" + tar -xzf "$SCRIPT_DIR/extjs/extjs.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" } +# Javascript library install_d3() { - D3_TMP_DIR=/tmp/d3 - _clone_from_git $D3_GIT_URL $D3_GIT_TAG $D3_TMP_DIR - mv $D3_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/d3 + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/d3" + tar -xzf "$SCRIPT_DIR/d3/d3.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" } +# Javascript library install_jsroot() { - JSROOT_TMP_DIR=/tmp/jsroot - _clone_from_git $JSROOT_GIT_URL $JSROOT_GIT_TAG $JSROOT_TMP_DIR - mv $JSROOT_TMP_DIR $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/jsroot + mkdir -p "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/jsroot" + tar -xzf "$SCRIPT_DIR/jsroot/jsroot.tar.gz" -C "$INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external" +} + +# Extract the ROOT tar to a tmp folder for compilation +install_root() { + tar -xzf "$SCRIPT_DIR/root/root.tar.gz" -C /tmp + #if [ -d $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/root ]; then + # rm -rf $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/root + #fi + # + #cp -r /tmp/root $INSTALLATION_DIR/$DMWM_GIT_TAG/sw/external/src/root +} + +compile_root() { + ROOT_TMP_DIR=/tmp/root + ROOT_TMP_BUILD_DIR=/tmp/root_build + if [ ! -d $ROOT_TMP_DIR ]; then + echo "ERROR: ROOT source was not found in $ROOT_TMP_DIR" + exit 1 + fi + + if source "$ROOT_INSTALLATION_DIR/bin/thisroot.sh"; then + echo "INFO: ROOT installation found, not re-compiling ROOT" + return + fi + mkdir -p $ROOT_TMP_BUILD_DIR + cd $ROOT_TMP_BUILD_DIR + cmake -DCMAKE_INSTALL_PREFIX=$ROOT_INSTALLATION_DIR $ROOT_TMP_DIR -DPython3_ROOT_DIR=$(which python3) + cmake --build . --target install -j $(nproc) + cd $INSTALLATION_DIR + rm -rf $ROOT_TMP_DIR $ROOT_TMP_BUILD_DIR } ### Main script ### @@ -485,6 +478,8 @@ declare -a installation_steps=(preliminary_checks install_classlib compile_classlib install_dmwm + install_root + compile_root install_dqmgui compile_dqmgui install_yui diff --git a/download_dependencies.sh b/download_dependencies.sh new file mode 100644 index 0000000..f3253cd --- /dev/null +++ b/download_dependencies.sh @@ -0,0 +1,119 @@ +#!/bin/bash + +# Script that downloads all online dependencies for [Legacy] DQMGUI +# in order to install them at P5, where there is no access to neither GitHub, +# nor PyPI. This script is to be called whenever those packages are to be recreated, e.g +# + +set -ex + +# This scipt's directory +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) + +# Get constants from config file +source "$SCRIPT_DIR/config.sh" + +# Helper function to clone a repo and checkout to specific tag +# Takes the following positional arguments: +# - URL of the repo, +# - A reference (tag, branch, commit) to checkout to and +# - The directory to clone the code to. +_clone_from_git() { + # repo url + url=$1 + + # Which branch/tag to checkout to + ref=$2 + + # The path to the directory to clone to + dir=$3 + + # If tmp dir exists, remove it + if [[ -d $dir ]]; then + rm -rf "$dir" + fi + + # Shallow clone. We don't use --branch as we may need this to work + # for a specific commit, too. + git clone "$url" "$dir" --depth 1 + + # Checkout to ref. + cd "$dir" + # Shallow clone misses remote branches, so + # fix that. + git remote set-branches origin '*' + git fetch --all --tags + git fetch --depth=1 + # DIRTY HACK + git checkout "$ref" || (git fetch --all --depth=50 && git checkout "$ref") + cd - +} + +download_repos() { + for repo in "${repos_to_download[@]}"; do + download_repo_flag_name=do_download_$repo + if [ "${!download_repo_flag_name}" -eq 0 ]; then + echo "INFO: Skipping downloading $repo" + continue + fi + + echo "INFO: Downloading $repo" + + prefix="$(echo $repo | tr '[:lower:]' '[:upper:]')" + git_url=${prefix}_GIT_URL + git_tag=${prefix}_GIT_TAG + if [ -z "${!git_url}" ] || [ -z "${!git_tag}" ]; then + echo "WARNING: git url or tag not configured for repository $repo, skipping" + continue + fi + + temp_dir=/tmp/$repo + mkdir -p "$temp_dir" + _clone_from_git ${!git_url} ${!git_tag} $temp_dir + rm -rf "$temp_dir/.git" # Not needed, and will slow things down + mkdir -p "$SCRIPT_DIR/$repo" + if [ -f "$SCRIPT_DIR/$repo/$repo.tar.gz" ]; then + rm "$SCRIPT_DIR/$repo/$repo.tar.gz" + fi + tar -cf "$SCRIPT_DIR/$repo/$repo.tar.gz" --directory=/tmp "$repo" -I "gzip --best" + rm -rf "$temp_dir" + done +} + +download_python_packages() { + declare -a pypi_requirements=(pip sphinx wheel cherrypy jsmin pycurl numpy matplotlib pillow Cheetah3 virtualenv) + PIP_TEMP_DIR=/tmp/pip + mkdir -p $PIP_TEMP_DIR + mkdir -p "$SCRIPT_DIR/pypi" + python_exe=$(which python3) + eval "$python_exe -m pip download ${pypi_requirements[*]} --destination-directory $PIP_TEMP_DIR" + tar -cf "$SCRIPT_DIR/pypi/pypi.tar.gz" --directory=/tmp pip -I "gzip --best" + rm -rf $PIP_TEMP_DIR +} + +### Main script + +# For GitHub repos, clone the source on a specific branch/tag/ref, then make a tar. +declare -a repos_to_download=(rotoglup boost_gil dmwm dqmgui yui extjs d3 jsroot root) + +# Create dynamic flags to selectively disable/enable steps of the download procedure +# Those flags are named "do_download" with the name of the repo, e.g. "do_download_root" for +# the "root" repo. +# We set those flags to 1 by default. +for repo in "${repos_to_download[@]}"; do + eval "do_download_${repo}=1" +done + +# Parse command line arguments -- use = to override the flags mentioned above. +# e.g. DQMGUI_GIT_TAG=9.9.0 +for ARGUMENT in "$@"; do + KEY=$(echo "$ARGUMENT" | cut -f1 -d=) + KEY_LENGTH=${#KEY} + VALUE="${ARGUMENT:$KEY_LENGTH+1}" + eval "$KEY=$VALUE" +done + +download_repos +download_python_packages + +echo "INFO: Done!" diff --git a/numeric/numeric.tar.gz b/numeric/numeric.tar.gz deleted file mode 100644 index 063b86136aa2f51a49d6cae15bdd0a236ac66456..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8488 zcmV+@A=lm?iwFSq%Irn}1MEF(Z`-)C{am2`fx8zdg7l5H0g=Z#u7yihn(SXI2?}r$(Tl-v;Ag`R|~Ga zR*U_9?8x6X`xV!lcI)HbPW!`ddk50noxPo%H)MC6NXcswCw4^0n^8ZQL@vFl$~XQV z+HrjAJ_Sxsc=>|<_F&-ov^fmJ)z^Up-mRwp-cDyn(tqdU?gya%hrM>^4QZ_&mE!du zPXFzVcW((f4#H{VT@Mq|0a9(jpFMKq27O8{r*T5ZF**00APR%XPN<6|PP{mYy#53p z$;5YQM3NyTpUkYSR9``g<=2w;FF4sF_qN)sS$k2vtSIZVc*#=Ey0+wb1Kd%L}{?u%9P zU)}&(Iw1cz|70DVot=LP4>Ntvlb>SZJWjy$oPcCT)+55m?owkXu zKC;Z~?5lTgeS1vf&~_;42SJ>Wdy=JGdn0ncc4F}!x%9>%kPJFM^6p3y5H}#5lM322 z$$3J&7{**Ou}7mR*}2;xFhbeL9NEweCm2n}KIZweMOvi&_u9J>NEeQfst3sw#xryx zxjKCJ)&X>4(X5@j4)$j_>BHdOM`=K^H%5c(6WeGvOwu=6@Yi8~U6w-G{Q&-w`-gnN zD{^F#EAkuuImSOH_~#V=oN=H>4YP0j)-WC8xM8}+NyDVZX~P^CXJzml=;0)Levh{t z4%{?th9KVc8IpMCFhudrWyslbb_sUGjjeTT$!AjB4+R!JsrqPu^{eD=UDB zHSrvU%GWfpq5x7UP$#n8SZ$H0;0l@&C+_~f6T}9~(}lE{r^TKx(?p#LeazV0icQCC znXLxHq7nx-ZjdWQYq|VAxH^1KfTZ_D*lj`(tckx#^A7$`=dR+4+#%0Rb?Vht0R_mE zkQr6Pj?JZcu@!M?D`LPLtB9LZvxU_yP{o|BK+Lg7_ubovF1{%hLu|f2`+9MFdj91TOxo|>rnNvXW`iO>CRi7bVKORB?=(=F zAu4&);l07%sMomNw8;jPgP%=&>hmYZtjpWi^19`D9SyEq3)j)$y0vhr26rIf;!~$d zYm4-?A}%j700IJ`s_0~r^!}`bW`ga{iUd;zL|7!&*&&l{I;~P*1_;-Uuz80VHXXQ| zvO8+RpMLtnO%WVm9G()ft){M6TyUpT?v= zWp7HghqWTUQA*j*c6jEaG7$IbJ&#%HVV zUIZj_#*%0&|_t|m^BYipG#JA+OvEpVW+J2_Ol64x==&`~?DMrNL=W3f{^ z7_*Kj9e8#kdqx)N9P-}DWIjBhoUjui-XF*VTQ234^Zk9$$k}3x`mQ&4_x49GsbAEU z+J8pZLF6UF@fu7(_4c3k?%wWh&i=Fe;bZI7{__+UxBo0-0a~>U=p|Z#zWmSU(~I+C z>*&)zzFwSP{r9UNS>_8Lqa+=pIWvr6azP7B-YT;aGt4KnA521g<&wZB zXS5$p>}X0pGNa6%BtmNOI&fgP&?q@n)4YVMXj)?0EG<&Of zhS0mCBTLp*k!eg5dmvPeSmc@C@FPic!KNy%aYb|h$^xheB#eSPVB$Cq9FKL_Ei?~! zo2v+@tPSlLp8pETuW6}Y$(Y-@#Pr-$>_Ar~;JtjUSk7byA66ZX%@81M1(0&QxtZl; zhN5xc(osylVP6TwrqWy^lndUky!L&<4K@IE9NbVV1+J7trA(xP?!mPRrSI619#jO2 z+Rl*M97$*c;|2pFi5HjRbS~bu%|cQqOZpAe-e@Y-Noa3Jlb+%-pjDN|dRVcH<;%UD{(?s!=@qn zWQ(c!f{vL0aY5v88IsXf*tPxzRgp{h$wr&!`!qttU3jv-TiHq|EzHfR*0~=}l59+n zvgA&hd=(b@QoP|@{go}Ffvq!0xfh*7&7cDi2`oECQzM57u`Ds5NCowBILZQmECRJG z&-aq*!g+k5K%XU9LD;Thm_lRJQkn6FWo7k~o(9*DSgFN~jjXBiw$qR!Tj7!dQ4f&l zTW#bqNmkL90^nxblmOW+t9-%1ptWD^Bfaec-G|}|u!buST8FQUS_n*$Og$CV4yyOP`$*=O>=QYY`MXZJ^R z3%QPbr;$%brK%9qTaO%PGB)egqkl3O#q@JOq1SW~JxZcWg(_wFd1cD#(`(N!RjKCf zWF1}X0RdFt%TnU}SggBp${YvR$lC%}ud$JD=l^^9$-2Bawk}T3x{t1IhI%HdJGs27 ztF9EX-sc){!Gh8=2}O~f)n%r4n<*boAJoT1HBdbqFf*@?@1u;{>;WQ95}siD`naWPN5x#O}t0fU;$b zHe{RdHZ7ui{o&DDC4rUK~2WHC=B^eO&RxMx)n*Oj=4~p+PQiTCL;|4VYIxcz!~L6gO*d5H|U(A=@zsA_H?o zQLojFq67ZAySW)f+E^_%=`OXI6ECpVI`ieKrTY#$8QMF+W*~Yebo1VJMGr)-s}GFQ(Uk6zmM7fJ^uaZ z%a_w^llSYl(~G04^RHhniyOhS@-DI?e94TllZeJ_-%JeNty>TJT#R6njo=@@Tbw!Z z@3a&resVm4wTPdLri6EtD;%G&W&0@}QZ{ilju@NhVlC3fp3f7vP`2g@C^8#zY$(NC z@5Xa^@^(o%vvst8c3Mm)aMoEro)BSR`mJ?P|NsoiHiWG7RQ0g)=A$4;Fc>mt zf!yNer(}Yy6_Q@nCDg&M=oK%9^d(EypOXHHj<7;bNHv6X%;R~xY#-%iFhRj8@pE#( z_BuEKO*I+={0O2KNt#-abW%>rDJj7ej$Kw#E2Wb#;x0mImB!VSbB5c!PP4VT`5Jmu zAE_X*(V+G`2qsN?FPL6EQN(DiP{n4p&1b)8FVZtJ$}YM7>dZ^s&SS_s_d;zEB0ZbI@hyYi=|J&*8?dIdZ?e4(q*ZtooxL)^vAGQDc z^~)b$KmAb>2Tp%e*xS{1b2}%XlAxUKk1${d4#WwY2H6I!;#(RkM?#zW20@A$=6 ztL}`VJsypv*CWeM>`_E*cS>quYq-q;dnU3>6|((gUV!*cQiL$~K&cy*ohwD4C~QF_ zvy!S`dcd64xlby0QfJ4_*67xfeahh%=&Pk_2%bBeHq4w;C0}!7 zD~!yS_SSQgA@eh3cTlw2yY|t3^xl9Nyh5P8jugJ)EZOVTN?Y>pf0G#*S(T;kz9qC~ z-s5Uh=5v)zJ@I96wY$@H9-udJBcpr4J9my|GvLKx3aZpd#x+0@P=?0?fwweYCtHAX zB}1hb=}!i*a^;BUJ#y!Dtd^G^ccI`)ZRQ-C&G)?W<{%3E#AxKJFDK$+KG%=$^0f0< z-}?ry6puPIM|6-_oW`Y;NGoSWHhZ};B{?&hYvej7-&vcCVw>%)jL+(ia;So$+yPrf zwFvzYomz2ZxT06tqHI>TJ``QU3v53R%Wq~4+-j?pEW-O4@D1!h4!@TpShpYp#8uga znzpyhfZNJ5>8!IpxdFOaAjd2KY3mF0;?$|*7)1dsZETKRcs+%2n4f)0v72WoE-K6_ zo1Uwgv0X`)>gH(_sA}K*9V`{{(oR-pd&GrCC72dwOzlue?lefT<2c;1UYd`>)#aZx zeHZDqT-|?ewf|nK+ceH82k@XjE;<2KO9XYksmxTEx-~^cJ@Zq$NJ|5(XXsl#i?JSg zH&93fSyvAsvtH@uqOhyQ4K+9W!Q;65yr7XL$dKmq2Ctg$6f;6(3r1x(o5etDvt!AHEZD)~YD!XZ$x4YJs=ab^E0Bj5Kzg#G^z0@cxnA2K%QB~lf6 zY9sE{+Uin_VBy<7DUE$X>`}l1JU@Y@rYyKliJ4~QCYl-MnYlD084RarQ7YI75IICf z7=Q95QP^o^l$EO<(q?s}<=^G_;nW2*o+ov6s%q3>ai+r{rY!8f{r{fidiDQ%jQ`(1 zPcJZ}OTqh3P0#!OX|8|ldtM%Q)Zz|0ioivGzOqFO_Y34Rg8NM4ymUUwI4_0of~?vx za7@QRG}V%r0Nl2t$YySS%Qy_`pX#04L9fg!OQX1)MO`~YQzd%m1wP$M-G%HCPWMz0 z^BBE7d`eyB9}`9a9?v3T3q$TF;JygX3f%hn=ijDIS^dhqK1}I3z{BXJjs#}vy>I5u z1C$?y02I?52jTV8b}iY!^vnZmO8q7TjOv8#v1z920!;Q%Sdm%%_$$3b&mk`vMBx`DltZl9PdM29Omg>+DXM9W#t z-}P$ou_+#SzgB=ReyO&yrhC1y`}2aAQ_dDPoLF znq+B{e8HJq zCzh|ZGa1h^o4zINwgA&rRy>;rXQMNl&G@GFJUphY_;aZWzs<*9!gKxb)aMX2k#9fj_zUzpdTf-B__!&@7}_iJs`F==^biwXZkuu)iV}dlvLJpSe8nZd$S=>TUrJ*H&XD)Yi7++75fMsY z+LrgE`m29(IiI!wGM7cn5VfQ_oJ<#+@NAz5BbFFYRoF3kzWM_STHHV7o2_YB4@fvj z8?nrHSnEM{Rt-}9P;0?zr1e~RG@6BZS`Ga7GcdN4t3L~8i}tL8wb#Mh&x^U$(1vPM zQ^qQ_JiZvOir8^PWz{T{BIKIE;R`W%iFQ!V?4-!;z`u`*LpML8&&A|2hLqcZwSG+y z#~Iwe+1%3FZ8V&i@J>5`P-8gs&gP>E( z{fE`uukm8K?$US=IZb0jg#k<0O0W-LYEj!60{7(OV9v(`<@0Q~V*N1+3~tFZm=MSIG3t<)>{s`V z3^#+QSu%h>Zy;;ky|k-8qD^{(goS-x?r>8tAo20V^4OL9TTgb(c?L74Au$yWQo~`c zB-Uj%i!HTuxtNzPGb>7tGiS_0=;m1~Xj3@i3z0Lq86l!qXd!~<8Yhr?$o1-f_Hn?4 zBWFtcKTH)iGTYO~e{Ao5Xzk_WKX&#$yxRYt;^OxI#jXFVwf(<5%m2mc< z>Til(i6eX5cWv>L_fFLFpgvpSzsS7I(O+=eox&Ig>Bao-^(V_76)ng108vd%zuRlN z=G@JQCn0bZpO3Ox4{#?!A_3!Q+Y(>CUY(*EfGZkkj>mUF=pB<@7~Giz{%G1H=Lx=p zxsdtkk73$#`F@Ey+dcK}$X6!`3SU87Ck^H|?XRxR>@S<4_`D(x-S4vik2uK+RlxI^ z;7~tNjRze-wc!VZ*b0W7Fd>p1h}=&h!HtkR>1vVU!H^(w5f+VeAGSuoDtaqxajDw$ z*$&VAL$R?%#&#G2?jlw;xi9#jTt0(-sXrLRv*5@Re#mZKKfMj2?|eLVPQ)li*fATD zrJZ!in1dNR*aa1Q=9;2-b6jUA5od^sg)d<62IC{gp>fd%-H_$w|PzK`r}~a}14o;#rBwR}3&0 z1|GUkF*vR_j(f5mEy87Yp5kFXmp&*Ccz2<16EZiCOo+`5yV4ICS<7Icn$upZOT_Jg zM6Nl-&t;3$8Q~9=$a#0LAnK(ZsD$8=kn~7s5{0`Ag}V%ey9|Z9LQpB*^F}P$KBH!4 z@KPzp;51Cc@J|8r@5-P+PbkO9?U4HHLkzf%gaxNI-0cV?)WqL{c^77A9$(ADsj*4c zB!A0}T`;oOh@9Lol|}$2u5-<`27cx-D!$v0Zlha4|Njn(1rB6ILJAzJyui{3u`*gK zM3uwT5^SO^`0~{lm7`q<4i4kJ9Q5qaLzeCT+dJ0Y#%c^CKzIdRTUvzAnS80%>B&dXN-Ji4tj08{Pv@g@U56 zz5`GNPG-WTo14P9xG6mZ!IYonBU%+?Q(&t#mo`n-*5W{cq_%@*E7P-^GySECGo9P4}PW)^&V4 z!mXX4wM`N~ilndL%erekxbArfzDC}_l5M1YruSuiLkm6EJ$ZNbIr2tQLot;Id${Te z_=ND*d1bOo#xbC~OGMzfBG`gjQ`y>c{9bomWbcUI>#l`MAG&+Pm&Y&G_Y(M(bGrL@ z{F00^<}a@9T%22CYY&kQJdzM&^IG?j5;wcdZmxJ(M-$xCD^LaRw?2y~Oj39`)k1%o zvSxjMLy$R}SMhDg8Nayy)F|FdJ6!l;N z|9sWHlKuq4`?#UAOtMvguX4~Khd{m$*5V85tMln1is(zc`C62SGjRZeoFxbD;N-&f z0NyyDhOTsuxhBQgr&hiCv5-ogyZva?e89Fw&po~Dl!f=&p>6z(96XYOO@Et*DurD*E*G1}*XzRhj_ikADpf>7}uBYqj$&Gq) zZ#~F9s{4a(SXCr|w*TGgKH$3?GJs!}ug~LbHQl`5^tdcOfhsi6OC?TD?vhL%g+Lk& z+C{&Qdr>D8ZQU1q&=QB+YeP1Z1R5d78W^Do-(zE-t;4O=N;VL*i9jAvUJpuR{7BP@ z9G^zwksmzLfIF!Es#!ld5nluhtJaKw{o7>S|sht zM!$Xf^!dZ((CP-->CQi>cSGDXTj`=FC%|_`+TbS!109IY&XUanVb0jtjjiPj=+K09RU36M zClhV2e>NU|F0;#8^8rFWL+T9r2Fc5=FWFbJI5zc(YjF zROT(ysq7BM<13m7`7O_RNPu@@`wdYUlX$jHcx}L8GGc%iAG(|@>MVz=v5Ho1uaVTl ziZ~br;8igNd;OlWZ#l+lE54!=Y29eE6vj{eG=PG8c>^<#+>5#gM*ZU@?qrYFI#Nx* zqxq%BsSZ~2DAiAMjFsnYZB!U%sr_#CJbnI?$VUf3iEnSo4miJ`;chY4&OMRE#ppqS zqoR1A$~}}{`S`dfdwDgw`s?B&mK2AgYAY15 z89K1FT=ZU4LYn(mImin&JO-_-T=AdX&S_Vnt*@vdkd+fTX?@Q*b>1gtOU+}@DYK+5 zt?o(}Md`y=T)*PGl1Fe=;N%)`Fsh%SuQqc1>t@{QxP^}jN3*NYR2rLB{Y;Ts;_;W|z`it1PRk;A*$JkT8K}f0ei7lO zW`j7oOID^8B}0spD1@F>n`oFNb19{y3+R9q~9t%~HTh*yVEeVgNoZJf^M zFn>VaC8CWdqb+wj0jaS@+?c|ZMlayPFIU$#f}1gDA@M_)-HgRxGuqY2PR_TLNbN_ z0y_3_hVDKO_6fe-3ff}PMIOW#-;$M*{Gc?%bbFL!zlb4+PVJJ10JTA)_SLkb0qiZA zU#;up#>#KL40X1ao6