Making it work offline

Author: nothingface0

.github/workflows/build_installation_package.yaml

@@ -0,0 +1,42 @@
+name: Build DQMGUI Installation Package
+
+on: [push]
+
+jobs:
+  build_and_upload:
+    runs-on: ubuntu-20.04
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - name: Download and compress dependencies
+        run: |
+          sudo apt update && sudo apt install -y libcurl4-gnutls-dev
+          $(which python3) --version
+          bash download_dependencies.sh
+          rm -rf .git
+          mkdir -p /tmp/dqmgui
+          tar -cf /tmp/dqmgui/dqmgui_installation_package.tar.gz . -I "gzip --best"
+      - name: Archive package artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: dqmgui-installation-package
+          path: |
+            /tmp/dqmgui/dqmgui_installation_package.tar.gz
+      # Create a dummy release, so that the artifact is downloadable
+      # without a GitHub account
+      # This will automatically update the artifacts of the "latest" release,
+      # even though the "latest" tag does match with the actually latest commit.
+      - name: Release
+        uses: ncipollo/release-action@v1
+        with:
+          artifacts: "/tmp/dqmgui/dqmgui_installation_package.tar.gz"
+          body:
+            "Automatically generated dummy release for making the installation
+            package publicly available. The `latest` tag is
+            put on a random commit, and does not reflect the current state of the release,
+            while the artifact of this release is updated with the actually latest files,
+            on each push."
+          tag: "latest"
+          allowUpdates: "true"

README.md

@@ -1,8 +1,13 @@
 # [Legacy] DQMGUI Deployment
 
-This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.6 for the needs of moving DQM production to new machines.
+[![Build DQMGUI Installation Package](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml/badge.svg)](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml)
 
-It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while this script downloads and builds all requirements from source (hence takes longer to deploy).
+This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.8 for the needs of moving DQM production to new machines.
+
+It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while the method we implement in this repository depends on two steps:
+
+1. Download all the external resources needed (e.g. python packages from PyPI, github repositories) and compress them. This is done automatically with GitHub actions, and you can download a package ready to install [here](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml).
+2. Copy the archives to the P5 machine and extract, then build from source (hence takes longer to deploy).
 
 > **Warning**
 > This deployment script should *not* be run as a sudo user.
@@ -12,15 +17,13 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
 - RHEL8 (Tested with version 8.8)
 - `sudo` permissions:
   - To install the system-wide packages (listed below).
-<!--  - To create a soft link in `/usr/lib64`. -->
-- Python 3.6
+- Python 3.8
 - Git
-- Access to PyPI for downloading python packages.
-- Access to GitHub for cloning several external dependencies.
 - Several system packages, installed via `yum`:
   <details>
-  <summary>Package list</summary>  
+  <summary>Package list</summary>
 
+  - unzip
   - bzip2
   - libglvnd-opengl
   - libX11-devel
@@ -42,12 +45,10 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
   - perl-Digest-MD5
   - tcsh
   - zsh
-  - root
-  - python3-root
   - epel-release
-  - python3-pip
   - libcurl-devel
-  - python36-devel
+  - python38
+  - python38-devel
   - boost-python3-devel
   - protobuf-devel
   - jemalloc-devel
@@ -56,60 +57,77 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
   - lzo-devel
   - cmake
   - xz-devel
-  - python3-sphinx
   - openssl-devel
   - libjpeg-turbo-devel
   - libpng-devel
+  - gcc-c++
+  - gcc
+  - binutils
+  - gcc-gfortran
+  - mesa-libGL-devel
+  - mesa-libGLU-devel
+  - glew-devel
+  - ftgl-devel
+  - fftw-devel
+  - cfitsio-devel
+  - graphviz-devel
+  - libuuid-devel
+  - avahi-compat-libdns_sd-devel
+  - openldap-devel
+  - python3-numpy
+  - libxml2-devel
+  - gsl-devel
+  - readline-devel
+  - R-devel
+  - R-Rcpp-devel
+  - R-RInside-devel
+  - xrootd-client
   </details>
 
 ## Deploying [Legacy] DQMGUI
 
-This procedure has been tested on a RHEL8 Openstack VM.
+This procedure has been tested on a RHEL8 Openstack VM. Instructions below are primarily for a personal VM.
 
-1. Install the system packages:
+0. Download the latest build artifact and copy it to the machine you want to install it to:
 
   ```bash
-  sudo yum install -y tmux git bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel libpng-devel libjpeg-turbo-devel 
-  ```
-<!-- 
-2. Create a link to `libDQMGUI.so` which we will be compiling shortly:
-  
-  ```bash
-  ln -s /data/srv/<DMWM tag>/sw/el8_amd64_gcc11/cms/dqmgui/<DQMGUI tag>/128/lib/libDQMGUI.so /usr/lib64/libDQMGUI.so
+  curl -L https://github.com/cms-DQM/dqmgui_prod_deployment/releases/download/latest/dqmgui_installation_package.tar.gz --output dqmgui_installation_package.tar.gz
+  scp dqmgui_installation_package.tar.gz root@<VM machine>:/tmp/
   ```
 
-  Replace:
-
-  - `<DMQM tag>` with the [`dmwm/deployment`](https://github.com/dmwm/deployment/tags) tag you want to use (contains the layouts).
-  - `<DQMGUI tag>` with the [DQMGUI](https://github.com/cms-DQM/dqmgui_prod/tags) tag you want to use (contains the underlying GUI code).
-
-  e.g.:
+1. Connect to the VM and install the system packages:
 
   ```bash
-  ln -s /data/srv/HG2903c/sw/el8_amd64_gcc11/cms/dqmgui/10.0.0/128/lib/libDQMGUI.so /usr/lib64/libDQMGUI.so
+  sudo yum install -y unzip bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh epel-release libcurl-devel python38 python38-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel openssl-devel libjpeg-turbo-devel libpng-devel gcc-c++ gcc binutils gcc-gfortran mesa-libGL-devel mesa-libGLU-devel glew-devel ftgl-devel fftw-devel cfitsio-devel graphviz-devel libuuid-devel avahi-compat-libdns_sd-devel openldap-devel python3-numpy libxml2-devel gsl-devel readline-devel R-devel R-Rcpp-devel R-RInside-devel xrootd-client
   ```
---> 
 
 2. Add a non-privileged user, create and give access to necessary directories and switch to it:
-  
+
   ```bash
   adduser dqm
+
+  # Installation directory
   sudo mkdir -p /data/srv
   sudo chown -R dqm /data/srv
+
+  # Data directory
   sudo mkdir -p /dqmdata/dqm
   sudo chown -R dqm /dqmdata
+
+  # Installation package
+  sudo chown dqm /tmp/dqmgui_installation_package.tar.gz
+
   sudo su dqm
   ```
 
 3. Start the deployment (`dev` flavor):
-  
+
   ```bash
   cd ~
-  # Clone this script and all necessary files, like patches
-  git clone https://github.com/cms-DQM/dqmgui_prod_deployment
-  
+  tar -xf dqmgui_installation_package.tar.gz -C dqmgui_deployment
+
   # Start the deployment script, it will take some time to finish
-  bash /home/dqm/dqmgui_prod_deployment/deploy_dqmgui.sh
+  bash /home/dqm/dqmgui_deployment/deploy_dqmgui.sh
 
   # Start all the services
   /data/srv/current/config/dqmgui/manage -f dev start "I did read documentation"
@@ -126,7 +144,7 @@ This procedure has been tested on a RHEL8 Openstack VM.
   firewall-cmd --zone=public --add-port=8081/tcp # relval
   ```
 
-## Command-line arguments
+## Deployment command-line arguments
 
 This script uses some internal variables to specify options such as the main installation directory, or the git tags/refs to use when cloning repositories. It's not recommended that you edit those, as most of them are hand-picked so that the project compiles (we're mostly referring to the git refs).
 
@@ -137,12 +155,10 @@ Below is a list of the values that you will most probably need to change to cust
 | Variable name | Description |Default value |
 |---------------|-------------|--------------|
 | `INSTALLATION_DIR` | The directory to install the GUI into. It should be writable by the user running the script | `/data/srv` |
-| `DMWM_GIT_TAG` | The git tag/ref to checkout `dmwm/deployment` to. See [here](https://github.com/dmwm/deployment/tags) for a list. | `HG2903c` |
-| `DQMGUI_GIT_TAG` | The git tag/ref to checkout `cms-DQM/dqmgui_prod` to. See [here](https://github.com/cms-DQM/dqmgui_prod) for a list. | `9.8.0` |
 
-## [Debug] Selectively run parts of the installation script
+### [Debug] Selectively run parts of the installation script
 
-The script is split into steps (see: `installation_steps`), which can all be toggled off or on by arguments when running the script.
+The script is split into steps (see the `installation_steps` array declared in the `deploy_dqmgui.sh` script), which can all be toggled off or on by arguments when running the script.
 
 The flags are named by concatenating `do_` with the name of the step, so, for example, `do_check_dependencies` or `do_install_rotoglup`.
 
@@ -152,8 +168,16 @@ The are all set to `1` by default, and you can override them when running the sc
 bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0
 ```
 
-A useful combination that can be used when you've already downloaded all external dependencies once, and you want just to re-compile the DQMGUI part for testing:
+A useful combination that can be used when you've already installed and built all steps once, but you only want to re-compile the DQMGUI part for testing:
 
 ```bash
-bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0
+bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_root=0 do_compile_root=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0
 ```
+
+## Download script
+
+The installation package is created by running `download_dependencies.sh`. This is done automatically by github actions (see `.github/workflows/build_installation_package` in this repository). The versions of the packages downloaded are specified in `config.sh`.
+
+## Notes
+
+- We're not using the RHEL8 `root` package, due to the fact that they are built for python3.6, hence we need to build it with python3.8.

config.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Configuration file, which is sourced by both download_dependencies.sh and deploy_dqmgui.sh
+# Here we specify both the URLs and tags/branches/refs of the repositories that DQMGUI
+# depends on.
+
+# Tag to use for getting the layouts and manage/deploy scripts
+# See: https://github.com/dmwm/deployment/tags
+#DMWM_GIT_URL=https://github.com/dmwm/deployment
+DMWM_GIT_TAG=debug
+DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment
+
+# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags
+#DQMGUI_GIT_TAG=9.8.0
+DQMGUI_GIT_TAG=python3_backup
+DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod
+
+# Boost.GIL. At most version 1.67!! The API changed radically after that.
+BOOST_GIL_GIT_URL=https://github.com/boostorg/gil
+BOOST_GIL_GIT_TAG=boost-1.66.0
+
+# OLD rotoglup code. Commit was found with lots of pain, so that the patch
+# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch
+ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494
+ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad
+
+# Yahoo!(TM) UI
+YUI_GIT_URL=https://github.com/yui/yui2
+YUI_GIT_TAG=master
+
+# Extjs
+EXTJS_GIT_URL=https://github.com/probonogeek/extjs
+EXTJS_GIT_TAG=3.1.1
+
+# D3
+D3_GIT_URL=https://github.com/d3/d3
+D3_GIT_TAG=v2.7.4
+
+# JSROOT
+JSROOT_GIT_URL=https://github.com/root-project/jsroot
+JSROOT_GIT_TAG=5.1.0
+
+# ROOT
+ROOT_GIT_URL=https://github.com/root-project/root/
+ROOT_GIT_TAG=v6-28-08