Making it work offline

cms-DQM · Nov 8, 2023 · f71fe3a · f71fe3a
1 parent ad07ab0
commit f71fe3a
Show file tree

Hide file tree

Showing 6 changed files with 374 additions and 191 deletions.
diff --git a/.github/workflows/build_installation_package.yaml b/.github/workflows/build_installation_package.yaml
@@ -0,0 +1,25 @@
+name: Build DQMGUI Installation Package
+
+on: [push]
+
+jobs:
+  build_and_upload:
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - name: Download and compress dependencies
+        run: |
+          sudo apt update && sudo apt install -y libcurl4-gnutls-dev
+          $(which python3) --version
+          bash download_dependencies.sh
+          rm -rf .git
+          mkdir -p /tmp/dqmgui
+          tar -cf /tmp/dqmgui/dqmgui_installation_package.tar.gz . -I "gzip --best"
+      - name: Archive package artifact
+        uses: actions/upload-artifact@v3
+        # test
+        with:
+          name: dqmgui-installation-package
+          path: |
+            /tmp/dqmgui/dqmgui_installation_package.tar.gz
diff --git a/README.md b/README.md
@@ -1,8 +1,13 @@
 # [Legacy] DQMGUI Deployment
 
-This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.6 for the needs of moving DQM production to new machines.
+[![Build DQMGUI Installation Package](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml/badge.svg)](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml)
 
-It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while this script downloads and builds all requirements from source (hence takes longer to deploy).
+This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.8 for the needs of moving DQM production to new machines.
+
+It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while the method we implement in this repository depends on two steps:
+
+1. Download all the external resources needed (e.g. python packages from PyPI, github repositories) and compress them.
+2. Copy the archives to the P5 machine and extract, then build from source (hence takes longer to deploy).
 
 > **Warning**
 > This deployment script should *not* be run as a sudo user.
@@ -12,15 +17,13 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
 - RHEL8 (Tested with version 8.8)
 - `sudo` permissions:
   - To install the system-wide packages (listed below).
-<!--  - To create a soft link in `/usr/lib64`. -->
-- Python 3.6
+- Python 3.8
 - Git
-- Access to PyPI for downloading python packages.
-- Access to GitHub for cloning several external dependencies.
 - Several system packages, installed via `yum`:
   <details>
   <summary>Package list</summary>  
 
+  - unzip
   - bzip2
   - libglvnd-opengl
   - libX11-devel
@@ -42,12 +45,10 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
   - perl-Digest-MD5
   - tcsh
   - zsh
-  - root
-  - python3-root
   - epel-release
-  - python3-pip
   - libcurl-devel
-  - python36-devel
+  - python38
+  - python38-devel
   - boost-python3-devel
   - protobuf-devel
   - jemalloc-devel
@@ -56,10 +57,31 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
   - lzo-devel
   - cmake
   - xz-devel
-  - python3-sphinx
   - openssl-devel
   - libjpeg-turbo-devel
   - libpng-devel
+  - gcc-c++
+  - gcc
+  - binutils
+  - gcc-gfortran
+  - mesa-libGL-devel
+  - mesa-libGLU-devel
+  - glew-devel
+  - ftgl-devel
+  - fftw-devel
+  - cfitsio-devel
+  - graphviz-devel
+  - libuuid-devel
+  - avahi-compat-libdns_sd-devel
+  - openldap-devel
+  - python3-numpy
+  - libxml2-devel
+  - gsl-devel
+  - readline-devel
+  - R-devel
+  - R-Rcpp-devel
+  - R-RInside-devel
+  - xrootd-client
   </details>
 
 ## Deploying [Legacy] DQMGUI
@@ -69,26 +91,8 @@ This procedure has been tested on a RHEL8 Openstack VM.
 1. Install the system packages:
 
   ```bash
-  sudo yum install -y tmux git bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel libpng-devel libjpeg-turbo-devel 
-  ```
-<!-- 
-2. Create a link to `libDQMGUI.so` which we will be compiling shortly:
-  
-  ```bash
-  ln -s /data/srv/<DMWM tag>/sw/el8_amd64_gcc11/cms/dqmgui/<DQMGUI tag>/128/lib/libDQMGUI.so /usr/lib64/libDQMGUI.so
-  ```
-
-  Replace:
-
-  - `<DMQM tag>` with the [`dmwm/deployment`](https://github.com/dmwm/deployment/tags) tag you want to use (contains the layouts).
-  - `<DQMGUI tag>` with the [DQMGUI](https://github.com/cms-DQM/dqmgui_prod/tags) tag you want to use (contains the underlying GUI code).
-
-  e.g.:
-
-  ```bash
-  ln -s /data/srv/HG2903c/sw/el8_amd64_gcc11/cms/dqmgui/10.0.0/128/lib/libDQMGUI.so /usr/lib64/libDQMGUI.so
+  sudo yum install -y unzip bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh epel-release libcurl-devel python38 python38-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel openssl-devel libjpeg-turbo-devel libpng-devel gcc-c++ gcc binutils gcc-gfortran mesa-libGL-devel mesa-libGLU-devel glew-devel ftgl-devel fftw-devel cfitsio-devel graphviz-devel libuuid-devel avahi-compat-libdns_sd-devel openldap-devel python3-numpy libxml2-devel gsl-devel readline-devel R-devel R-Rcpp-devel R-RInside-devel xrootd-client
   ```
---> 
 
 2. Add a non-privileged user, create and give access to necessary directories and switch to it:
 
@@ -137,12 +141,10 @@ Below is a list of the values that you will most probably need to change to cust
 | Variable name | Description |Default value |
 |---------------|-------------|--------------|
 | `INSTALLATION_DIR` | The directory to install the GUI into. It should be writable by the user running the script | `/data/srv` |
-| `DMWM_GIT_TAG` | The git tag/ref to checkout `dmwm/deployment` to. See [here](https://github.com/dmwm/deployment/tags) for a list. | `HG2903c` |
-| `DQMGUI_GIT_TAG` | The git tag/ref to checkout `cms-DQM/dqmgui_prod` to. See [here](https://github.com/cms-DQM/dqmgui_prod) for a list. | `9.8.0` |
 
 ## [Debug] Selectively run parts of the installation script
 
-The script is split into steps (see: `installation_steps`), which can all be toggled off or on by arguments when running the script.
+The script is split into steps (see the `installation_steps` array declared in the `deploy_dqmgui.sh` script), which can all be toggled off or on by arguments when running the script.
 
 The flags are named by concatenating `do_` with the name of the step, so, for example, `do_check_dependencies` or `do_install_rotoglup`.
 
@@ -152,8 +154,12 @@ The are all set to `1` by default, and you can override them when running the sc
 bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0
 ```
 
-A useful combination that can be used when you've already downloaded all external dependencies once, and you want just to re-compile the DQMGUI part for testing:
+A useful combination that can be used when you've already installed and built all steps once, but you only want to re-compile the DQMGUI part for testing:
 
 ```bash
-bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0
+bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_root=0 do_compile_root=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0
 ```
+
+## Notes
+
+- We're not using the RHEL8 `root` package, due to the fact that they are built for python3.6, hence we need to build it with python3.8.
diff --git a/config.sh b/config.sh
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# Tag to use for getting the layouts and manage/deploy scripts
+# See: https://github.com/dmwm/deployment/tags
+#DMWM_GIT_URL=https://github.com/dmwm/deployment
+DMWM_GIT_TAG=debug
+DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment
+
+# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags
+#DQMGUI_GIT_TAG=9.8.0
+DQMGUI_GIT_TAG=python3_backup
+DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod
+
+# Boost.GIL. At most version 1.67!! The API changed radically after that.
+BOOST_GIL_GIT_URL=https://github.com/boostorg/gil
+BOOST_GIL_GIT_TAG=boost-1.66.0
+
+# OLD rotoglup code. Commit was found with lots of pain, so that the patch
+# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch
+ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494
+ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad
+
+# Yahoo!(TM) UI
+YUI_GIT_URL=https://github.com/yui/yui2
+YUI_GIT_TAG=master
+
+# Extjs
+EXTJS_GIT_URL=https://github.com/probonogeek/extjs
+EXTJS_GIT_TAG=3.1.1
+
+# D3
+D3_GIT_URL=https://github.com/d3/d3
+D3_GIT_TAG=v2.7.4
+
+# JSROOT
+JSROOT_GIT_URL=https://github.com/root-project/jsroot
+JSROOT_GIT_TAG=5.1.0
+
+# ROOT
+ROOT_GIT_URL=https://github.com/root-project/root/
+ROOT_GIT_TAG=v6-28-08