Skip to content

Commit

Permalink
Making it work offline
Browse files Browse the repository at this point in the history
  • Loading branch information
nothingface0 committed Nov 10, 2023
1 parent ad07ab0 commit 20ed6f9
Show file tree
Hide file tree
Showing 6 changed files with 425 additions and 201 deletions.
42 changes: 42 additions & 0 deletions .github/workflows/build_installation_package.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Build DQMGUI Installation Package

on: [push]

jobs:
build_and_upload:
runs-on: ubuntu-20.04
permissions:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download and compress dependencies
run: |
sudo apt update && sudo apt install -y libcurl4-gnutls-dev
$(which python3) --version
bash download_dependencies.sh
rm -rf .git
mkdir -p /tmp/dqmgui
tar -cf /tmp/dqmgui/dqmgui_installation_package.tar.gz . -I "gzip --best"
- name: Archive package artifact
uses: actions/upload-artifact@v3
with:
name: dqmgui-installation-package
path: |
/tmp/dqmgui/dqmgui_installation_package.tar.gz
# Create a dummy release, so that the artifact is downloadable
# without a GitHub account
# This will automatically update the artifacts of the "latest" release,
# even though the "latest" tag does match with the actually latest commit.
- name: Release
uses: ncipollo/release-action@v1
with:
artifacts: "/tmp/dqmgui/dqmgui_installation_package.tar.gz"
body:
"Automatically generated dummy release for making the installation
package publicly available. The `latest` tag is
put on a random commit, and does not reflect the current state of the release,
while the artifact of this release is updated with the actually latest files,
on each push."
tag: "latest"
allowUpdates: "true"
109 changes: 66 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
# [Legacy] DQMGUI Deployment

This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.6 for the needs of moving DQM production to new machines.
[![Build DQMGUI Installation Package](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml/badge.svg)](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml)

It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while this script downloads and builds all requirements from source (hence takes longer to deploy).
This repository contains all the requirements for deploying the [Legacy] [DQMGUI](https://github.com/cms-DQM/dqmgui_prod) on a Red Hat Enterprise Linux 8 machine, with Python3.8 for the needs of moving DQM production to new machines.

It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](https://github.com/dmwm/deployment/tree/master/Deploy), which only targets OS up to SLC7. The main difference is that the `Deploy` script relies on pre-built libraries and executables, found on `cmsrep.cern.ch`, while the method we implement in this repository depends on two steps:

1. Download all the external resources needed (e.g. python packages from PyPI, github repositories) and compress them. This is done automatically with GitHub actions, and you can download a package ready to install [here](https://github.com/cms-DQM/dqmgui_prod_deployment/actions/workflows/build_installation_package.yaml).
2. Copy the archives to the P5 machine and extract, then build from source (hence takes longer to deploy).

> **Warning**
> This deployment script should *not* be run as a sudo user.
Expand All @@ -12,15 +17,12 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
- RHEL8 (Tested with version 8.8)
- `sudo` permissions:
- To install the system-wide packages (listed below).
<!-- - To create a soft link in `/usr/lib64`. -->
- Python 3.6
- Git
- Access to PyPI for downloading python packages.
- Access to GitHub for cloning several external dependencies.
- Python 3.8
- Several system packages, installed via `yum`:
<details>
<summary>Package list</summary>
<summary>Package list</summary>

- unzip
- bzip2
- libglvnd-opengl
- libX11-devel
Expand All @@ -42,12 +44,10 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
- perl-Digest-MD5
- tcsh
- zsh
- root
- python3-root
- epel-release
- python3-pip
- libcurl-devel
- python36-devel
- python38
- python38-devel
- boost-python3-devel
- protobuf-devel
- jemalloc-devel
Expand All @@ -56,60 +56,77 @@ It aims to replace the existing one (`Deploy`), found in [`dmwm/deployment`](htt
- lzo-devel
- cmake
- xz-devel
- python3-sphinx
- openssl-devel
- libjpeg-turbo-devel
- libpng-devel
- gcc-c++
- gcc
- binutils
- gcc-gfortran
- mesa-libGL-devel
- mesa-libGLU-devel
- glew-devel
- ftgl-devel
- fftw-devel
- cfitsio-devel
- graphviz-devel
- libuuid-devel
- avahi-compat-libdns_sd-devel
- openldap-devel
- python3-numpy
- libxml2-devel
- gsl-devel
- readline-devel
- R-devel
- R-Rcpp-devel
- R-RInside-devel
- xrootd-client
</details>

## Deploying [Legacy] DQMGUI

This procedure has been tested on a RHEL8 Openstack VM.
This procedure has been tested on a RHEL8 Openstack VM. Instructions below are primarily for a personal VM.

1. Install the system packages:
0. Download the latest build artifact and copy it to the machine you want to install it to:

```bash
sudo yum install -y tmux git bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh root python3-root epel-release python3-pip libcurl-devel python36-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel python3-sphinx openssl-devel libpng-devel libjpeg-turbo-devel
```
<!--
2. Create a link to `libDQMGUI.so` which we will be compiling shortly:
```bash
ln -s /data/srv/<DMWM tag>/sw/el8_amd64_gcc11/cms/dqmgui/<DQMGUI tag>/128/lib/libDQMGUI.so /usr/lib64/libDQMGUI.so
curl -L https://github.com/cms-DQM/dqmgui_prod_deployment/releases/download/latest/dqmgui_installation_package.tar.gz --output dqmgui_installation_package.tar.gz
scp dqmgui_installation_package.tar.gz root@<VM machine>:/tmp/
```

Replace:
- `<DMQM tag>` with the [`dmwm/deployment`](https://github.com/dmwm/deployment/tags) tag you want to use (contains the layouts).
- `<DQMGUI tag>` with the [DQMGUI](https://github.com/cms-DQM/dqmgui_prod/tags) tag you want to use (contains the underlying GUI code).
e.g.:
1. Connect to the VM and install the system packages:

```bash
ln -s /data/srv/HG2903c/sw/el8_amd64_gcc11/cms/dqmgui/10.0.0/128/lib/libDQMGUI.so /usr/lib64/libDQMGUI.so
sudo yum install -y unzip bzip2 libglvnd-opengl libX11-devel libXext-devel libXft-devel libXpm-devel mesa-libGLU mesa-libGLU-devel perl-Env perl-Switch perl-Thread-Queue glibc-headers libidn libXcursor libXi libXinerama libXrandr perl perl-Digest-MD5 tcsh zsh epel-release libcurl-devel python38 python38-devel boost-python3-devel protobuf-devel jemalloc-devel pcre-devel boost-devel lzo-devel cmake xz-devel openssl-devel libjpeg-turbo-devel libpng-devel gcc-c++ gcc binutils gcc-gfortran mesa-libGL-devel mesa-libGLU-devel glew-devel ftgl-devel fftw-devel cfitsio-devel graphviz-devel libuuid-devel avahi-compat-libdns_sd-devel openldap-devel python3-numpy libxml2-devel gsl-devel readline-devel R-devel R-Rcpp-devel R-RInside-devel xrootd-client
```
-->

2. Add a non-privileged user, create and give access to necessary directories and switch to it:

```bash
adduser dqm

# Installation directory
sudo mkdir -p /data/srv
sudo chown -R dqm /data/srv

# Data directory
sudo mkdir -p /dqmdata/dqm
sudo chown -R dqm /dqmdata

# Installation package
sudo chown dqm /tmp/dqmgui_installation_package.tar.gz

sudo su dqm
```

3. Start the deployment (`dev` flavor):

```bash
cd ~
# Clone this script and all necessary files, like patches
git clone https://github.com/cms-DQM/dqmgui_prod_deployment

tar -xf dqmgui_installation_package.tar.gz -C dqmgui_deployment

# Start the deployment script, it will take some time to finish
bash /home/dqm/dqmgui_prod_deployment/deploy_dqmgui.sh
bash /home/dqm/dqmgui_deployment/deploy_dqmgui.sh

# Start all the services
/data/srv/current/config/dqmgui/manage -f dev start "I did read documentation"
Expand All @@ -126,7 +143,7 @@ This procedure has been tested on a RHEL8 Openstack VM.
firewall-cmd --zone=public --add-port=8081/tcp # relval
```

## Command-line arguments
## Deployment command-line arguments

This script uses some internal variables to specify options such as the main installation directory, or the git tags/refs to use when cloning repositories. It's not recommended that you edit those, as most of them are hand-picked so that the project compiles (we're mostly referring to the git refs).

Expand All @@ -137,12 +154,10 @@ Below is a list of the values that you will most probably need to change to cust
| Variable name | Description |Default value |
|---------------|-------------|--------------|
| `INSTALLATION_DIR` | The directory to install the GUI into. It should be writable by the user running the script | `/data/srv` |
| `DMWM_GIT_TAG` | The git tag/ref to checkout `dmwm/deployment` to. See [here](https://github.com/dmwm/deployment/tags) for a list. | `HG2903c` |
| `DQMGUI_GIT_TAG` | The git tag/ref to checkout `cms-DQM/dqmgui_prod` to. See [here](https://github.com/cms-DQM/dqmgui_prod) for a list. | `9.8.0` |

## [Debug] Selectively run parts of the installation script
### [Debug] Selectively run parts of the installation script

The script is split into steps (see: `installation_steps`), which can all be toggled off or on by arguments when running the script.
The script is split into steps (see the `installation_steps` array declared in the `deploy_dqmgui.sh` script), which can all be toggled off or on by arguments when running the script.

The flags are named by concatenating `do_` with the name of the step, so, for example, `do_check_dependencies` or `do_install_rotoglup`.

Expand All @@ -152,8 +167,16 @@ The are all set to `1` by default, and you can override them when running the sc
bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0
```

A useful combination that can be used when you've already downloaded all external dependencies once, and you want just to re-compile the DQMGUI part for testing:
A useful combination that can be used when you've already installed and built all steps once, but you only want to re-compile the DQMGUI part for testing:

```bash
bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0
bash deploy_dqmgui.sh do_preliminary_checks=0 do_check_dependencies=0 do_create_directories=1 do_install_boost_gil=0 do_install_gil_numeric=0 do_install_rotoglup=0 do_install_classlib=0 do_compile_classlib=0 do_install_dmwm=0 do_install_root=0 do_compile_root=0 do_install_dqmgui=0 do_compile_dqmgui=1 do_install_yui=0 do_install_extjs=0 do_install_d3=0 do_install_jsroot=0
```

## Download script

The installation package is created by running `download_dependencies.sh`. This is done automatically by github actions (see `.github/workflows/build_installation_package` in this repository). The versions of the packages downloaded are specified in `config.sh`.

## Notes

- We're not using the RHEL8 `root` package, due to the fact that they are built for python3.6, hence we need to build it with python3.8.
45 changes: 45 additions & 0 deletions config.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/bin/bash

# Configuration file, which is sourced by both download_dependencies.sh and deploy_dqmgui.sh
# Here we specify both the URLs and tags/branches/refs of the repositories that DQMGUI
# depends on.

# Tag to use for getting the layouts and manage/deploy scripts
# See: https://github.com/dmwm/deployment/tags
#DMWM_GIT_URL=https://github.com/dmwm/deployment
DMWM_GIT_TAG=debug
DMWM_GIT_URL=https://github.com/nothingface0/cms_dmwm_deployment

# DQMGUI tag to use, see https://github.com/cms-DQM/dqmgui_prod/tags
#DQMGUI_GIT_TAG=9.8.0
DQMGUI_GIT_TAG=python3_backup
DQMGUI_GIT_URL=https://github.com/cms-DQM/dqmgui_prod

# Boost.GIL. At most version 1.67!! The API changed radically after that.
BOOST_GIL_GIT_URL=https://github.com/boostorg/gil
BOOST_GIL_GIT_TAG=boost-1.66.0

# OLD rotoglup code. Commit was found with lots of pain, so that the patch
# applies: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/dqmgui-rtgu.patch
ROTOGLUP_GIT_TAG=d8ce23aecd0b1fb7d45c9bedb615abdab27a5494
ROTOGLUP_GIT_URL=https://github.com/rotoglup/rotoglup-scratchpad

# Yahoo!(TM) UI
YUI_GIT_URL=https://github.com/yui/yui2
YUI_GIT_TAG=master

# Extjs
EXTJS_GIT_URL=https://github.com/probonogeek/extjs
EXTJS_GIT_TAG=3.1.1

# D3
D3_GIT_URL=https://github.com/d3/d3
D3_GIT_TAG=v2.7.4

# JSROOT
JSROOT_GIT_URL=https://github.com/root-project/jsroot
JSROOT_GIT_TAG=5.1.0

# ROOT
ROOT_GIT_URL=https://github.com/root-project/root/
ROOT_GIT_TAG=v6-28-08
Loading

0 comments on commit 20ed6f9

Please sign in to comment.