DIRAC 7.3 (v7r3)

Make sure the "Rename of scripts" in the v7r2 wiki has been completed before continuing.

Python3

Python 3 based client installations

Python3 DIRAC clients are now fully supported, and they should be considered the default way of installing DIRAC clients (see instructions).

SiteDirectors are now by default installing python3 clients.

Python 3 based server installations

Python3 servers have been extensively tested. The suggestion is that DIRAC servers should be moved, one at a time, from python2 to python3, while running DIRAC 7.3 (v7r3).

This is the last version of DIRAC supporting python2.

bashrc

Server bashrc files should be updated to be compatible with both the Python 2 and Python 3 based installations. Installations at CERN (LHCb, CLIC, Certification) are using the following bashrc which is compatible with both:

Click to expand!

# DIRAC bashrc file, used by service and agent run scripts to set environment
# Find which DIRAC installation to use
default_dirac="/opt/dirac/pro"

# If "pro" points to a Python 3 version there will be a Linux-x86_64 folder
system_arch="$(uname -s)-$(uname -m)"
if [ -d "${default_dirac}/${system_arch}" ]; then
    default_dirac="${default_dirac}/${system_arch}"
fi

if [[ -n "${DIRAC:-}" ]] && [[ "${default_dirac}" != "${DIRAC:-}" ]]; then
    echo DIRAC installation path overridden to ${DIRAC}
fi

export DIRAC=${DIRAC:-${default_dirac}}
unset default_dirac system_arch

# CAs path for SSL verification
export X509_CERT_DIR=${X509_CERT_DIR:-"/etc/grid-security/certificates"}
export X509_VOMS_DIR=${X509_VOMS_DIR:-"/etc/grid-security/vomsdir"}
export X509_VOMSES=${X509_VOMSES:-"/etc/vomses"}

if [[ -d "${DIRAC}/diracos" ]]; then
    py3_based=0
    . $DIRAC/diracos/diracosrc
else
    py3_based=1
    . $DIRAC/diracosrc
fi

# Configuration which is only required for legacy installations
if [[ "${py3_based}" -eq 0 ]]; then
    export PYTHONUNBUFFERED=yes
    export PYTHONOPTIMIZE=x
    export DIRACSCRIPTS=$DIRAC/scripts
    export TERMINFO=$DIRACOS/usr/share/terminfo:/usr/share/terminfo:/etc/terminfo
    export RRD_DEFAULT_FONT=$DIRACOS/usr/share/fonts/DejaVuSansMono-Roman.ttf
    # Prepend the PYTHONPATH, the LD_LIBRARY_PATH, and the DYLD_LIBRARY_PATH
    ( echo $PATH | grep -q $DIRACSCRIPTS ) || export PATH=$DIRACSCRIPTS:$PATH
    ( echo $PYTHONPATH | grep -q $DIRAC ) || export PYTHONPATH=$DIRAC
    # new OpenSSL version require OPENSSL_CONF to point to some accessible location
    export OPENSSL_CONF=/tmp
    # probably not needed but we'll keep it for Python 2
    export SSL_CERT_DIR=${SSL_CERT_DIR:-"/etc/grid-security/certificates"}
fi

# The remainder of this file is custom environment configuration for this specific installation

# IPv6 support
export GLOBUS_IO_IPV6=TRUE
export GLOBUS_FTP_CLIENT_IPV6=TRUE

# Fork support for xrootd
export XRD_RUNFORKHANDLER=1

export DIRAC_GFAL_GRIDFTP_SESSION_REUSE=True

# CONDOR config
export _condor_DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME=0

# If there is an extra_bashrc file, source it
current_dir=$(dirname $(readlink -f "$BASH_SOURCE"));
extra_bashrc=$current_dir/extra_bashrc

if [ -f "$extra_bashrc" ]; then
  source $extra_bashrc
fi

⚠️ Some older installations have source /path/to/bashrc in the runsvdir-start file. This should be removed before updating to Python 3. You will have to restart runsvdir for the change to take effect.

WebApp

If using nginx: Add the /WebApp/StaticResourceLinkDir option to the dirac.cfg so Python 3 based webapps can be installed, the recommended value is /opt/dirac/webRoot/resources/.

Update run script paths to .cfg files

Older DIRAC versions used to install components with cfg files at paths like /opt/dirac/pro/etc/ResourceStatus_EmailAgent.cfg. These need to be replaced with the target of the symlink (i.e. /opt/dirac/pro/etc/ → /opt/dirac/etc/).

# Find files with matches
grep -r -E '/opt/dirac/pro/etc/[^ /]+\.cfg' /opt/dirac/runit/*/*/run
# Check the new files look reasonable
grep --files-with-matches -r -E '/opt/dirac/pro/etc/[^ /]+\.cfg' /opt/dirac/runit/*/*/run | xargs -I £ sed -E 's@/opt/dirac/pro/etc/([^ /]+\.cfg)@/opt/dirac/etc/\1@g' £
# Assuming the output from the above command looks reasonable, edit the files in place
grep --files-with-matches -r -E '/opt/dirac/pro/etc/[^ /]+\.cfg' /opt/dirac/runit/*/*/run | xargs -I £ sed -i -E 's@/opt/dirac/pro/etc/([^ /]+\.cfg)@/opt/dirac/etc/\1@g' £

Initial update

After the above changes are made the system administrator tools can be used to update a host to a Python 3 based installation. This can be done by adding DIRAC==v7.3 (edit to specify a different primary extension and/or a specific patch release as required). The DIRAC== prefix is only required for the first update. Future updates can be performed by simply entering a PEP-440 style version such as v7.3.999.

Python 3 based installations not able to install Python 2 based versions of DIRAC however you can roll back by either clicking Revert in the system administrator or by manually editing the pro symlink to point to a different version.

VMDIRAC merged into DIRAC

The VMDIRAC extension (https://github.com/DIRACGrid/VMDIRAC) has been merged into DIRAC (and its web page, into WebAppDIRAC): https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/VMDIRAC.html

If you have been using before VMDIRAC, you should at a minimum update the DIRAC configuration to take into account that the VMDIRAC extension is not anymore needed.

Framework

Removed SystemLogging

Following this PR, the following services and agents have been removed:

• service Framework/SystemLogging
• service Framework/SystemLoggingReport
• agent Framework/ErrorMessageMonitor
• agent Framework/SystemLoggingDBCleaner
• agent Framework/TopErrorMessageReporter

And the SystemLogging DB can be removed: DROP DATABASE SystemLoggingDB

Users are encouraged to use the ES stack for logs aggregation like described in this documentation

Database changes

Extend Extension column in InstalledComponentsDB to be String(512):

ALTER TABLE `HostLogging` MODIFY COLUMN `Extension` VARCHAR(512)

PathFinder

Note that the PathFinder methods API has been modified as follows. The main change in API is that the system name and component name are no longer transmitted by tuple, instead the second argument you can insert the name of the component:

PathFinder.getComponentSection('Framework/ProxyManager')

OR

PathFinder.getComponentSection('Framework', 'ProxyManager')

method name	before	now
divideFullName	entityName	entityName, componentName=None
getComponentSection	componentName, componentTuple=False, setup=False, componentCategory="Services"	system, component=False, setup=False, componentCategory="Services"
getSystemSection	serviceName, serviceTuple=False, instance=False, setup=False	system, instance=False, setup=False
getSystemInstance	systemName, setup=False	system, setup=False
getServiceSection	serviceName, serviceTuple=False, setup=False	system, serviceName=False, setup=False
getExecutorSection	executorName, executorTuple=False, setup=False	system, executorName=None, component=False, setup=False
getAgentSection	serviceName, serviceTuple=False, setup=False	system, agentName=None, component=False, setup=False
getDatabaseSection	dbName, dbTuple=False, setup=False	system, dbName=False, setup=False
getSystemURLSection	serviceName, serviceTuple=False, setup=False	system, setup=False
getServiceURL	serviceName, serviceTuple=False, setup=False	system, service=None, setup=False
getServiceFailoverURL	serviceName, serviceTuple=False, setup=False	system, service=None, setup=False
getGatewayURLs	serviceName=""	system="", service=None

WorkloadManagementSystem

StatesMonitoringAgent -> StatesAccountingAgent

The agent WorkloadManagement/StatesMonitoringAgent has been removed (follow-up of this change in v7r2)

New agent TaskQueuesAgent

The agent WorkloadManagement/TaskQueuesAgent has to be installed (following this PR)

3 new HTTPs based services

This PR introduced 3 new, optional HTTPs services: JobManager, JobMonitor, JobStateUpdate. Do consider use them.

Transformation System

Refactorize TaskManager

Following this PR, RequestTasks and WorkflowTasks are now into distinct module instead of being in TaskManager. If your extension relies on it, you need to fix the import path:

- from DIRAC.TransformationSystem.Client.TaskManager import RequestTasks, WorkflowTasks
+ from DIRAC.TransformationSystem.Client.WorkflowTasks import WorkflowTasks
+ from DIRAC.TransformationSystem.Client.RequestTasks import RequestTasks