Skip to content

DIRAC 9.0

Jump to bottom Edit New page
Federico Stagni edited this page Oct 8, 2024 · 24 revisions

General

The concept of "Setup" is in the process of disappearing. The concept (which is explained in https://dirac.readthedocs.io/en/latest/AdministratorGuide/Introduction/diraccomponents.html) has a long history and effectively enabled the possibility to use one single machine as a server for multiple setups/installations (e.g. production and testing setups on the same node). This possibility is recognized now as being not useful. Therefore, this functionality is being removed.

Core/Framework

use InstalledComponentsDB
ALTER TABLE `HostLogging` DROP COLUMN `Setup`;
DROP DATABASE NotificationDB;

Accounting

PR https://github.com/DIRACGrid/DIRAC/pull/6565/ makes several simplifications to the Accounting system, and removes the concept of "Setup". This concept has always been reflected in the MySQL database table names, which need to be updated. In order to do so:

  1. Stop inserting of records, e.g. by stopping the Accounting/DataStore service(s)
  2. You will have several tables for which the name has to be altered. The following selection will print out the SQL command that you will need to issue to have things done (beware to replace "DIRAC-Certification" with the name of your setup).
SET group_concat_max_len=5000;SELECT group_concat(v.name separator '; ')
 FROM (
     SELECT concat('RENAME TABLE `', t.table_name, '` TO `', replace(t.table_name, '_DIRAC-Certification_', '_'), '`') name
     FROM information_schema.tables t
     WHERE table_name like '%_DIRAC-Certification_%'
 ) v;

(you might need to run the above more than once).

And then:

DELETE FROM `ac_catalog_Types` where name LIKE 'DIRAC-Certification%'

(again, replace 'DIRAC-Certification%' with the name of your setup).

  1. Restart the DataStore service(s)

Remove of DIRACSetup and OwnerDN in WMS and Transformations, plus several other simplifications and changes

Following changes in PRs including:

  1. Stop pilots submission, e.g. by stopping the SiteDirector agent(s)
  2. Stop job and PilotManager registration and jobs matching, e.g. by stopping the JobManager, PilotManager, Matcher service(s)
  3. Stop transformations submission, e.g. by stopping the TransformationManager service(s)
  4. Alter the following tables:
use JobDB;
ALTER TABLE `Jobs` ADD COLUMN `VO` VARCHAR(64);
use PilotAgentsDB;
ALTER TABLE `PilotAgents` ADD COLUMN `VO` VARCHAR(64);
use TaskQueueDB;
ALTER TABLE `tq_TaskQueues` ADD COLUMN `Owner` VARCHAR(255) NOT NULL;
ALTER TABLE `tq_TaskQueues` ADD COLUMN VO VARCHAR(64);
use SandboxMetadataDB;
ALTER TABLE `sb_Owners` ADD COLUMN `VO` VARCHAR(64);
use TransformationDB;
ALTER TABLE `Transformations` ADD COLUMN `Author` VARCHAR(255) NOT NULL;
ALTER TABLE `Transformations` MODIFY COLUMN `AuthorDN` VARCHAR(255) DEFAULT NULL;
  1. Save the following script in a (whatever, e.g. in /opt/dirac) directory of a DIRAC server machine:

Then run it with python script_name.py -o /DIRAC/Security/UseServerCertificate=yes

  1. Restart the JobManager, PilotManager and Matcher service(s)

  2. Restart the TransformationManager service(s)

  3. Restart the SiteDirector(s)

  4. Further DB changes can take place now:

use JobDB;
ALTER TABLE `Jobs` DROP COLUMN `DIRACSetup`;
ALTER TABLE `Jobs` DROP COLUMN `OwnerDN`;
ALTER TABLE `Jobs` DROP COLUMN `JobSplitType`;
ALTER TABLE `Jobs` DROP COLUMN `MasterJobID`;
ALTER TABLE `Jobs` DROP COLUMN `ApplicationNumStatus`;
ALTER TABLE `Jobs` DROP COLUMN `SystemPriority`;
ALTER TABLE `Jobs` DROP COLUMN `DeletedFlag`;
ALTER TABLE `Jobs` DROP COLUMN `KilledFlag`;
ALTER TABLE `Jobs` DROP COLUMN `FailedFlag`;
ALTER TABLE `Jobs` DROP COLUMN `ISandboxReadyFlag`;
ALTER TABLE `Jobs` DROP COLUMN `OSandboxReadyFlag`;
ALTER TABLE `Jobs` DROP COLUMN `RetrievedFlag`;
DROP TABLE `SiteMask`
DROP TABLE `SiteMaskLogging`
use PilotAgentsDB;
ALTER TABLE `PilotAgents` DROP COLUMN `GridRequirements`;
ALTER TABLE `PilotAgents` DROP COLUMN `OwnerDN`;
ALTER TABLE `PilotAgents` DROP COLUMN `ParentID`;
ALTER TABLE `PilotAgents` DROP COLUMN `TaskQueueID`;
ALTER TABLE `PilotAgents` DROP COLUMN `OwnerGroup`;
ALTER TABLE `PilotAgents` DROP COLUMN `Broker`;
ALTER TABLE `PilotAgents` DROP COLUMN `OutputReady`;
use TaskQueueDB;
ALTER TABLE `tq_TaskQueues` DROP COLUMN `Setup`;
ALTER TABLE `tq_TaskQueues` DROP COLUMN `OwnerDN`;
DROP TABLE `tq_TQToSubmitPools`;
use SandboxMetadataDB;
ALTER TABLE `sb_EntityMapping` DROP COLUMN `EntitySetup`;
ALTER TABLE `sb_Owners` DROP COLUMN `OwnerDN`;
use ProxyDB;
DROP TABLE ProxyDB_Proxies;
DROP TABLE ProxyDB_Tokens;

Workload Management System

SandboxStore

PR https://github.com/DIRACGrid/DIRAC/pull/7439 removes the possibility to use external StorageElements as SandboxStore. Most probably you have nothing to do.

Pilot

You can add the command RegisterPilot in the list of commands run by the Pilot (this is already the default, and it only applies if you were previously running with a non-default list of commands)

Transformations

With the changes to the permission enforcement for the Transformation one has to make sure that the Hosts running TransformationSystem Agents have the ProductionManagement property. The same is true for a shifterProxy and groups that are used for Transformations

Added index

Following https://github.com/DIRACGrid/DIRAC/issues/7335

ALTER TABLE `Transformations` ADD INDEX `status_index` (`Status`), ADD INDEX `type_index` (`Type`), ALGORITHM=INPLACE;

Remove VMDIRAC

VMDIRAC has been removed from the release.

  1. Stop agent(s) WorkloadManagement/CloudDirector and service(s) WorkloadManagement/VirtualMachineManager, either directly from the machine where they are running:
runsvctrl d /opt/dirac/startup/WorkloadManagement_CloudDirector
runsvctrl d /opt/dirac/startup/WorkloadManagement_WorkloadManagement

or better using the dirac-admin-sysadmin-cli CLI.

  1. Remove VirtualMachineDB
DROP DATABASE VirtualMachineDB;
  1. uninstall VirtualMachineManager and CloudDirector. Use the dirac-admin-sysadmin-cli CLI.
  2. remove any credentials you might have used from the server

Enable remote pilot logging

PR https://github.com/DIRACGrid/DIRAC/pull/6208 introduces a possibility to store pilot log files to remote storage. It is foreseen to use a plugin for this purpose. The PR contains a FileCacheLoggingPlugin which sends the logs to a SE.

  • Install WorkloadManagement/TornadoPilotLoggingHandler service
  • Install WorkloadManagement/PilotLoggingAgent agent

Configuration: Configuration is done in a VO by VO basis, in a VO-specific Pilot section in Operations. Defaults section can be used as usual to set up initial settings for all VOs.

  • Enable remote login in the Pilot section of a VO: RemoteLogging = True
  • Set the service URL: RemoteLoggerURL = https://dirac.host.name:8444/WorkloadManagement/TornadoPilotLogging
  • Set the upload SE, e.g.: UploadSE = UKI-LT2-IC-HEP-disk
  • Uploading is done by a Shifter called DataManager, so a shifter of this name should be defined in a shifter section of the VO
  • Set the upload path a VO can write to, e.g.: UploadPath = /gridpp/pilotlogs

The TornadoPilotLoggingHandler service requires a plugin name to be specified under Services/TornadoPilotLogging:

  • LoggingPlugin = FileCacheLoggingPlugin

Remove OptimizerModule

PR https://github.com/DIRACGrid/DIRAC/pull/7710 removes the old OptimizerModule. Chances are you have nothing to do, but in case your extension use it, move its corresponding optimizers to the Executor framework.

ProductionsManagement

DB changes

Following changes in PR https://github.com/DIRACGrid/DIRAC/pull/7162

  1. Stop submission of productions, e.g. by stopping the ProductionManager service(s)
  2. Alter the following table:
use ProductionDB;
ALTER TABLE `Productions` ADD COLUMN `Author` VARCHAR(255) NOT NULL;
ALTER TABLE `Productions` MODIFY COLUMN `AuthorDN` VARCHAR(255) DEFAULT NULL;
  1. Restart the ProductionManager service(s)

RMS

DB changes

Following changes in PR https://github.com/DIRACGrid/DIRAC/pull/6419 and https://github.com/DIRACGrid/DIRAC/pull/6653

  1. Stop submission of requests, e.g. by stopping the ReqManager service(s)
  2. Alter the following table:
use ReqDB;
ALTER TABLE `Request` DROP COLUMN `DIRACSetup`;
ALTER TABLE `Request` ADD COLUMN `Owner` VARCHAR(255);
  1. Restart the ReqManager service(s)

RSS

Following changes in PR https://github.com/DIRACGrid/DIRAC/pull/7762, RSS is now mandatory. If RSS is not yet active for you, BEFORE installing Dirac v9, you need to activate it by following instructions at https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/ResourceStatus/install.html

Resources

ARC and ARC6 have been removed and should be replaced with AREX.

  1. From a DIRAC client, execute the following command to get the port used by the CE to provide the AREX interface:
$ arcinfo -c <ce name>
Computing service: arc (production)
  Information endpoint: ldap://<ce name>:2135/Mds-Vo-Name=local,o=grid (org.nordugrid.ldapng)
  Information endpoint: ldap://<ce name>:2135/o=glue (org.nordugrid.ldapglue2)
  Information endpoint: https://<ce name>:443/arex (org.nordugrid.arcrest) <------------ get the port used here
  Information endpoint: https://<ce name>:443/arex (org.ogf.glue.emies.resourceinfo)
  Submission endpoint: https://<ce name>:443/arex (status: ok, interface: org.nordugrid.arcrest)
  Submission endpoint: https://<ce name>:443/arex (status: ok, interface: org.ogf.glue.emies.activitycreation)
  Submission endpoint: gsiftp://<ce name>:2811/jobs (status: ok, interface: org.nordugrid.gridftpjob)
  1. Simply replace CEType=ARC or ARC6 with CEType=AREX in the configuration
  2. If the port is not 443 (the default one), then you also need to add the following option to your configuration:
Port = <AREX port>
Add a custom footer
Add a custom sidebar
Clone this wiki locally