Skip to content

DIRAC 9.0

Jump to bottom Edit New page
Federico Stagni edited this page Jan 25, 2024 · 24 revisions

General

The concept of "Setup" is in the process of disappearing. The concept (which is explained in https://dirac.readthedocs.io/en/latest/AdministratorGuide/Introduction/diraccomponents.html) has a long history and effectively enabled the possibility to use one single machine as a server for multiple setups/installations (e.g. production and testing setups on the same node). This possibility is recognized now as being not useful. Therefore, this functionality is being removed.

Core/Framework

use InstalledComponentsDB
ALTER TABLE `HostLogging` DROP COLUMN `Setup`;

Accounting

PR https://github.com/DIRACGrid/DIRAC/pull/6565/ makes several simplifications to the Accounting system, and removes the concept of "Setup". This concept has always been reflected in the MySQL database table names, which need to be updated. In order to do so:

  1. Stop inserting of records, e.g. by stopping the Accounting/DataStore service(s)
  2. You will have several tables for which the name has to be altered. The following selection will print out the SQL command that you will need to issue to have things done (beware to replace "DIRAC-Certification" with the name of your setup).
SET group_concat_max_len=5000;SELECT group_concat(v.name separator '; ')
 FROM (
     SELECT concat('RENAME TABLE `', t.table_name, '` TO `', replace(t.table_name, '_DIRAC-Certification_', '_'), '`') name
     FROM information_schema.tables t
     WHERE table_name like '%_DIRAC-Certification_%'
 ) v;

(you might need to run the above more than once).

And then:

DELETE FROM `ac_catalog_Types` where name LIKE 'DIRAC-Certification%'

(again, replace 'DIRAC-Certification%' with the name of your setup).

  1. Restart the DataStore service(s)

Remove of DIRACSetup and OwnerDN in WMS and Transformations, plus several other simplifications and changes

Following changes in PRs including:

  1. Stop pilots submission, e.g. by stopping the SiteDirector agent(s)
  2. Stop job and PilotManager registration and jobs matching, e.g. by stopping the JobManager, PilotManager, Matcher service(s)
  3. Stop transformations submission, e.g. by stopping the TransformationManager service(s)
  4. Alter the following tables:
use JobDB;
ALTER TABLE `Jobs` ADD COLUMN `VO` VARCHAR(64);
use PilotAgentsDB;
ALTER TABLE `PilotAgents` ADD COLUMN `VO` VARCHAR(64);
use TaskQueueDB;
ALTER TABLE `tq_TaskQueues` ADD COLUMN `Owner` VARCHAR(255) NOT NULL;
ALTER TABLE `tq_TaskQueues` ADD COLUMN VO VARCHAR(64);
use SandboxMetadataDB;
ALTER TABLE `sb_Owners` ADD COLUMN `VO` VARCHAR(64);
use TransformationDB;
ALTER TABLE `Transformations` ADD COLUMN `Author` VARCHAR(255) NOT NULL;
ALTER TABLE `Transformations` MODIFY COLUMN `AuthorDN` VARCHAR(255) DEFAULT NULL;
  1. Save the following script in a (whatever, e.g. in /opt/dirac) directory of a DIRAC server machine:

Then run it with python script_name.py -o /DIRAC/Security/UseServerCertificate=yes

  1. Restart the JobManager, PilotManager and Matcher service(s)

  2. Restart the TransformationManager service(s)

  3. Restart the SiteDirector(s)

  4. Further DB changes can take place now:

use JobDB;
ALTER TABLE `Jobs` DROP COLUMN `DIRACSetup`;
ALTER TABLE `Jobs` DROP COLUMN `OwnerDN`;
ALTER TABLE `Jobs` DROP COLUMN `JobSplitType`;
ALTER TABLE `Jobs` DROP COLUMN `MasterJobID`;
ALTER TABLE `Jobs` DROP COLUMN `ApplicationNumStatus`;
ALTER TABLE `Jobs` DROP COLUMN `SystemPriority`;
ALTER TABLE `Jobs` DROP COLUMN `DeletedFlag`;
ALTER TABLE `Jobs` DROP COLUMN `KilledFlag`;
ALTER TABLE `Jobs` DROP COLUMN `FailedFlag`;
ALTER TABLE `Jobs` DROP COLUMN `ISandboxReadyFlag`;
ALTER TABLE `Jobs` DROP COLUMN `OSandboxReadyFlag`;
ALTER TABLE `Jobs` DROP COLUMN `RetrievedFlag`;
use PilotAgentsDB;
ALTER TABLE `PilotAgents` DROP COLUMN `GridRequirements`;
ALTER TABLE `PilotAgents` DROP COLUMN `OwnerDN`;
ALTER TABLE `PilotAgents` DROP COLUMN `ParentID`;
ALTER TABLE `PilotAgents` DROP COLUMN `TaskQueueID`;
ALTER TABLE `PilotAgents` DROP COLUMN `OwnerGroup`;
use TaskQueueDB;
ALTER TABLE `tq_TaskQueues` DROP COLUMN `Setup`;
ALTER TABLE `tq_TaskQueues` DROP COLUMN `OwnerDN`;
DROP TABLE `tq_TQToSubmitPools`;
use SandboxMetadataDB;
ALTER TABLE `sb_EntityMapping` DROP COLUMN `EntitySetup`;
ALTER TABLE `sb_Owners` DROP COLUMN `OwnerDN`;

Transformations

With the changes to the permission enforcement for the Transformation one has to make sure that the Hosts running TransformationSystem Agents have the ProductionManagement property. The same is true for a shifterProxy and groups that are used for Transformations

Added index

Following https://github.com/DIRACGrid/DIRAC/issues/7335

ALTER TABLE `Transformations` ADD INDEX `status_index` (`Status`), ADD INDEX `type_index` (`Type`), ALGORITHM=INPLACE;

Remove VMDIRAC

VMDIRAC has been removed from the release.

  1. Stop agent(s) WorkloadManagement/CloudDirector and service(s) WorkloadManagement/VirtualMachineManager, either directly from the machine where they are running:
runsvctrl d /opt/dirac/startup/WorkloadManagement_CloudDirector
runsvctrl d /opt/dirac/startup/WorkloadManagement_WorkloadManagement

or better using the dirac-admin-sysadmin-cli CLI.

  1. Remove VirtualMachineDB
DROP DATABASE VirtualMachineDB;
  1. uninstall VirtualMachineManager and CloudDirector. Use the dirac-admin-sysadmin-cli CLI.
  2. remove any credentials you might have used from the server

Enable remote pilot logging

PR https://github.com/DIRACGrid/DIRAC/pull/6208 introduces a possibility to store pilot log files to remote storage. It is foreseen to use a plugin for this purpose. The PR contains a FileCacheLoggingPlugin which sends the logs to a SE.

  • Install WorkloadManagement/TornadoPilotLoggingHandler service
  • Install WorkloadManagement/PilotLoggingAgent agent

Configuration: Configuration is done in a VO by VO basis, in a VO-specific Pilot section in Operations. Defaults section can be used as usual to set up initial settings for all VOs.

  • Enable remote login in the Pilot section of a VO: RemoteLogging = True
  • Set the service URL: RemoteLoggerURL = https://dirac.host.name:8444/WorkloadManagement/TornadoPilotLogging
  • Set the upload SE, e.g.: UploadSE = UKI-LT2-IC-HEP-disk
  • Uploading is done by a Shifter called DataManager, so a shifter of this name should be defined in a shifter section of the VO
  • Set the upload path a VO can write to, e.g.: UploadPath = /gridpp/pilotlogs

The TornadoPilotLoggingHandler service requires a plugin name to be specified under Services/TornadoPilotLogging:

  • LoggingPlugin = FileCacheLoggingPlugin

ProductionsManagement

DB changes

Following changes in PR https://github.com/DIRACGrid/DIRAC/pull/7162

  1. Stop submission of productions, e.g. by stopping the ProductionManager service(s)
  2. Alter the following table:
use ProductionDB;
ALTER TABLE `Productions` ADD COLUMN `Author` VARCHAR(255) NOT NULL;
ALTER TABLE `Productions` MODIFY COLUMN `AuthorDN` VARCHAR(255) DEFAULT NULL;
  1. Restart the ProductionManager service(s)

RMS

DB changes

Following changes in PR https://github.com/DIRACGrid/DIRAC/pull/6419 and https://github.com/DIRACGrid/DIRAC/pull/6653

  1. Stop submission of requests, e.g. by stopping the ReqManager service(s)
  2. Alter the following table:
use ReqDB;
ALTER TABLE `Request` DROP COLUMN `DIRACSetup`;
ALTER TABLE `Request` ADD COLUMN `Owner` VARCHAR(255);
  1. Restart the ReqManager service(s)

Configuration

Now there exist different pilot "submission policies".

  • WaitingSupportedJobs (the default one): will only fill up some of the available slots, according to the number of waiting jobs. It does not take into account the number of already submitted pilots for these jobs (might result in some unused pilots but should not be a big issue, as it should target sites that are just occasionally used).
  • AggressiveFilling: will just fill up the available slots, no matter whether there are waiting jobs

To configure the submission policy, add the following option to the SiteDirector section of the CS:

SubmissionPolicy = <SubmissionPolicy name>
Add a custom footer
Add a custom sidebar
Clone this wiki locally