Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating workflow json templates #11949

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Updating workflow json templates #11949

wants to merge 4 commits into from

Conversation

amaltaro
Copy link
Contributor

@amaltaro amaltaro commented Mar 26, 2024 •
edited
Loading

Fixes #11860

Status

In development

Description

List of templates update is:

  • 1st commit: refreshed all of the ReReco templates (1 in DMWM, 4 in Integration - but only 1 actually executable
  • 2nd commit: [Integration] StepChain from scratch with PREMIX, 5 steps, multicore, with EL8 release.
  • 4th commit: refreshed all of the DQMHarvest templates (1 in DMWM and 2 in Integration)
    • input data locked (permanently) for T2_CH_CERN and T1_US_FNAL_Disk

Breakdown of the templates to be created are described in:
#11860

ToDo list:

  • ReReco workflows
  • DQMHarvest workflows
  • TaskChain workflows
  • StepChain workflows

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

None

External dependencies / deployment changes

None

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14995/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests no longer failing
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15092/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15093/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15094/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Can one of the admins verify this patch?

@amaltaro amaltaro force-pushed the fix-11860 branch 2 times, most recently from 3682b43 to f2ba45b Compare January 30, 2025 16:19
@dmwm-bot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 3 tests no longer failing
    • 3 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/317/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

I have migrated a script from my repository over to wmcore-tools:
https://gitlab.cern.ch/dmwm/wmcore-tools/-/blob/master/mstransferor/makeRucioRules.py?ref_type=heads

and used it to lock the DQMIO dataset both at FNAL and CERN, e.g.:

(WMAgent-2.3.7.6) [xxx@vocmsxxx:logs]$ python makeRucioRules.py -c /JetMET0/Run2024D-2024CDEReprocessing-v1/DQMIO -r T1_US_FNAL_Disk
2025-01-30 21:00:09,584:INFO:Rucio: WMCore Rucio initialization parameters: ...
2025-01-30 21:00:09,585:INFO:Rucio: Rucio client initialization parameters: ...
2025-01-30 21:00:09,585:INFO:makeRucioRules: 
Creating rule for DID: /JetMET0/Run2024D-2024CDEReprocessing-v1/DQMIO, with RSE: T1_US_FNAL_Disk and other attrs: {'copies': 1, 'activity': 'Production Input', 'lifetime': None, 'account': 'wmcore_transferor', 'grouping': 'ALL', 'comment': 'WMCore MSTransferor input data placement'}
2025-01-30 21:00:09,647:INFO:makeRucioRules: Response: ['5d3b05b9a8e5419a814e2ef0f1bf7056']

FYI @mapellidario

@mapellidario
Copy link
Member

I injected the wfs [1] to cmsweb-testbed / vocms0192

[1]

dmapelli_DQMHarvest_RunWhitelist_test_wfjson_v1_250131_114159_8688
dmapelli_DQMHarvesting_test_wfjson_v1_250131_114439_891
dmapelli_DQMHarvesting_LumiMask_MultiRun_test_wfjson_v1_250131_114623_2896

@dmwm-bot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
  • Python3 Pylint check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/321/artifact/artifacts/PullRequestReport.html

@dmwm-bot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/322/artifact/artifacts/PullRequestReport.html

@amaltaro amaltaro force-pushed the fix-11860 branch 2 times, most recently from c0845b5 to 3ec9e56 Compare February 1, 2025 13:07
amaltaro and others added 4 commits February 1, 2025 08:09
@amaltaro
Two fully-featured ReReco templates; remove unneded; move others to I…
529aefc 
…ntegration

Two fully-featured ReReco templates; remove unneded; move others to Integration

Remove unneeded ReReco template

Remove unneeded parents ReReco workflow
@amaltaro
StepChain template from scratch + PREMIX, 5 steps
6dc6ea3 
Set AAA for secondary
@amaltaro
Update list of dev pileups in MSPileup
94340ad
@amaltaro @mapellidario
Refresh all DQMHarvest templates
a107f08 
update number of lumis

remove a closing brackets

Co-authored-by: Dario Mapelli <[email protected]>
@amaltaro
Copy link
Contributor Author

amaltaro commented Feb 1, 2025

Rules for the ReReco templates have been created:

Creating rule for DID: ['/MinimumBias/Run2023C-22Sep2023_v4-v1/MINIAOD#146eb70b-12ad-47dd-a109-b7feef4927cf', '/MinimumBias/Run2023C-22Sep2023_v4-v1/MINIAOD#5b047ae4-f01e-44b4-b960-ae443ed46c2e', '/MinimumBias/Run2023C-PromptReco-v4/AOD#8ba900bd-a353-47d6-aeca-e81e875e2690', '/MinimumBias/Run2023C-PromptReco-v4/AOD#859fe3fb-7402-49a2-9345-8f37b09de617', '/MinimumBias/Run2023C-PromptReco-v4/AOD#0880f752-9eb5-4d4c-88cc-e1c49d2d7ebe', '/MinimumBias/Run2023C-PromptReco-v4/AOD#ae818ccd-2395-43a8-be51-5d87f8c175b3'], with RSE: T1_US_FNAL_Disk and other attrs: {'copies': 1, 'activity': 'Production Input', 'lifetime': None, 'account': 'wmcore_transferor', 'grouping': 'DATASET', 'comment': 'WMCore MSTransferor input data placement'}
2025-02-01 12:31:45,577:INFO:makeRucioRules: Response: ['496efe034ffc487b8e664c0566d95d6e', '84ed8be94a0a4e989e173473684b3bfe', 'ce775b70dc094023a15e25d3a9a2d36f', 'c581cad573794a05aaec3030d9665174', '0576ca4bb7414a69b5d2c761d0cffc6e', '9b16d5edbd8649e19af65e44f91f1afb']

and

Creating rule for DID: /MuonEG/Run2022C-v1/RAW#00353169-1291-4f60-a332-b8d66d2e1647, with RSE: T2_CH_CERN and other attrs: {'copies': 1, 'activity': 'Production Input', 'lifetime': None, 'account': 'wmcore_transferor', 'grouping': 'DATASET', 'comment': 'WMCore MSTransferor input data placement'}
2025-02-01 12:55:11,708:INFO:makeRucioRules: Response: ['c833adf9da8646daa2cdb5d44666897a']

@dmwm-bot
Copy link

dmwm-bot commented Feb 1, 2025

Jenkins results:

  • Python3 Unit tests: succeeded
    • 5 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/324/artifact/artifacts/PullRequestReport.html

@dmwm-bot
Copy link

dmwm-bot commented Feb 1, 2025

Jenkins results:

  • Python3 Unit tests: succeeded
    • 6 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/325/artifact/artifacts/PullRequestReport.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mapellidario mapellidario mapellidario left review comments

Assignees
No one assigned
Labels
PR: Do not merge yet PR: Work in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refresh test json templates due to removal of input data
4 participants
@amaltaro @cmsdmwmbot @dmwm-bot @mapellidario