Run WMCore central services validation for HG2302

[amaltaro]

Impact of the new feature
WMCore central services

Is your feature request related to a problem? Please describe.
Monthly task

Describe the solution you'd like
Validate central services in cmsweb-testbed and provide the final feedback by the February deadline, specified by the CMSWEB team.
It also includes the creation of the service release notes and the validation check-list twiki.

Describe alternatives you've considered
None

Additional context
None

[amaltaro]

@todor-ivanov Todor, as we discussed before your vacation, CMSWEB production upgrade is planned for this Thursday (Feb 9th). I just wanted to check whether you have started these tests yet?

[todor-ivanov]

Hi @amaltaro I am on it, but I am pretty sure we won't make it for Feb 9th.

[amaltaro]

@todor-ivanov then please communicate accordingly with HTTP team.

[todor-ivanov]

The postponed deployment has been communicated with CMSWEB Team already.

Given the reduced set of workflows we can use for those validations this is the result:
https://twiki.cern.ch/twiki/bin/view/CMS/WMAgentEndtoEndValidationWMC217

@amaltaro I did notice a workflow or two failing which I think were OK in the past. Those could be minor, but still I'd prefer you to take a look before we announce this release as ok. I am talking about:
https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=tivanov_SC_LumiMask_Rules_WMC217_2302_Val_230209_085305_4862

And the ERROR:

tivanov_SC_LumiMask_Rules_WMC217_2302_Val_230209_085305_4862

    agent: global_workqueue
    thread: WorkQueue
    type: agent-error
    update time: 2023/2/9 (Thu) 09:45:35 UTC
    error message:

    No work in spec: 'tivanov_SC_LumiMask_Rules_WMC217_2302_Val_230209_085305_4862' Check inputs: Failed to add work. Input data: ["input.section_('WMBS')", 'input.sandbox = None', "input.section_('splitting')", "input.splitting.section_('performance')", 'input.splitting.performance.timePerEvent = 1.0', 'input.splitting.performance.memoryRequirement = 13500.0', 'input.splitting.performance.sizePerEvent = 1.0', "input.splitting.algorithm = 'LumiBased'", 'input.splitting.lumis_per_job = 1', 'input.splitting.runs = [319450]', 'input.splitting.deterministicPileup = False', "input.splitting.lumis = ['1,100']", 'input.splitting.include_parents = False', "input.section_('dataset')", "input.dataset.section_('blocks')", 'input.dataset.blocks.whitelist = []', 'input.dataset.blocks.blacklist = []', "input.dataset.section_('runs')", 'input.dataset.runs.whitelist = []', 'input.dataset.runs.blacklist = []', "input.dataset.dbsurl = 'https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader'", "input.dataset.processed = 'Run2018C-v1'", "input.dataset.tier = 'RAW'", "input.dataset.primary = 'DoubleMuon'", "input.dataset.name = '/DoubleMuon/Run2018C-v1/RAW'"], mask: None.

    id: 4393093f1c5c7b41a785259fd963a225

[amaltaro]

@todor-ivanov Todor, indeed there should be no problem with this workflow. The input dataset + run results in 2 blocks and I see none of those blocks have any invalid files in DBS Production.

However, this workflow is configured to read the input data from DBS Integration:

  "DbsUrl": "https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader",

and it could be that someone (@vkuznet ?) replaced the DBS Integration DB by a given snapshot and/or run something in the Integration server that would mark those files as invalid. It's not the first neither the second time that I see this disparity (I also see a different number of files in those blocks, between DBS prod and int).

Valentin, have you made any changes to the DBS testbed database in the last 4 or 5 days?
Todor, we will need to switch those files in testbed to is_file_valid:1.

[todor-ivanov]

Services are already in production. I am closing this issue now.

[amaltaro]

@todor-ivanov as discussed on Slack. Please do start a discussion with Valentin to assess the situation of the DBS Integration server. We cannot afford losing more test templates, and refreshing those is very time consuming.

[vkuznet]

Somehow I missed this issue to reply, but I'm not aware that anyone has changed DBS database directly. Said that it is possible using DBSWriter account to change the data in DB though. I opened DBS ticket dmwm/dbs2go#96 to investigate this issue with DBS DB. The bottom line I'm not aware that DBS codebase (dbs2go) has any problem with query but rather we have wrong data in database itself. I suggested on a ticket that we may ask CMS DBA to replicate production DB again to testbed to resolve this issue.