Create new main ProductRegistry if input products change #47584

Dr15Jones · 2025-03-12T19:21:01Z

PR description:

Instead of having one ProductRegistry shared by many objects, now if the ProductRegistry used by the Source adds new entries (which can only happen when reading a new file and the job which made the file has different dropped products) we make a new ProductRegistry and explicitly tell all Principals to update.

This would make it possible in the future to have multiple concurrent files open (open a new one while the last events in the previous file are finishing) where different Principals were using different ProductRegistries.

PR validation:

All framework unit tests pass and newly added unit test also passes.

resolves cms-sw/framework-team#1304

Only provenance of dropped ancestors are stored in ROOT files. Using untracked parameter for InputTag keeps the ProcessHistory the same so there is only one Run in the workflow.

cmsbuild · 2025-03-12T19:21:25Z

cms-bot internal usage

cmsbuild · 2025-03-12T19:23:49Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47584/44065

There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/src/EventProcessor.cc modified in PR(s): Access PathsAndConsumesOfModules from new signal #47467, Add Process name, reduced configuration ID, and ParameterSet ID to the JobReport #47548
- File FWCore/Framework/test/stubs/ToyIntProducers.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372, Removed -Wconversion warnings in part of framework #47339
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821, Implement a GenericCloner test module #47504, Remove configuration support for SubProcess and its tests #47565

cmsbuild · 2025-03-12T19:24:14Z

A new Pull Request was created by @Dr15Jones for master.

It involves the following packages:

FWCore/Framework (core)
FWCore/Integration (core)
FWCore/Modules (core)
FWCore/TestProcessor (core)

@Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @missirol, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

Dr15Jones · 2025-03-12T19:32:27Z

please test

FWCore/Framework/src/EventProcessor.cc

FWCore/Framework/src/Principal.cc

If the ProductRegistry of the source changes, we now create a new main ProductRegistry. The main Registry is then passed to the Principals which update their data structures.

Dr15Jones · 2025-03-12T20:14:27Z

please test

cmsbuild · 2025-03-12T20:14:29Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47584/44066

There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/src/EventProcessor.cc modified in PR(s): Access PathsAndConsumesOfModules from new signal #47467, Add Process name, reduced configuration ID, and ParameterSet ID to the JobReport #47548
- File FWCore/Framework/test/stubs/ToyIntProducers.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372, Removed -Wconversion warnings in part of framework #47339
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821, Implement a GenericCloner test module #47504, Remove configuration support for SubProcess and its tests #47565

cmsbuild · 2025-03-12T20:14:52Z

Pull request #47584 was updated. @Dr15Jones, @makortel, @smuzaffar can you please check and sign again.

cmsbuild · 2025-03-13T00:16:17Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 92KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b67e6/44947/summary.html
COMMIT: af6456c
CMSSW: CMSSW_15_1_X_2025-03-12-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47584/44947/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b67e6/44947/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b67e6/44947/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test TestIOPoolInputNoParentDictionary had ERRORS

Comparison Summary

Summary:

You potentially added 115 lines to the logs
Reco comparison results: 25 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3927028
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3926988
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 14690.869999999999 KiB( 49 files compared)
DQMHistoSizes: changed ( 4.22,... ): 4896.605 KiB HLT/Tracking
DQMHistoSizes: changed ( 7.3 ): 4897.660 KiB HLT/Tracking
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: found differences in 1 / 48 workflows

Dr15Jones · 2025-03-13T12:06:46Z

ignore tests-rejected with manual-override

The failing unit test is expected and will clear up when run in the IB.

makortel · 2025-03-13T13:34:28Z

Workflow 17034.0 shows differences in L1TEMU/L1TdeStage2uGT DQM folder. I'd guess they originate from #47423 that was tested at the same time, but the test results of that PR have already been cleaned up so I can't check. Let's run another round of tests just to make sure.

makortel · 2025-03-13T13:34:35Z

@cmsbuild, please test

cmsbuild · 2025-03-13T17:22:44Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 92KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b67e6/44964/summary.html
COMMIT: af6456c
CMSSW: CMSSW_15_1_X_2025-03-12-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47584/44964/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test TestDQMGUIUpload had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3927484
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3927461
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

makortel · 2025-03-13T17:40:17Z

I don't know why the DQM GUI upload failed. It can't be from this PR (and in earlier tests that test passed)

makortel · 2025-03-13T17:40:35Z

+core

cmsbuild · 2025-03-13T17:40:41Z

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

makortel · 2025-03-13T17:41:59Z

ignore tests-rejected with external-failure

mandrenguyen · 2025-03-14T06:15:26Z

+1

Test provenance of dropped products

e533960

Only provenance of dropped ancestors are stored in ROOT files. Using untracked parameter for InputTag keeps the ProcessHistory the same so there is only one Run in the workflow.

cmsbuild added this to the CMSSW_15_1_X milestone Mar 12, 2025

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Mar 12, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 12, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 12, 2025

makortel reviewed Mar 12, 2025

View reviewed changes

FWCore/Framework/src/EventProcessor.cc Outdated Show resolved Hide resolved

FWCore/Framework/src/Principal.cc Outdated Show resolved Hide resolved

Make new main ProductRegistry when input updates

af6456c

If the ProductRegistry of the source changes, we now create a new main ProductRegistry. The main Registry is then passed to the Principals which update their data structures.

Dr15Jones force-pushed the multipleProductRegistries branch from dcaae10 to af6456c Compare March 12, 2025 20:12

cmsbuild added tests-pending code-checks-pending and removed tests-started code-checks-approved labels Mar 12, 2025

cmsbuild added tests-started code-checks-approved and removed tests-pending code-checks-pending labels Mar 12, 2025

cmsbuild added tests-rejected and removed tests-started labels Mar 13, 2025

cmsbuild added tests-approved tests-manual-override and removed tests-rejected labels Mar 13, 2025

cmsbuild added tests-started and removed tests-approved tests-manual-override labels Mar 13, 2025

cmsbuild added tests-approved tests-manual-override and removed tests-started labels Mar 13, 2025

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Mar 13, 2025

cmsbuild added tests-external-failure and removed tests-manual-override labels Mar 13, 2025

cmsbuild added orp-approved and removed orp-pending labels Mar 14, 2025

cmsbuild merged commit 690f962 into cms-sw:master Mar 14, 2025
10 of 11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create new main ProductRegistry if input products change #47584

Create new main ProductRegistry if input products change #47584

Dr15Jones commented Mar 12, 2025 •

edited

Loading

cmsbuild commented Mar 12, 2025 •

edited

Loading

cmsbuild commented Mar 12, 2025

cmsbuild commented Mar 12, 2025

Dr15Jones commented Mar 12, 2025

Dr15Jones commented Mar 12, 2025

cmsbuild commented Mar 12, 2025

cmsbuild commented Mar 12, 2025

cmsbuild commented Mar 13, 2025

Dr15Jones commented Mar 13, 2025

makortel commented Mar 13, 2025

makortel commented Mar 13, 2025

cmsbuild commented Mar 13, 2025

makortel commented Mar 13, 2025

makortel commented Mar 13, 2025

cmsbuild commented Mar 13, 2025

makortel commented Mar 13, 2025

mandrenguyen commented Mar 14, 2025

Create new main ProductRegistry if input products change #47584

Create new main ProductRegistry if input products change #47584

Conversation

Dr15Jones commented Mar 12, 2025 • edited Loading

PR description:

PR validation:

cmsbuild commented Mar 12, 2025 • edited Loading

cmsbuild commented Mar 12, 2025

cmsbuild commented Mar 12, 2025

Dr15Jones commented Mar 12, 2025

Dr15Jones commented Mar 12, 2025

cmsbuild commented Mar 12, 2025

cmsbuild commented Mar 12, 2025

cmsbuild commented Mar 13, 2025

Unit Tests

Comparison Summary

Dr15Jones commented Mar 13, 2025

makortel commented Mar 13, 2025

makortel commented Mar 13, 2025

cmsbuild commented Mar 13, 2025

Unit Tests

Comparison Summary

makortel commented Mar 13, 2025

makortel commented Mar 13, 2025

cmsbuild commented Mar 13, 2025

makortel commented Mar 13, 2025

mandrenguyen commented Mar 14, 2025

Dr15Jones commented Mar 12, 2025 •

edited

Loading

cmsbuild commented Mar 12, 2025 •

edited

Loading