Skip to content

Conversation

@timj
Copy link
Member

@timj timj commented Nov 18, 2025

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes
  • (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

Copy link
Contributor

@erykoff erykoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a changelog fragment!

And are you sure you don't want a cap on (e.g.) fits files as well? It's not like a fits file with 400Mb of header provenance would be something people want.

@timj
Copy link
Member Author

timj commented Nov 18, 2025

I usually add the change log fragment once I know it's not all changing.

@TallJimbo you implied that FITS outputs won't really have the tens of thousands of inputs but I can patch obs_base if we think that is a risk. I have pondered whether we can auto switch from including the dataset type and run to not including them but having 3x the number of inputs -- easy to implement but it might be confusing.

@TallJimbo
Copy link
Member

Guarding against massive provenance counts in the FITS variant seems prudent even if we don't expect to hit those limits.

But I don't think we should try to support UUID-only provenance as an alternative just to move the cap a little higher. The fallback can just be external (QG) provenance.

@codecov
Copy link

codecov bot commented Nov 18, 2025

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.42%. Comparing base (f17f41f) to head (806b7c3).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
python/lsst/daf/butler/_dataset_provenance.py 80.00% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1289   +/-   ##
=======================================
  Coverage   89.42%   89.42%           
=======================================
  Files         367      367           
  Lines       49540    49554   +14     
  Branches     5837     5839    +2     
=======================================
+ Hits        44299    44313   +14     
  Misses       3856     3856           
  Partials     1385     1385           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

timj added 3 commits November 18, 2025 16:25
If the number of inputs are very large we need options to
control how to serialize the provenance.
@timj timj merged commit 127869a into main Nov 19, 2025
24 checks passed
@timj timj deleted the tickets/DM-53326 branch November 19, 2025 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants