[COST-5216] Delete filtering optimization

[myersCody]

Jira Ticket

COST-5216

Description

This change will switch us to using head_object to download only the metadata during the delete filtering instead of the s3 object. This will save us some time since we will no longer need to spend cycles retrieving the additional attributes available to the object from s3 such as the ACL, owner, storage type, etc.

Testing

I wrote a little test script we can use for this:
https://gist.github.com/myersCody/0c57dc2132a7a712c035ae1210c30d20

Output:

TEST NOT MATCHING
------
2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/
Metadata: {}


------
2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/0d34ec6a-1cab-4c89-a03e-cb471c0ba067.csv
Metadata: {'test-key': '1'}


------
2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/0d34ec6a-1cab-4c89-a03e-cb471c0ba067.csv.metadata
Metadata: {}


['2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/', '2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/0d34ec6a-1cab-4c89-a03e-cb471c0ba067.csv.metadata']
TEST MATCHING
------
2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/
Metadata: {}


------
2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/0d34ec6a-1cab-4c89-a03e-cb471c0ba067.csv
Metadata: {'test-key': '1'}


------
2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/0d34ec6a-1cab-4c89-a03e-cb471c0ba067.csv.metadata
Metadata: {}


['2023/04/30/a678b047-f78e-4ad8-9fbe-1e3b73bd5a24/0d34ec6a-1cab-4c89-a03e-cb471c0ba067.csv']

Release Notes

• proposed release note

* [COST-5216](https://issues.redhat.com/browse/COST-5216) Optimize delete filtering logic to only collect necessary metadata

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.2%. Comparing base (abd4261) to head (9240cc1).

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #5197   +/-   ##
=====================================
  Coverage   94.2%   94.2%           
=====================================
  Files        376     376           
  Lines      31252   31254    +2     
  Branches    3734    3734           
=====================================
+ Hits       29426   29431    +5     
+ Misses      1163    1160    -3     
  Partials     663     663