[COST-4912] Calculate node based unattributed storage (Azure)

[myersCody]

Jira Ticket

COST-3912

Description

This change will calculate the node based unattributed storage for Azure.

Testing

1. Checkout Branch
2. Create the test customer
3. Update the unleash flag:

def is_feature_unattributed_storage_enabled(account):
    """Should unattributed storage feature be enabled."""
    unleash_flag = "cost-management.backend.unattributed_storage"
    account = convert_account(account)
    context = {"schema": account}
    # TODO: Turn on fallback when QE is ready to start testing
    # UNLEASH_CLIENT.is_enabled(unleash_flag, context, fallback_development_true)
    # return UNLEASH_CLIENT.is_enabled(unleash_flag, context)
    return True

4. make create-test-customer
5. make load-test-customer-data test_source=Azure
6. Ensure that the Storage unattributed namespace was populated:

select sum(infrastructure_raw_cost), resource_id from reporting_ocpusagelineitem_daily_summary where namespace='Storage unattributed' group by resource_id;

        sum        |               resource_id
-------------------+-----------------------------------------
 4.236326764000000 | pv-123-claimless
 2.337303951000000 | azure-cloud-prefix-pvc-partial-matching
 1.554439789000000 | disk-id-1234567

The claimless will be be 100% unattributed cost; however, we can check our other resource to see that the cost was split:

select namespace, sum(infrastructure_raw_cost) from reporting_ocpusagelineitem_daily_summary where resource_id='azure-cloud-prefix-pvc-partial-matching' group by namespace;

      namespace       |        sum
----------------------+-------------------
 news-site            | 0.001427496000000
 Storage unattributed | 2.337303951000000

7. Undo the changes to the unleash flag:

git checkout -- koku/masu/processor/__init__.py

This will set it back to being False.
8. Resummarize the Azure source
9. Recheck the database to see if unattributed storage was added:

select count(*) from reporting_ocpusagelineitem_daily_summary where namespace='Storage unattributed';

 count
-------
    0
(1 row)

Release Notes

• proposed release note

• https://gist.github.com/myersCody/b300fea8db309e55d50a7bf020a8e4b8

[myersCody]

/retest

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.1%. Comparing base (c08f7c6) to head (a7ad4b6).

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #5066     +/-   ##
=======================================
- Coverage   94.1%   94.1%   -0.0%     
=======================================
  Files        371     371             
  Lines      30929   30938      +9     
  Branches    4542    4543      +1     
=======================================
+ Hits       29107   29113      +6     
- Misses      1163    1165      +2     
- Partials     659     660      +1     

[myersCody]

Successful Azure Smoke Run

[maskarb]

some questions and comments:

[maskarb]

(question) What is the result of max(ocp.node) here for claimless PVs? Below (in lines 461-564), when inserting for Storage unattributed, the node is null, but here it seems like the node would be not null. Is that accurate?

[myersCody]

The node column is null in the ocp daily summary table for the claimless persistent volume. So, it would be null here too:

|       node        | persistentvolumeclaim |  persistentvolume   | storageclass |       resource_id        |
| NULL              |                       | pv-volume_claimless | sc_disk_one  | abc_pv-volume_claimless  |