[COST-3814] Add Azure source processing to subs flow

[cgoodfred]

Jira Ticket

COST-3814

Description

This change will add Azure processing to the subs flow. It extends the AWS processing to handle Azure sources. We are processing the same information as before with an additional tag option of com_redhat_rhel_instance to allow someone to add that tag instead of giving the additional Reader permission to the Azure scope.

Example of a kafka message that would be sent:

b'{"event_id": "f2105b64-1f1c-4fd6-bbaa-3ca6411f5942", "event_source": "cost-management", "event_type": "snapshot", "account_number": "10001", "org_id": "1234567", "service_type": "RHEL System", "instance_id": "<my-instance-id>", "timestamp": "2023-12-10 23:00:00+00:00", "expiration": "2023-12-11 00:00:00+00:00", "measurements": [{"value": "1", "uom": "vCPUs"}], "cloud_provider": "Azure", "hardware_type": "Cloud", "product_ids": ["69", "204"], "role": "Red Hat Enterprise Linux Server", "sla": "Premium", "usage": "Production", "billing_provider": "azure", "billing_account_id": "<my-billing-id>"}'

Testing

1. Checkout Branch
2. Restart Koku
3. Add an Azure source with the required com_redhat_rhel tag on a VM. (reach out to Corey if you need credentials for a source that has this)
4. Watch as we process azure subs records and query Azure for the VMID and then attempt to send a message to the subscriptions folks.

Notes

The AWS smokes label was run to ensure no regression in the AWS Subs Processing flow since there are not yet smokes for the Azure flow.

Example from setting only AWS enabled in the environment to give us more control over where different providers process:

subs_worker  | [2023-12-12 21:01:32,629] INFO 0ec0e507-5859-4d4f-9118-cbb6e8dedfc8 27 {'message': 'provider type not valid for subs processing', 'tracing_id': '1d5f8607-12ed-4ee4-8d1b-5566b7c0ef5a', 'schema': 'org1234567', 'provider_type': 'Azure', 'provider_uuid': '3b3b21da-03ca-414d-ab35-bdfa17f459b5'}
subs_worker  | [2023-12-12 21:01:32,629: INFO/ForkPoolWorker-1] {'message': 'provider type not valid for subs processing', 'tracing_id': '1d5f8607-12ed-4ee4-8d1b-5566b7c0ef5a', 'schema': 'org1234567', 'provider_type': 'Azure', 'provider_uuid': '3b3b21da-03ca-414d-ab35-bdfa17f459b5'}
subs_worker  | [2023-12-12 21:01:44,274] INFO 0ec0e507-5859-4d4f-9118-cbb6e8dedfc8 27 {'message': 'provider type not valid for subs processing', 'tracing_id': '413d821e-e225-4463-9666-b0d48e3a56ec', 'schema': 'org1234567', 'provider_type': 'Azure', 'provider_uuid': '3b3b21da-03ca-414d-ab35-bdfa17f459b5'}
subs_worker  | [2023-12-12 21:01:44,274: INFO/ForkPoolWorker-1] {'message': 'provider type not valid for subs processing', 'tracing_id': '413d821e-e225-4463-9666-b0d48e3a56ec', 'schema': 'org1234567', 'provider_type': 'Azure', 'provider_uuid': '3b3b21da-03ca-414d-ab35-bdfa17f459b5'}

...

[sonarcloud]

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
91.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[djnakabaale]

lgtm! 👍🏾

[codecov]

Codecov Report

Merging #4831 (a9e27af) into main (5fdf225) will decrease coverage by 0.0%.
The diff coverage is 90.8%.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #4831     +/-   ##
=======================================
- Coverage   94.0%   94.0%   -0.0%     
=======================================
  Files        364     364             
  Lines      30132   30204     +72     
  Branches    3589    3598      +9     
=======================================
+ Hits       28312   28379     +67     
- Misses      1160    1164      +4     
- Partials     660     661      +1