r.2024.07.04.0

Summary:

Fix and improve S3 operations, Docker build improvements, and filtering optimization changes.

What's Changed

• [COST-5213] - fix S3 prepare by @lcouzens in #5194
• [COST-5214] Remove directive to make the build work with docker by @samdoran in #5196
• [COST-5216] Delete filtering optimization by @myersCody in #5197
• Revert "[COST-5216] Delete filtering optimization" by @myersCody in #5200
• [COST-5226] - Skip S3 delete (daily flow) if we have marked deletion complete. by @lcouzens in #5198
• [COST-5213] - Improve collection for delete from S3. by @lcouzens in #5194

Full Changelog: r.2024.06.28.0...r.2024.07.04.0

r.2024.06.28.0

Summary:

Prevent downloads queuing for a provider when a current download is already in progress

What's Changed

• COST-4844 Tweak serializer for storageclass order by @myersCody in #5174
• COST-5078 Switch to RHEL 8 VMs for CI jobs by @samdoran in #5193
• COST-5180 Only attempt to fetch new reports if current processing is complete
  by @lcouzens in #5181
• COST-5214 Update containerfile to build with podman by @samdoran in #5195

Full Changelog: r.2024.06.27.0...r.2024.06.28.0

r.2024.06.27.0

Summary:

Made a few changes to data processing to ensure tasks are sent to the correct workers and that task caching works as expected.

What's Changed

• Improve processing times by using queues more efficiently by @cgoodfred in #5186
• COST-5196 Improve processing times by using queues more efficiently by @lcouzens in #5187
• COST-5176 Correctly pass context dictionary within log_json function call by @djnakabaale in #5182
• COST-5178 Improve processing S3 deletions by @lcouzens in #5180
• Bump urllib3 from 1.26.18 to 1.26.19 in the pip group across 1 directory by @dependabot in #5172
• Add flower as a development dependency by @samdoran in #5189

Full Changelog: r.2024.06.24.0...r.2024.06.27.0

r.2024.06.24.0

Summary:

Optimizations in SQL and cost model task, and other operational improvements.

What's Changed

• [COST-4916] add v3.3.0 operator commits by @maskarb in #5143
• [COST-5124] Improve Trino migration management command by @samdoran in #5163
• Filter accounts by matching criteria during subs processing by @cgoodfred in #5184
• Update tasks.py by @lcouzens in #5185
• clean up grafana dashboard by @lcouzens in #5183

Full Changelog: r.2024.06.21.0...r.2024.06.24.0

r.2024.06.21.0

Summary:

Improvements to Trino migrations, a new data transfer direction column added to Trino tables for GCP and AWS, and a new queue to better distribute processing tasks.

What's Changed

• [COST-5141] Add command to remove expired hive partitions by @myersCody in #5166
• Update GitHub actions by @samdoran in #5165
• [COST-5141] Add group by to the outer query by @myersCody in #5170
• [COST-5141] Fix management command to use continue instead of return. by @myersCody in #5173
• [COST-5128] Process new subs tag to identify non-converted instances by @cgoodfred in #5162
• [COST-4745] Add data_transfer_direction to GCP network costs to Trino tables by @samdoran in #5130
• [COST-4741] Add data_transfer_direction for AWS network costs to Trino tables by @samdoran in #5129
• [COST-5168] - Adding new penalty pipeline by @lcouzens in #5176
• Improve our logging readability by @lcouzens in #5178
• add prometheus metrics for new queues by @lcouzens in #5179

Full Changelog: r.2024.06.17.0...r.2024.06.21.0

r.2024.06.17.0

Summary:

One new enhancement to enable saving plan costs for OCP on AWS calculations. Other minor bug fixes and enhancements detailed beow.

What's Changed

• Bump the pip group across 1 directory with 2 updates by @dependabot in #5161
• COST-5072 Check schema exists prior to querying it during HCS processing by @cgoodfred in #5137
• COST-5097 Format OCP data transfer fields like other usage fileds by @samdoran in #5152
• Remove suffix from masu trino endpoint by @myersCody in #5154
• catch None object by @maskarb in #5138
• COST-4966 Enable Savings plan costs for OCP on AWS calculations
• masu - Account for intermediate None values when getting cost model by @samdoran in #5167
• Update disable_source to work for tasks already in the download queue. by @myersCody in #5168

Full Changelog: r.2024.06.11.0...r.2024.06.17.0

r.2024.06.11.0

Summary:

Work around an issue in Azure with V2 cost exports by specifying the API version in the client.
Disable use of a new column in Trino tables due to incomplete data migrations. This will be reenabled once the Trino migrations complete successfully.

What's Changed

• Use newer API version for Azure Cost Management client by @samdoran, @lcouzens in #5158
• COST-5124 Pause use of csi_volume_handle in SQL by @myersCody in #5160

Full Changelog: r.2024.06.10.0...r.2024.06.11.0

r.2024.06.10.0

Summary:

Costs associated with ingress and egress network traffic for individual Azure nodes are now broken out separately. A new project, Network unattributed, shows costs related to node network traffic. These network costs can be distributed using a cost model in the same way as Platform and Worker unallocated costs. By default, Network unattributed costs are distributed, so existing project costs will not change.

This feature will be available for GCP and AWS in future releases.

Details on Network unattributed costs for OCP on Cloud

Previously the networking cost of the node was distributed amongst the projects on the node. Now those networking costs are removed into a separate NEW project called Network unattributed.

Example with numbers and cost model that does not distribute node network costs:

• I have a node called compute_1 and this node has 2 projects, projectA and projectB that each use 50% of the cluster leaving 0 unallocated costs.
• When I look at the costs for this node grouped by project today, projectA costs $15 and projectB costs $5 for a total of $20.
• Of that $20, I know that $5 is networking costs.
• After this change there will be 3 projects with costs for this node, projectA, projectB, and Network unattributed.
• The cost for projectA would now be $12.5, projectB would now be $2.5 and Network unattributed would be $5.
• The new Network unattributed project is the networking costs that can be specifically tied to this node but not broken down at the project level.

What's Changed

• Fix make load-test-customer-data by @maskarb in #5131
• COST-5035 Fix ocp on gcp tag matching SQL by @lcouzens in #5111
• COST-4744 Separate OCP on Azure network costs into a Network unattributed project by @cgoodfred in #5056
• COST-4962 Populate OCP Network UI Summary Tables during summary flow by @cgoodfred in #5057
• COST-4993 Unleash function to check "EC2 Individual VM Compute" feature flag by @bacciotti in #5091
• COST-4749 Add ability to distribute unattributed network cost by @cgoodfred in #5116
• COST-4906 Add CSI volume handle to hive OCP daily summary by @myersCody in #5124
• COST-5059 Reduce cadence of source status checks by @lcouzens in #5135
• COST-5023, COST-5065 Dependency update by @maskarb in #5139
• Update base image by @github-actions in #5133
• Update base image by @github-actions in #5140
• COST-4096 Migration to add volume handle to PVC topology by @myersCody in #5134
• COST-4906 Update resource matching to use CSI volume handle by @myersCody in #5132
• COST-4742 Add API for OCP on cloud network costs by @cgoodfred, @samdoran in #5088
• COST-5030 Removal of special case namespace tag matching (internal) by @lcouzens in #5142
• Update base image by @github-actions in #5141
• COST-5051 Prevent TABLE_NOT_FOUND error when looking up PVCs by @maskarb in #5120
• COST-5035 Fix OCP on GCP tag matching SQL by @lcouzens in #5144
• COST-4946 Add changes to populate AWSCostEntryLineItemSummaryByEC2Compute table by @djnakabaale in #5121
• Increase sources client resources in ephemeral by @lcouzens in #5151
• COST-5094 Data migration to default to distributing network and storage unattributed costs for existing cost models by @cgoodfred in #5145
• COST-5099 Add cost model calculation updates during OCP on Cloud processing flow by @lcouzens in #5150
• COST-5099 Change cost model update task to always run full month calculations by @lcouzens in #5149
• Update base image by @github-actions in #5156
• COST-5095 Default to distributing networking and storage unattributed if no cost model exists by @cgoodfred in #5146
• COST-5099 Don't break GCP cost models by @lcouzens in #5157

Full Changelog: r.2024.05.30.0...r.2024.06.10.0

r.2024.05.30.0

Summary:

• migrations for future features

What's Changed

• [COST-4930] Migration to drop unrequested fields from ec2-compute model by @djnakabaale in #5118
• Set locale in migration script by @bacciotti in #5126
• Add table exists check to trino migration management commands. by @myersCody in #5125
• [COST-4906] Update create table for Trino. by @myersCody in #5127
• [COST-4744] Add data_transfer_direction to create table SQL for OCP on Azure trino tables by @cgoodfred in #5128

Full Changelog: r.2024.05.23.0...r.2024.05.30.0

r.2024.05.23.0

Summary:

Some tag matching SQL fixes and model changes and migrations for AWSCostEntryLineItemSummaryByEC2Compute table.

What's Changed

• move gcp local create_daily_archives return outside for loop by @lcouzens in #5112
• [COST-5034] Fix ocp on azure tag matching SQL by @lcouzens in #5110
• [COST-5020] Use org_id as message key when producing swatch kafka messages by @cgoodfred in #5115
• [COST-4930] Migration to alter max_length for instance_name field by @djnakabaale in #5122
• [COST-4930] Increase max_length for instance_name field by @djnakabaale in #5123
• [COST-4930] Updates to AWSCostEntryLineItemSummaryByEC2Compute model by @djnakabaale in #5119

Full Changelog: r.2024.05.20.0...r.2024.05.23.0