- Cloud Intelligence Dashboards - Data Collection
- Table of Contents
- Overview
- Architecture of Data Exports
- Architecture of Data Collection
- Cost
- Prerequisites
- Regions
- Deployment and Cleanup Steps
- Changelogs
- Feedback
- Contribution
- Security
- License
- Notices
This repository is a part of Cloud Intelligence Dashboards, a project that provides AWS customers with a series of in-depth and customizable dashboards for the most comprehensive cost and usage details to help optimize cost, track usage goals, and achieve operational excellence.
This repository contains following elements:
- data-exports - a Cloud Formation Templates for AWS Data Exports, such as Cost and Usage Report 2.0 and others. This allows a replication of Exports from your Management Account(s) to a Dedicated Data Collection Accounts as well as aggregation of multiple Exports from a set of Linked Accounts.
- data-collection - a set of Cloud Formation Templates for collecting infrastructure operational data from Management and Linked Accounts. Such as data from AWS Trusted Advisor, AWS Compute Optimizer, Inventories, Pricing, AWS Health, AWS Support Cases etc. See more about types of data collected here.
- case-summarization - an additional Cloud Formation Template for deploying the AWS Support Case Summarization plugin that offers the capability to summarize cases through Generative AI powered by Amazon Bedrock.
- rls - a stack for managing Row Level Security for CID Dashboards.
- security-hub - Collection of data from AWS Security Hub.
All Data Collections can be used independently from Dashboards. Typically data collections store data on Amazon S3 Bucket and provide AWS Glue tables and Amazon Athena Views to explore and use these data.
- AWS Data Exports delivers daily the Cost & Usage Report (CUR2) to an Amazon S3 Bucket in the Management Account.
- Amazon S3 replication rule copies Export data to a dedicated Data Collection Account S3 bucket automatically.
- Amazon Athena allows querying data directly from the S3 bucket using an AWS Glue table schema definition.
- Amazon QuickSight datasets can read from Amazon Athena. Check Cloud Intelligence Dashboards.
See more in data-exports.
-
The Advanced Data Collection can be deployed to enable advanced dashboards based on AWS Trusted Advisor, AWS Health Events and other sources. Additional data is retrieved from AWS Organization or Linked Accounts. In this case Amazon EventBridge rule triggers an AWS Step Functions for data collection modules on a configurable schedule.
-
The "Account Collector" AWS Lambda in AWS Step Functions retrieves linked account details using AWS Organizations API.
-
The "Data Collection" Lambda function in AWS Step Functions assumes role in each linked account to retrieve account-specific data via AWS SDK.
-
Retrieved data is stored in a centralized Amazon S3 Bucket.
-
Advanced Cloud Intelligence Dashboards leverage Amazon Athena and Amazon QuickSight for comprehensive data analysis.
See more details in data-collection.
The following table provides a sample cost breakdown for deploying of Foundational Dashboards with the default parameters in the US East (N. Virginia) Region for one month.
| AWS Service | Dimensions | Monthly Cost [USD] |
|---|---|---|
| S3 | Monthly storage | $5-10* |
| AWS Lambda | On the schedule 1/14 days | $<3* |
| AWS Step Functions | On the schedule 1/14 days | $<3* |
| AWS Glue Crawler | On schedule | $<3* |
| AWS Athena | Data scanned monthly | $15* |
| Total Estimated Monthly Cost | <$50 |
* Costs are relative to the size of collected data (number of workloads, modules activated, AWS Accounts, Regions etc) and configured data collection frequency.
Pleas use AWS Pricing Calculator for precise estimation.
You need access to AWS Accounts. We recommend deployment of the Data Collection in a dedicated Data Collection Account, other than your Management (Payer) Account. You can use it to aggregate data from multiple Management (Payer) Accounts or multiple Linked Accounts.
If you do not have access to the Management/Payer Account, you can still collect some types fo data across multiple Linked accounts.
Make sure you are installing data collection in the same region where you are going to use the data to avoid cross region charges.
| Region Name | Region Code | Available |
|---|---|---|
| Africa (Cape Town) | af-south-1 | |
| Asia Pacific (Tokyo) | ap-northeast-1 | ✔️ |
| Asia Pacific (Seoul) | ap-northeast-2 | ✔️ |
| Asia Pacific (Mumbai) | ap-south-1 | ✔️ |
| Asia Pacific (Singapore) | ap-southeast-1 | ✔️ |
| Asia Pacific (Sydney) | ap-southeast-2 | ✔️ |
| Asia Pacific (Jakarta) | ap-southeast-3 | |
| Canada (Central) | ca-central-1 | ✔️ |
| China (Beijing) | cn-north-1 | |
| Europe (Frankfurt) | eu-central-1 | ✔️ |
| Europe (Zurich) | eu-central-2 | |
| Europe (Stockholm) | eu-north-1 | ✔️ |
| Europe (Milan) | eu-south-1 | |
| Europe (Spain) | eu-south-2 | |
| Europe (Ireland) | eu-west-1 | ✔️ |
| Europe (London) | eu-west-2 | ✔️ |
| Europe (Paris) | eu-west-3 | ✔️ |
| South America (São Paulo) | sa-east-1 | ✔️ |
| US East (N. Virginia) | us-east-1 | ✔️ |
| US East (Ohio) | us-east-2 | ✔️ |
| AWS GovCloud (US-East) | us-gov-east-1 | |
| AWS GovCloud (US-West) | us-gov-west-1 | |
| US West (Oregon) | us-west-2 | ✔️ |
Reference to folders.
Check Releases
Please reference to this page
See CONTRIBUTING for more information.
When you build systems on AWS infrastructure, security responsibilities are shared between you and AWS. This shared responsibility model reduces your operational burden because AWS operates, manages, and controls the components including the host operating system, the virtualization layer, and the physical security of the facilities in which the services operate. For more information about AWS security, visit AWS Cloud Security.
See SECURITY for more information.
This project is licensed under the Apache-2.0 License. See the LICENSE file.
Dashboards and their content: (a) are for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
