docs: Refactor base README (#161)

With the push to release the project publicly, I wanted to refactor the base README. For internal usages it was fine, but if it's public I think it's lacking clarity on what exactly the project is trying to solve. This is a first draft on improving the messaging about what the project is, isn't, and how to get started.

There's sections that will be sparse until we have a better story around a helm chart or other release mechanisms. IE, how to operate the thing.

- refs #5

README.md

@@ -1,49 +1,64 @@
 # Cloud Cost Exporter
 
-Cloud Cost exporter is a metrics exporter designed to collect cost data from cloud providers and export the data in Prometheus format.
-This data can then be combined with usage data from tools such as stackdriver, yace, and promitor to provide a better picture of cloud costs.
+Cloud Cost exporter is a designed to collect cost data from cloud providers and export the data in Prometheus format.
+The cost data can then be combined with usage data from tools such as stackdriver, yace, and promitor to measure the spend of resources at a granular level.
 
-## Roadmap
+> [!WARNING]
+> This project is in the early stages of development and is subject to change.
+> Grafana Labs builds and maintains this project as part of our commitment to the open-source community, but we do not provide support for it.
+> In its current state, the exporter exports rates for resources and not the total spend.
+> We intend to opensource recording rules we use internally to measure the spend of resources.
+> For a better understanding of how we view measuring costs, view a talk given at [KubeCon NA 2023](https://youtu.be/8eiLXtL3oLk?si=wm-43ZQ9Fr51wS4a&t=1)
 
-The roadmap is as follows:
-- [x] GCP Cloud Storage
-- [x] AWS S3
-- [ ] Azure Blob Storage
-- [ ] GCP Cloud SQL
-- [ ] AWS RDS
+## Goals
 
-* We don't take into account currencies for now and assume all costs are in USD.
+The goal of this project is to provide a consistent interface for collecting the rate of cost data from multiple cloud providers and exporting the data in Prometheus format.
+There was a need to track the costs of both kubernetes and non-kubernetes resources across multiple cloud providers at a per minute interval.
+Billing data for each cloud provider takes hours to days for it to be fully accurate, and we needed a way of having a more real-time view of costs.
 
-## Contributing
+Primary goals:
+- Track the rate(IE, $/cpu/hr) for resources across
+- Export the rate in Prometheus format
+- Support the major cloud providers(AWS, GCP, Azure)
 
-Grafana Labs is always looking to support new contributors!
-Please take a look at our [contributing guide](CONTRIBUTING.md) for more information on how to get started.
+Non Goals:
+- Billing level accuracy
+- Measure the spend of resources
+- Take into account CUDs/Discounts/Reservations pricing information
+
+## Supported Cloud Providers
 
-## Architecture
+- AWS
+- GCP
 
-### AWS
+Azure support is planned but not yet implemented.
 
-AWS will export four metrics:
-- `aws_s3_operations_cost`
-- `aws_s3_storage_hourly_cost`
-- `aws_cost_exporter_requests_total`
-- `aws_cost_exporter_next_scrape`
+## Usage
 
-The AWS exporter is built upon [aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2).
-aws-sdk exposes pricing information in two ways:
-- [costexplore api](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/costexplorer#Client)
-- [pricing api](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/pricing#Client)
+Each tagged version will publish a docker image to https://hub.docker.com/r/grafana/cloudcost-exporter with the version tag.
+The image can be run locally or deployed to a kubernetes cluster.
+Cloud Cost Exporter has an opinionated way of authenticating against each cloud provider.
 
-We opted to use the costexplore api because AWS has a tiered pricing structure for storage costs.
-The majority of our buckets are _over_ a size threshold that makes it cost less $/GiB.
-Using the blended costs for cost explorer provides a more accurate $/GiB then if we were to use the pricing api.
-The shortcoming for this is that we only export metrics for regions we operate in _and_ we need to maintain a [mapping](https://github.com/grafana/deployment_tools/blob/f94b56492b0b4e3bd0c8641200629e2480c49f24/docker/cloudcost-exporter/pkg/aws/aws.go#L27-L54) of the way `costexplore` represents regions.
+| Provider | Notes |
+|-|-|
+| GCP | Depends on [default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) |
+| AWS | Uses profile names from your [credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) or `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION` env variables |
 
-We craft a [cost explore](https://us-east-1.console.aws.amazon.com/cost-management/home#/cost-explorer?chartStyle=STACK&costAggregate=unBlendedCost&endDate=2023-06-30&excludeForecasting=false&filter=%5B%5D&futureRelativeRange=CUSTOM&granularity=Monthly&groupBy=%5B%22Service%22%5D&historicalRelativeRange=LAST_6_MONTHS&isDefault=true&reportName=New%20cost%20and%20usage%20report&showOnlyUncategorized=false&showOnlyUntagged=false&startDate=2023-01-01&usageAggregate=undefined&useNormalizedUnits=false) query to fetch the previous months worth of billing data.
-See https://github.com/grafana/deployment_tools/blob/f94b56492b0b4e3bd0c8641200629e2480c49f24/docker/cloudcost-exporter/pkg/aws/aws.go#L219-L240 for the specific section of code that crafts the query.
-Here is a cost explore query that we generate in code: [cost explore](https://us-east-1.console.aws.amazon.com/cost-management/home#/cost-explorer?chartStyle=STACK&costAggregate=unBlendedCost&endDate=2023-07-16&excludeForecasting=false&filter=%5B%7B%22dimension%22:%7B%22id%22:%22Service%22,%22displayValue%22:%22Service%22%7D,%22operator%22:%22INCLUDES%22,%22values%22:%5B%7B%22value%22:%22Amazon%20Simple%20Storage%20Service%22,%22displayValue%22:%22S3%20(Simple%20Storage%20Service)%22%7D%5D%7D%5D&futureRelativeRange=CUSTOM&granularity=Daily&groupBy=%5B%22UsageType%22%5D&historicalRelativeRange=LAST_6_MONTHS&isDefault=true&reportName=New%20cost%20and%20usage%20report&showOnlyUncategorized=false&showOnlyUntagged=false&startDate=2023-06-16&usageAggregate=undefined&useNormalizedUnits=false)
+When running in a kubernetes cluster, it is recommended to use a service account with the necessary permissions for the cloud provider.
+- [ ] TODO: Document the necessary permissions for each cloud provider.
 
+There is no helm chart available at this time, but one is planned.
 
-### GCP
+Check out the follow docs for metrics:
+- [provider level](docs/metrics/providers.md)
+- gcp
+  - [compute](docs/metrics/gcp/compute.md)
+  - [gke](docs/metrics/gcp/gke.md)
+  - [gcs](docs/metrics/gcp/gcs.md)
+- aws
+  - [s3](docs/metrics/aws/s3.md)
 
-- TODO
+## Contributing
+
+Grafana Labs is always looking to support new contributors!
+Please take a look at our [contributing guide](CONTRIBUTING.md) for more information on how to get started.

docs/contribute/developer-guide.md

@@ -27,3 +27,11 @@ go run cmd/exporter/exporter.go -provider gcp -project-id=$GCP_PROJECT_ID -gcp.b
 # AWS - Prod
 go run cmd/exporter/exporter.go -provider aws -aws.profile $AWS_PROFILE
 ```
+
+## Project Structure
+
+The main entrypoint for the exporter is `cmd/exporter/exporter.go`. This file is responsible for setting up the exporter and starting the server.
+When running the application, there is a `--provider` flag that is used to determine which cloud provider to use. 
+Within `pkg/collector`, there are subdirectories for each cloud provider that contain the logic for collecting cost data from that provider.
+Each provider is composed of a set of _collectors_.
+Each collector represents a cloud resource such as `GKE` or `GCS`, and is responsible for collecting cost data for that resource.

pkg/aws/README.md

@@ -0,0 +1,6 @@
+# AWS
+
+This module is responsible for collecting and exporting costs associated with AWS resources.
+`aws.go` is the entrypoint for the module and is responsible for setting up the AWS session and starting the collection process.
+The module is built upon the aws-sdk-go library and uses the Cost Explorer API to collect cost data.
+

pkg/google/README.md

@@ -0,0 +1,6 @@
+# Google
+
+This package contains the Google Cloud Platform (GCP) exporter and reporter.
+`gcp.go` is responsible for setting up the GCP session and starting the collection process.
+The module is built upon the [google-cloud-go](https://github.com/googleapis/google-cloud-go) library and uses the GCP Billing API to collect cost data.
+Pricing data is fetched from the [GCP Pricing API](Pricing data is fetched from the [GCP Pricing API](https://cloud.google.com/billing/docs/how-to/understanding-costs#pricing).