-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #238 from iriusrisk/bugfix/OPT-793
[bugfix/OPT-793] to release/1.14.0
- Loading branch information
Showing
8 changed files
with
345 additions
and
2 deletions.
There are no files selected for viewing
Binary file added
BIN
+359 KB
docs/img/ingesting-click-logs-from-web-application-iriusrisk-diagram-full.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+224 KB
docs/img/ingesting-click-logs-from-web-application-iriusrisk-diagram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
123 changes: 123 additions & 0 deletions
123
docs/startleft-processors/iac/tfplan/Terraform-Plan-Quickstart.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
## What is Terraform Plan? | ||
|
||
--- | ||
From the <a href="https://developer.hashicorp.com/terraform/cli/commands/plan" target="_blank">official Terraform page</a>: | ||
> The terraform plan command creates an execution plan, which lets you preview the changes that | ||
> Terraform plans to make to your infrastructure. | ||
> | ||
> - Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date | ||
From the StartLeft perspective, the Terraform Plan Processor (`slp_tfplan`) leverages the Terraform CLI capabilities | ||
to generate a unique Terraform resource file with the final infrastructure and its configuration. | ||
|
||
## The `slp_tfplan` module | ||
|
||
--- | ||
The `slp_tfplan` module is the StartLeft Processor responsible for converting an existing Terraform execution plan | ||
(`tf-plan`) along with its graphic representation (`tf-graph`) into an OTM. | ||
This operation is based on a mapping file that enables the client to define the translations between the Terraform resources and the expected | ||
output in the OTM file. | ||
|
||
???+ abstract "How does it work?" | ||
|
||
We rely on the Terraform CLI to deal with the complexity of a Terraform infrastructure deployment, which is summarized in two single files: | ||
|
||
- **Terraform Plan.** It contains all the resources along with its configuration. | ||
- **Terraform Graph.** It contains the relationships between the resources. | ||
|
||
With these two files, we are able to compose a complete architecture diagram by parsing all the components and their relationships. | ||
|
||
|
||
Once you got familiarized yourself with the basics explained on this page, you will need to know more about how to | ||
use and customize the behavior of the processor. To configure your conversions, you should take a look at | ||
[TFplan mapping page](Terraform-Plan-how-to-create-a-mapping-file.md), where you will find all the information you | ||
need, from basic to advanced, to build your mapping files. | ||
|
||
Apart from this, you may also find interesting the generic usage manuals for the [CLI](../../../usage/Command-Line-Interface.md) | ||
and [REST API](../../../usage/REST-API.md). | ||
|
||
## How to use it? | ||
|
||
--- | ||
Firstly, it is necessary to generate the `tf-plan` and `tf-graph` files by the Terraform CLI. | ||
|
||
???+ abstract "Generating my own Terraform files" | ||
|
||
:zero: Execute the Terraform init (only once): | ||
``` | ||
terraform init | ||
``` | ||
|
||
:one: Generate the Terraform plan file in JSON format: | ||
``` | ||
terraform plan -out=tf-plan | ||
terraform show -json tf-plan >> tf-plan.json | ||
``` | ||
|
||
:two: Generate the Terraform graph file: | ||
``` | ||
terraform graph -type=plan -plan=tf-plan >> tf-graph.gv | ||
``` | ||
|
||
Next, you need to have a mapping file to configure the processor mapping behavior. | ||
??? example "Terraform Plan Mapping File Example" | ||
|
||
> In the [TFplan mapping page](Terraform-Plan-how-to-create-a-mapping-file.md) you will find all the information you need, from | ||
basic to advanced, to build your mapping files. | ||
|
||
```yaml | ||
--8<-- "examples/tfplan/iriusrisk-tfplan-aws-mapping.yaml" | ||
``` | ||
|
||
Now everything is set up. The last step is to execute the commands for IaC `TFPLAN` parsing to generate the OTM. | ||
|
||
### CLI | ||
> **Note**: Before continue, make sure you have | ||
> [StartLeft properly installed](../../../Quickstart-Guide-for-Beginners.md) in your machine. | ||
Save the files above in your file system with these names: | ||
|
||
* `tf-plan.json` for the Terraform plan file. | ||
* `tf-graph.gv` for the Terraform graph file. | ||
* `ir-mappings.yaml` for the mapping file. | ||
|
||
Now we are going to execute StartLeft for these files so that an `ec2.otm` file will be generated in our working | ||
directory. | ||
```shell | ||
startleft parse \ | ||
--iac-type TFPLAN \ | ||
--mapping-file ir-mappings.yaml \ | ||
--output-file output.otm \ | ||
--project-id "my-project" \ | ||
--project-name "My project" \ | ||
tf-plan.json tf-graph.gv | ||
``` | ||
|
||
### cURL | ||
You can get the same result through the StartLeft REST API. For that, first, we need to set up the | ||
server with the command: | ||
```shell | ||
startleft server | ||
``` | ||
|
||
Then, execute the following command to retrieve the OTM file: | ||
```shell | ||
curl --location --request POST localhost:5000/api/v1/startleft/iac \ | ||
--header "Content-Type: multipart/form-data" \ | ||
--header "Accept: application/json" \ | ||
--form iac_type="TFPLAN" \ | ||
--form iac_file=@"./tf-plan.json" \ | ||
--form iac_file=@"./tf-graph.gv" \ | ||
--form mapping_file=@"./ir-mappings.yaml" \ | ||
--form id="my-project" \ | ||
--form name="My project" | ||
``` | ||
|
||
## More examples | ||
|
||
--- | ||
The infrastructure built with Terraform may be as complex as you want. For that reason StartLeft, through the mapping files, | ||
is intended to be configurable, so you can extend or modify its behavior and/or create your mappings on demand. | ||
|
||
To help you to walk through more complex situations with larger Terraform and mapping files, you can see all available | ||
examples under the Terraform Examples section. |
3 changes: 3 additions & 0 deletions
3
.../startleft-processors/iac/tfplan/Terraform-Plan-how-to-create-a-mapping-file.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
|
||
At this point, the Terraform Plan StartLeft processor uses the same mapping that | ||
[Terraform mapping page](../tf/Terraform-how-to-create-a-basic-mapping-file.md). |
212 changes: 212 additions & 0 deletions
212
...left-processors/iac/tfplan/examples/aws-ingesting-click-logs-using-terraform.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,212 @@ | ||
## Threat Model Architecture Representation | ||
|
||
Let's use an Official example for | ||
<a href="https://github.com/aws-samples/aws-ingesting-click-logs-using-terraform" target="_blank"> | ||
Ingesting Web Application Click Logs</a>: | ||
|
||
> This post provides an API based ingestion application system for websites & applications to push user interactions, | ||
> click actions from their website into AWS. The ingestion process will be exposed using a web-based interaction with an API Gateway endpoint. | ||
This example gives the source code for the following AWS architecture diagram: | ||
|
||
![ingesting-click-logs-from-web-application.png](../../../../img/ingesting-click-logs-from-web-application.png) | ||
|
||
The `slp_tfplan` processor can generate an accurate threat model by only following the steps below. | ||
|
||
???+ abstract "Generating my own Terraform files" | ||
|
||
:one: Clone and prepare the example reporitory: | ||
> :material-information-outline: These previous set-up commands are required because the example needs some | ||
> lambda code to be compiled. | ||
|
||
```shell | ||
$ git clone https://github.com/aws-samples/aws-ingesting-click-logs-using-terraform.git | ||
$ cd aws-ingesting-click-logs-using-terraform/source/clicklogger | ||
$ mvn clean package | ||
$ cd ../../terraform/templates | ||
``` | ||
|
||
:two: Generate the tf-plan and tf-graph files: | ||
```shell | ||
$ terraform init | ||
$ terraform plan -out=tf-plan | ||
$ terraform show -json tf-plan >> tf-plan.json | ||
$ terraform graph -type=plan -plan=tf-plan >> tf-graph.gv | ||
``` | ||
|
||
Use the StartLeft CLI to generate the OTM file: | ||
```shell | ||
$ startleft parse \ | ||
--iac-type TFPLAN \ | ||
--mapping-file ir-mappings.yaml \ | ||
--output-file output.otm \ | ||
--project-id "my-project" \ | ||
--project-name "My project" \ | ||
tf-plan.json tf-graph.gv | ||
``` | ||
|
||
After, we need to use the Terraform CLI to create the required files and execute StartLeft parse to generate the | ||
final OTM. | ||
|
||
=== "IriusRisk Threat Model Example" | ||
|
||
<figure markdown> | ||
![ingesting-click-logs-from-web-application-iriusrisk-diagram.png](../../../../img/ingesting-click-logs-from-web-application-iriusrisk-diagram.png) | ||
</figure> | ||
|
||
=== "Mapping file" | ||
|
||
```yaml | ||
--8<-- "examples/tfplan/iriusrisk-tfplan-aws-mapping.yaml" | ||
``` | ||
=== "OTM" | ||
|
||
```yaml | ||
otmVersion: 0.1.0 | ||
project: | ||
name: name | ||
id: id | ||
representations: | ||
- name: Terraform | ||
id: Terraform | ||
type: code | ||
trustZones: | ||
- id: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
name: Public Cloud | ||
risk: | ||
trustRating: 10 | ||
components: | ||
- id: aws_dynamodb_table.click-logger-table | ||
name: click-logger-table | ||
type: dynamodb | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_dynamodb_table | ||
- id: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream | ||
name: click_logger_firehose_delivery_stream | ||
type: kinesis-data-firehose | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_kinesis_firehose_delivery_stream | ||
- id: aws_lambda_function.lambda_clicklogger | ||
name: lambda_clicklogger | ||
type: aws-lambda-function | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_lambda_function | ||
- id: aws_lambda_function.lambda_clicklogger_authorizer | ||
name: lambda_clicklogger_authorizer | ||
type: aws-lambda-function | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_lambda_function | ||
- id: aws_lambda_function.lambda_clicklogger_stream_consumer | ||
name: lambda_clicklogger_stream_consumer | ||
type: aws-lambda-function | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_lambda_function | ||
- id: aws_s3_bucket.click_logger_firehose_delivery_s3_bucket | ||
name: click_logger_firehose_delivery_s3_bucket | ||
type: s3 | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_s3_bucket | ||
- id: aws_api_gateway_account.click_logger_api_gateway_account | ||
name: api-gateway (grouped) | ||
type: api-gateway | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_api_gateway_method_response | ||
- aws_api_gateway_method_settings | ||
- aws_api_gateway_integration_response | ||
- aws_api_gateway_method | ||
- aws_api_gateway_model | ||
- aws_api_gateway_request_validator | ||
- aws_api_gateway_integration | ||
- aws_api_gateway_account | ||
- aws_api_gateway_resource | ||
- aws_api_gateway_rest_api | ||
- aws_api_gateway_authorizer | ||
- aws_api_gateway_deployment | ||
- id: aws_cloudwatch_log_group.click_logger_firehose_delivery_stream_log_group | ||
name: cloudwatch (grouped) | ||
type: cloudwatch | ||
parent: | ||
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91 | ||
tags: | ||
- aws_cloudwatch_log_group | ||
dataflows: | ||
- id: cb8953f8-1436-47a6-8939-ffcd07902614 | ||
name: clicklogger-authorizer to lambda_clicklogger_authorizer | ||
source: aws_api_gateway_account.click_logger_api_gateway_account | ||
destination: aws_lambda_function.lambda_clicklogger_authorizer | ||
bidirectional: false | ||
- id: 4f25e0b3-0b8a-4e84-90db-f0a8de34467e | ||
name: integration to lambda_clicklogger | ||
source: aws_api_gateway_account.click_logger_api_gateway_account | ||
destination: aws_lambda_function.lambda_clicklogger | ||
bidirectional: false | ||
- id: 66cecdff-ba09-42b1-a75f-8e98fdae2dc4 | ||
name: lambda_click_logger_authorizer_log_group to lambda_clicklogger_authorizer | ||
source: aws_cloudwatch_log_group.click_logger_firehose_delivery_stream_log_group | ||
destination: aws_lambda_function.lambda_clicklogger_authorizer | ||
bidirectional: false | ||
- id: e01769b2-d461-4c4d-b7aa-cc028f4533ab | ||
name: lambda_click_logger_log_group to lambda_clicklogger | ||
source: aws_cloudwatch_log_group.click_logger_firehose_delivery_stream_log_group | ||
destination: aws_lambda_function.lambda_clicklogger | ||
bidirectional: false | ||
- id: 9618b94c-b899-45d0-ba70-61d141e2ad54 | ||
name: click_logger_firehose_delivery_stream to lambda_clicklogger_stream_consumer | ||
source: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream | ||
destination: aws_lambda_function.lambda_clicklogger_stream_consumer | ||
bidirectional: false | ||
- id: e4431bbb-a8d5-413e-be2b-8c5b0cc06d81 | ||
name: click_logger_firehose_delivery_stream to | ||
click_logger_firehose_delivery_s3_bucket | ||
source: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream | ||
destination: aws_s3_bucket.click_logger_firehose_delivery_s3_bucket | ||
bidirectional: false | ||
- id: 3c58eb54-f1eb-4762-9f2b-dd88793032e3 | ||
name: lambda_clicklogger to click_logger_firehose_delivery_stream | ||
source: aws_lambda_function.lambda_clicklogger | ||
destination: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream | ||
bidirectional: false | ||
- id: 50058e60-209a-4a2f-99b3-066bb633e2eb | ||
name: lambda_clicklogger_stream_consumer to click-logger-table | ||
source: aws_lambda_function.lambda_clicklogger_stream_consumer | ||
destination: aws_dynamodb_table.click-logger-table | ||
bidirectional: false | ||
``` | ||
|
||
As we can see in the previous example, StartLeft can automatically reproduce an almost exact architecture diagram than the one present in the example website. | ||
|
||
## Full Architecture Representation | ||
This approach enables StartLeft not only to create threat models but also to explore all the components and the dataflows among them, | ||
by configuring this behavior in the mapping file. In that way, you can get this diagram from the example above: | ||
|
||
=== "IriusRisk Threat Model Example" | ||
|
||
<figure markdown> | ||
![ingesting-click-logs-from-web-application-iriusrisk-diagram-full.png](../../../../img/ingesting-click-logs-from-web-application-iriusrisk-diagram-full.png) | ||
</figure> | ||
|
||
=== "Mapping file" | ||
|
||
> The following component mappers need to be added in the mapping file to enable this feature. | ||
|
||
```yaml | ||
- type: empty-component | ||
$source: { $skip: { $type: ["aws_db_subnet_group", "aws_internet_gateway"] } } | ||
|
||
- type: empty-component | ||
$source: { $singleton: { $catchall: { $type: { $regex: ^aws_\w*$ } } } } | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters