ADDF is a collection of modules for Scene Detection, Simulation (mock), Visualization, Compute, Storage, Centralized logging etc, deployed using SeedFarmer orchestration tool. ADDF allows you to build distinct, stand alone Infrastructure as code (IAAC) modules and exchange information about dependencies using metadata which can be exported from one module and imported into another. Each module can be found under the modules directory of this repository.
You can refer to the SeedFarmer guide to understand how SeedFarmer CLI can be used to bootstrap and deploy ADDF.
You can follow instructions in the Deployment Guide Readme. You can also follow the blogpost for understanding ADDF in detail.
| Type | Description |
|---|---|
| Rosbag Scene Detection Module | Deploys a Rosbag Scene Detection pipeline for use in ADDF |
| Rosbag WebViz Module | Deploys and Visualizes Rosbag Data on AWS using Webviz for use in ADDF |
| Type | Description |
|---|---|
| Networking Module | Deploys standard networking resources such as VPC, Public and Private subnets, endpoints for use in ADDF |
| DataLake Buckets Module | Deploys shared datalake buckets such as input, intermediate, output, logging, artifact buckets for use in ADDF |
| Type | Description |
|---|---|
| EKS Compute Module | Deploys shared EKS Cluster with commonly preferred addons for use in ADDF |
| Metadata Storage Module | Deploys shared Metadata storage module which deploys metadata resources like Glue, DDB etc for use in ADDF |
| Amazon Managed Workflows for Apache Airflow (MWAA) Module | Deploys shared MWAA module for use in ADDF |
| Opensearch Module | Deploys Amazon Opensearch Cluster for use in ADDF |
| Neptune Module | Deploys Amazon Managed Neptune Cluster for use in ADDF |
| Type | Description |
|---|---|
| DDB to Opensearch Module | This module integrates DynamoDB table with Opensearch cluster |
| EKS to Opensearch Module | This module integrates EKS Cluster with Opensearch cluster |
| EMR to Opensearch Module | This module integrates EMR Cluster with Opensearch cluster |
| Opensearch Proxy Module | This module deploys a Proxy server to access Opensearch cluster |
| Type | Description |
|---|---|
| K8s-Managed Module | This module helps running simulations on AWS EKS, when triggered by KubernetesJob Operator from airflow environment |
| AWS Batch Module | This module helps running simulations on AWS Batch, when triggered by Batch Operator from airflow environment |
| Type | Description |
|---|---|
| Self Managed JupyterHub Module | This module deploys self managed JupyterHub environment on AWS EKS |
| Self Managed VSCode Module | This module deploys self managed VSCode environment on AWS EKS |
| AWS Managed EMR Studio Module | This module deploys AWS managed EMR Studio with EMR on EKS |
| Type | Description |
|---|---|
| Example DAG Module | This module deploys a pattern to integrate a target DAG module to work with shared MWAA Cluster |
If you notice a defect, feel free to create an Issue
If you need to debug a deployment in ADDF, here are few things you can checkout Readme