feat: Migrating Flink-emr-eks to V2#929
Conversation
|
@bbgu1 Thanks for the PR! Will you please rebase your PR so that we can merge your changes to EMR on EKS datastack? |
resolved the conflict - ready for merge now |
| # IRSA for flink pods for "flink-team-a" | ||
| #--------------------------------------------------------------- | ||
| module "flink_irsa_jobs" { | ||
| source = "aws-ia/eks-blueprints-addon/aws" |
There was a problem hiding this comment.
We need to move away from eks-blueprints-addon and IRSA (where possible). Something like this: https://github.com/awslabs/data-on-eks/blob/v2/infra/terraform/datahub.tf
There was a problem hiding this comment.
There was a problem hiding this comment.
Move these resources to https://github.com/awslabs/data-on-eks/blob/v2/infra/terraform/emr-on-eks-flink.tf and a conditon of enable_emr_on_eks_flink=true
| @@ -0,0 +1,171 @@ | |||
| resource "kubernetes_namespace_v1" "flink_team_a" { | |||
There was a problem hiding this comment.
See if we can reuse the existing teams created for EMR Spark. We can use the same users instead of creating a new one for flink. https://github.com/awslabs/data-on-eks/blob/804202e2d7c13095c3fbdc42ea45db93d459b2fe/infra/terraform/emr-on-eks-spark.tf#L7C1-L76C2
| #--------------------------------------------------------------- | ||
| # S3 bucket for Flink related data,logs and checkpoint | ||
| #--------------------------------------------------------------- | ||
| module "flink_s3_bucket" { |
There was a problem hiding this comment.
EMR on EKS stack is creating s3 bucket which we can reuse for Flink as well
There was a problem hiding this comment.
Move these resources to https://github.com/awslabs/data-on-eks/blob/v2/infra/terraform/emr-on-eks-flink.tf and a condition of enable_emr_on_eks_flink=true
3 participants
What does this PR do?
Migrating Flink-on-EMR-on-EKS pattern to V2
🛑 Please open an issue first to discuss any significant work and flesh out details/direction. When we triage the issues, we will add labels to the issue like "Enhancement", "Bug" which should indicate to you that this issue can be worked on and we are looking forward to your PR. We would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.
Motivation
More
website/docsorwebsite/blogsection for this featurepre-commit run -awith this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes
The same EMR-on-EKS infrastructure can be used for emr-eks-spark as well.