This demo project is intended to illustrate how to use Amazon API Gateway and AWS Lambda to set up an HTTP service, then been integrated with Kubernetes as admission webhooks to receive admission requests and mutate or validate Kubernetes resources dynamically. Particularly this project will setup a mutating webhook to modify the docker image path in K8S Pod after the deployment been submitted to K8S API server and before it's been persisted in etcd.
- The same k8s cluster need to be deployed and use different docker registry, for example k8s cluster deployed in AWS Oregon and Singapore region, and use ECR registry in local region.
- Due to firewall or security restriction, public registry cannot be accessed and need to modify the image path to access other mirrored repositories.
- Make sure that the Kubernetes cluster is at least as new as v1.9.
- Make sure that MutatingAdmissionWebhook admission controllers are enabled.
- Make sure that the admissionregistration.k8s.io/v1beta1 API is enabled.
- If you are using AWS China regions, make sure your AWS account has been whitelisted for API Gateway access, you may contact AWS support for the procedure of the whitelisting.
Amazon EKS has been enabled MutatingAdmissionWebhook.
-
Setup webhook with API GW, there are 2 options to build the API GW:
This is the recommended approach, deploy it from SAR console:
Region Click and Deploy ap-northeast-1 ap-east-1 ap-northeast-2 ap-northeast-3 ap-south-1 ap-southeast-1 ap-southeast-2 ca-central-1 eu-central-1 eu-north-1 eu-west-1 eu-west-2 eu-west-3 me-south-1 sa-east-1 us-east-1 us-east-2 us-west-1 us-west-2 cn-north-1 cn-northwest-1 Firstly please install AWS CLI and SAM CLI by following AWS SAM documentation
1). Check out this repository
$ git clone https://github.com/aws-samples/amazon-api-gateway-mutating-webhook-for-k8.git $ cd amazon-api-gateway-mutating-webhook-for-k82). Build the SAM package, replace my_s3_bucket with your S3 bucket name at first.
$ export S3_BUCKET=my_s3_bucket $ sam package -t sam-template.yaml --s3-bucket ${S3_BUCKET} --output-template-file packaged.yaml
3). Deploy SAM package
$ sam deploy -t packaged.yaml --stack-name amazon-api-gateway-mutating-webhook-for-k8 --capabilities CAPABILITY_IAM
-
Go to the cloudformation stack console, find the stack which name is serverlessrepo-amazon-api-gateway-mutating-webhook-for-k8s if you are are using SAR deployment in previous step, or find the stack with name amazon-api-gateway-mutating-webhook-for-k8, get APIGatewayURL from the outputs of this stack.
-
Create k8s MutatingWebhookConfiguration resource
- Modify mutating-webhook.yaml，replace with the value of APIGatewayURL
- Create K8S resource:
$ kubectl apply -f mutating-webhook.yaml
-
Deploy sample k8s deployment
$ kubectl apply -f ./nginx-gcr.yaml
-
Check the image path
$ kubectl get pod nginx-gcr-deployment-784bf76d96-hjmv4 -o=jsonpath='{.spec.containers[0].image}' asia.gcr.io/nginxyou may noticed the image path has been changed from "gcr.io/nginx" to "asia.gcr.io/nginx"
The most famous use case of k8s mutating webhook is the istio sidecar injector, you may also reference K8S documentation or 3rd-party blog to get understand how k8s admission works.
This project will setup the API Gateway with Lambda to receive the admission request from k8s API server, you may get k8s Pod spec from "body" object of request payload json object:
{
"kind": "AdmissionReview",
"request": {
"kind": {
"kind": "Pod",
"version": "v1",
"group": ""
},
"resource": {
"resource": "pods",
"version": "v1",
"group": ""
},
"uid": "b06b6ec2-681d-11e9-a645-06b44ed6a042",
"object": {
"status": {},
"spec": {
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"serviceAccountName": "default",
"schedulerName": "default-scheduler",
"serviceAccount": "default",
"priority": 0,
"terminationGracePeriodSeconds": 30,
"restartPolicy": "Always",
"containers": [
{
"name": "nginx",
"image": "gcr.io/nginx:latest",
"imagePullPolicy": "Always",
"ports": [
{
"protocol": "TCP",
"containerPort": 80
}
],
"resources": {}
}
]
},
"metadata": {
}
},
"namespace": "admission-test",
"userInfo": {
"username": "system:unsecured",
"groups": [
"system:masters",
"system:authenticated"
]
},
"oldObject": null,
"dryRun": false,
"operation": "CREATE"
},
"apiVersion": "admission.k8s.io/v1beta1"
}then the response of webhook(API Gateway with Lambda) is like this:
{
"body": {
"response": {
"allowed": "True",
"patch": "patch_base64",
"patchType": "JSONPatch"
},
"headers": {
"Content-Type": "application/json"
},
"statusCode": 200
}where the patch_base64 is base64 encoded of JSON Patch, for example:
[
{
"op": "replace",
"path": "/spec/containers/0/image",
"value": "xxxx.dkr.ecr.us-west-2.amazonaws.com/nginx:latest"
}
]JSON patch will be applied to the Pod spec and then persisted into etcd.
This sample code is made available under the MIT-0 license. See the LICENSE file.