Heimdall

Heimdall is a dashboard for operating Flink jobs and deployments. Built-in Flink UI is extremely useful when dealing with a single job, but what if you have 10, 20 or 100 jobs? Heimdall helps to keep track of all your Flink jobs by searching, filtering, sorting and navigating them.

Read more in my blog: Heimdall: Making Operating Flink Deployments a Bit Easier.

Note: currently, Heimdall only supports Flink jobs deployed with Flink Kubernetes Operator.

Quick (and Dirty) Start

kubectl apply -f https://raw.githubusercontent.com/sap1ens/heimdall/main/tools/k8s-operator/service-account.yaml
kubectl run heimdall --image=ghcr.io/sap1ens/heimdall:0.10.0 --port=8080 --overrides='{ "spec": { "serviceAccount": "heimdall-service-account" }  }'
kubectl port-forward heimdall 8080:8080
open http://localhost:8080

Installation

Heimdall is available as a Docker container here. Since only Flink Kubernetes Operator as available at the moment you'd typically deploy Heimdall as a pod or deployment in the same cluster.

Heimdall uses port 8080 and exposes liveness and readiness endpoints at /q/health/live and /q/health/ready.

Job Locators

Job locator (implements FlinkJobLocator interface) is a mechanism for discovering running Flink jobs in Heimdall.

K8sOperatorFlinkJobLocator

Can be enabled with heimdall.joblocator.k8s-operator.enabled config. Currently, enabled by default. It loads flinkdeployment custom resource (CR) created by the Flink Kubernetes Operator.

A service account with read-only access to flinkdeployment CR is required. See tools/k8s-operator/service-account.yaml for an example.

Cache

By default, all data received from a locator is cached in memory for 5 seconds. This can be disabled or configured to use a different interval (see this page).

Configuration

Common

Environment variable	Default	Description
QUARKUS_HTTP_CORS_ORIGINS	http://localhost:5173	Comma separated list of valid origins allowed for CORS. Change to http://localhost:8001 when using kubectl proxy.
HEIMDALL_PATTERNS_DISPLAY_NAME	$jobName	Pattern for showing Flink job name. Metadata fields (e.g. Kubernetes labels) can be accessed via $metadata.labelName syntax.
HEIMDALL_ENDPOINT_PATH_PATTERNS_FLINK_UI	http://localhost/$jobName/ui	Pattern for the Flink UI endpoint. $jobName will be substituted. Set to an empty string to disable.
HEIMDALL_ENDPOINT_PATH_PATTERNS_FLINK_API	http://localhost/$jobName/api	Pattern for the Flink API endpoint. $jobName will be substituted. Set to an empty string to disable.
HEIMDALL_ENDPOINT_PATH_PATTERNS_METRICS	http://localhost/$jobName/metrics	Pattern for the Metrics endpoint. $jobName will be substituted. Set to an empty string to disable.
HEIMDALL_ENDPOINT_PATH_PATTERNS_LOGS	http://localhost/$jobName/logs	Pattern for the Logs endpoint. $jobName will be substituted. Set to an empty string to disable.

K8sOperatorFlinkJobLocator

Environment variable	Default	Description
HEIMDALL_JOBLOCATOR_K8S_OPERATOR_ENABLED	true	Is this locator enabled?
HEIMDALL_JOBLOCATOR_K8S_OPERATOR_NAMESPACE_TO_WATCH	default	Kubernetes namespace(s) to watch. Supports single namespace or comma-separated list (e.g. "default,prod,staging").

Development

This project uses:

• Java 17 and Quarkus on the back-end.
• JavaScript and Svelte on the front-end.

Running the application in dev mode

You can run your application in dev mode that enables live coding using:

./gradlew quarkusDev

Packaging and running the application

The application can be packaged using:

./gradlew build

It produces the quarkus-run.jar file in the build/quarkus-app/ directory. Be aware that it's not an über-jar as the dependencies are copied into the build/quarkus-app/lib/ directory.

The application is now runnable using java -jar build/quarkus-app/quarkus-run.jar.

If you want to build an über-jar, execute the following command:

./gradlew build -Dquarkus.package.type=uber-jar

The application, packaged as an über-jar, is now runnable using java -jar build/*-runner.jar.

Docker Image Build

Build local Docker image:

./gradlew imageBuild

Build and push multi-architecture image (AMD64/ARM64):

# Authenticate first
echo $GITHUB_TOKEN | docker login ghcr.io -u your-username --password-stdin

# Create buildx builder
docker buildx create --use --name multiarch-builder

# Build and push
docker buildx build --platform linux/amd64,linux/arm64 \
  -f src/main/docker/Dockerfile.jvm \
  -t ghcr.io/sap1ens/heimdall:latest \
  --push .

Sponsors

This project is sponsored by Goldsky ❤️.