Arm Support Broken due to reliance on amd64-only image

[mdesson]

What happened?

When running the operator in k3d on my Apple silicon machine (version 2.12 of the operator) it never entered a healthy state.

This is because the init container uses this image:
https://hub.docker.com/r/foundationdb/fdb-kubernetes-monitor/tags

When the operator starts, an init container copies binaries from this image (fdbcli, for example).

Looking at the tags, there is no arm support in this image, causing the operator to never successfully start.

What did you expect to happen?

Versions of the operator supporting ARM work in a dockerized environment on an arm-based machine.

How can we reproduce it (as minimally and precisely as possible)?

1. Run the operator on an apple silicon machine within k3d.
2. Run the following commands:

$ kubectl exec fdb-kubernetes-operator-controller-manager-$ID_GOES_HERE -- /usr/bin/fdb/7.3/fdbcli --version
$ kubectl exec fdb-kubernetes-operator-controller-manager-$ID_GOES_HERE -- uname -m
$ kubectl logs fdb-kubernetes-operator-controller-manager-$ID_GOES_HERE | grep -A 2 -B 2 "rosetta error"

3. Observe the errors showing the fdbcli binary being x86_64.

Anything else we need to know?

No response

FDB Kubernetes operator

$ kubectl fdb version
v2.12.0

Kubernetes version

$ kubectl version
Client Version: v1.33.1
Kustomize Version: v5.6.0
Server Version: v1.32.8+k3s1

Cloud provider

N/A. Running k3d on an MacBook Pro M2.

[johscheuer]

The arm64 support is currently only experimental: https://github.com/FoundationDB/fdb-kubernetes-operator?tab=readme-ov-file#running-locally. Since FDB doesn't publish arm64 images, running the operator without your own custom build FDB images is not possible. So even if you would remove the init container things wouldn't work because there is no arm64 FDB image to use. The forum has some information on how to build the FDB images for arm64: https://forums.foundationdb.org/t/foundationdb-on-docker-mac-m2/3876/13

[mdesson]

Indeed since v2.8 there is experimental arm support, but I suspect it never worked and therefore was never really supported.

FDB does publish arm builds, for example see all the releases labelled aarch64 on the latest release:
https://github.com/apple/foundationdb/releases/tag/7.3.69

The underlying issue is that the docker image I referenced doesn't publish arm versions, all it really seems to do is copy over fdbcli, fdbrestore, and fdbbackup to another container. These binaries do exist in v7.3.63:
https://github.com/apple/foundationdb/releases/tag/7.3.63

The thread you linked mentions building foundationDB itself to make this work, but it starts and runs via the operator, it just never enters a healthy state due to the image foundationdb/fdb-kubernetes-monitor that it's reliant on.

[johscheuer]

Indeed since v2.8 there is experimental arm support, but I suspect it never worked and therefore was never really supported.

It is supported but requires some manual steps since the FDB arm64 images are not available, that's the main reason why the arm64 support of the operator is marked as experimental.

I know that FDB releases arm64 builds :) But FDB doesn't release arm64 images yet (see my previous comment). The fdb-kubernetes-monitor does a bit more work, but for the init container case it's true, it only copies the binaries inside the main container where the operator binary is running.

The thread you linked mentions building foundationDB itself to make this work, but it starts and runs via the operator, it just never enters a healthy state due to the image foundationdb/fdb-kubernetes-monitor that it's reliant on.

I'm not sure I understand what you're trying to tell me here.

If you want to try out the operator on an arm64 Kubernetes cluster, you need to build the FoundationDB image by yourself. That image will be used as an init container for the operator to get the binaries and will be used for the actual FDB cluster managed by the operator. If you don't build the arm64 image by yourself and publish it to your cluster you don't have any image available to run the FDB cluster on the arm64 Kubernetes cluster.

[mdesson]

Thanks, understood!

[johscheuer]

Since FDB publishes now ARM images, this issue should be resolved.