Our binderhubs rebuild already built images when working against private container registries

[consideRatio]

Always rebuilding makes binderhub extremely clunky, and making demo's using it less cool as it takes several minutes to complete.

I think currently our binderhub setups working against private image repositories (as happens with quay.io) re-build all the time because it fails to check if its an image that already exists, because lacking permissions granted the binderhub software doing the check.

Technical overview of permissions of relevance

1. BinderHub the software needs it.
   Before a build is created, needs to know if it should or shouldn't build/push an image, so it checks if the registry contains an image it otherwise would build already.

2. A build pod needs it
   The build pod will use a docker client, and the docker client needs to provide permissions to push to a registry even though the actual work is delegated via the docker client to a docker daemon doing the cpu heavy work of building and pushing.

   We provide this to the build pods via binderhub-service.buildPodsRegistryCredentials.

3. Started k8s pods needs it
   This includes JupyterHub user servers and dask-gateway scheduler+worker pods.

   We provide this to the k8s pods by creating a k8s Secret via the z2jh helm chart, via the imagePullSecret config. This is then referenced automatically by z2jh user pods, and by configuring dask-gateway.gateway.backend.imagePullSecrets.

What to do

I think we have configured 2 and 3, but not 1 yet - and the failure is observed by always rebuilding the image. Figure out how to configure BinderHub to acquire the relevant permissions to read from the registry.

This is only needed if we push images that become private, an option could perhaps be to get these to become public as well. With quay.io I've not managed to get new image repositories (as created by binderhub) to default to be public.

Investigation spike needed

• [Spike] [60min *max*] Investigate binderhub's image re-building logic #4340

Feature functionality that can be useful

• Add extraVolumes and extraVolumeMounts config binderhub-service#122

[sgibson91]

@consideRatio I have solved this. The config we needed was

binderhub-service:
    config:
        BinderHub:
            DockerRegistry:
                url: https://quay.io
                username: xxx
                password: xxx

We missed BinderHub key before and I added the url key in. You can try this on https://binder.opensci.2i2c.cloud

2i2c-org/binderhub-service#122 is probably still relevant as all of this config is in JupyterHub's imagePullSecret so we "just" need to mount that to the binderhub pods.

(FYI, this took me about 5 extra mins just now - I have not been working on this since finishing the spike. I realised we hadn't tried this while taking a walk.)