Move from dockerhub to quay.io (and consider we got an 800$ charge!)

[yuvipanda]

@choldgraf's card is apparently attached to the jupyterhub org on dockerhub, and it just got an 800$ charge!

This is not supposed to happen, as we have 'sponsored oss' status (I'm not sure who got that for us, but THANK YOU to whoever that was).

While I guess this charge is a mistake on their part, it at least feels sufficient motivation for me to move us completely over to quay.io. We already moved repo2docker (jupyterhub/repo2docker#1076 (comment)), and while I can't find the issue right now, the 'sponsored OSS' status was why we didn't move everything else. Time to move it now I think.

Repos to move

• z2jh Use quay.io as source of docker images zero-to-jupyterhub-k8s#3254
• jupyterhub Move from dockerhub to quay.io jupyterhub#4612
• configurable-http-proxy Move from DockerHub to quay.io configurable-http-proxy#508
• repo2docker
• docker-image-cleaner
• binderhub Move to quay.io binderhub#1789
• kubespawner default Pull default image from quay.io, not dockerhub kubespawner#800
• dockerspawner default Switch default image to quay.io dockerspawner#504
• mybinder.org-deploy

[sgibson91]

• JupyterHub and Jupyter got into the open-source program for dockerhub! #559

Cc @consideRatio and @mathbunnyru who lead that work

[consideRatio]

Was @choldgraf's card charged and is it now removed so it can't be charged going onards?

[minrk]

👍 to moving everything to quay and leaving docker. I don't think anything's been charged, but let's make sure we don't give them a chance to charge in the future.

[mathbunnyru]

I see this charge on the jupyter DockerHub account as well (also unsuccessful), so we will also have to move it.

I think I understand what's going on.

https://www.docker.com/community/open-source/application/

Core contributors of your project namespace get access to a year-long Docker Team subscription

#559 (comment)

And, based on this message, it was exactly one year ago when the upgrade to Docker Team happened.
I didn't think that it would only last a year, and then Docker would charge us for the next year - doesn't really sound like a "Docker-Sponsored Open Source Program", more like "Docker Team One Year Trial".
I missed it a year ago, I'm so sorry.
Hope no one was charged (at least DockerHub says the payment failed, which is good).

[mathbunnyru]

👍 to moving the jupyter account to quay as well.

I will work on jupyter/docker-stacks move this or next week (I suppose it will be the most difficult repo to move, so it will take a bit of time).

[manics]

Could we push the important images (mostly jupyterhub) to both docker hub and quay.io for a release cycle or two, but update all docs, config and example repos to use quay.io? That'll ensure anyone pulling/checking latest will still get updates in the short term.

[choldgraf]

I've sent a support request to Docker asking them WTF is going on. I'll let you all know if I hear back. I definitely agree we should just leave the platform because this is too much hassle if there are alternatives that work just as well.

I'd suggest we:

• Try to get one more year of service.
• But plan to migrate off ASAP.
• Use that year as buffer to give us some breathing room.
• Cancel service as soon as we're migrated off.

[manics]

I don't think we need another year of service (though there's no harm if someone wants to apply). It means people may hit the rate limits when pulling, but hopefully that means they'll look at the readme and see that they should move to quay.io

[yuvipanda]

I built a small tool that'll migrate some but not all tagged images across the registries! https://github.com/yuvipanda/move-off-dockerhub.

I've moved k8s-hub image's released versions only over to quay.io now, so if people do end up being throttled by dockerhub, they can simply point to quay.io and get the exact same image (as it's copied).

I'll do this for all the z2jh and chp images, and the JupyterHub ones as well.

@manics we can do this in reverse for a bit to dockerhub as well.

[mathbunnyru]

@yuvipanda please, change the email of the jupyterhub quay.io account to [email protected]. It will only change the icon for the organization (and the images).

[mathbunnyru]

After 2 days of work with ~25 commits, I've finally switched http://github.com/jupyter/docker-stacks/ to Quay.io.
An example: https://quay.io/repository/jupyter/base-notebook
Only 1 of these 25 commits is actually renaming (mostly, sed-like), all the others are gonna stay if we ever decide to change the registry again.

Cool features work fine too - automatic registry description update, docker manifest to create multi-arch images, testing contributed recipes, and automatic wiki update.

Nice features of Quay.io I noticed:

• the tags page visually shows which tags have the same content. As we push many tags for one image, this is useful
• Security scan is built-in for free on the same page (remote Docker Scout is not free, as far as I know)
• Having usage logs for free is nice as well
• the tag history page shows when a tag was moved - it might be helpful during debugging (and lots of our tags are moving)

Things I don't like/don't work:

• The first opening of Quay.io where you need to set which cookies you allow is a banner. It doesn't look/feel nice and is so slow.
• I don't think there are badges with the number of stars/pulls/size of an image (currently, I use shields.io with Docker Hub values)

[choldgraf]

I went ahead and applied for another year of Open Source account status with DockerHub, and got an e-mail that it was approved today. Could somebody confirm that we do in-fact have this status now?

[consideRatio]

Not yet, but I recall that this wasn't reflected instantly last time either.

[manics]

I received an email yesterday:

Hello jupyterhub!

We’re sad to see you go. This email confirms that your Docker Team - Annual subscription for your account has been canceled.

🤷🏽

[consideRatio]

Dockerhub status looks like this:

I received this email:

[yuvipanda]

At least once the JupyterHub image moves are complete, we should go to all the READMEs in the jupyterhub dockerhub and write them out as redirects.

[yuvipanda]

Ok, everything that needs a PR now has a PR :) Reviews would be appreciated!

[yuvipanda]

@mathbunnyru just as I moved old jupyterhub images to quay.io, I want to move old jupyter/docker-stacks images as well. Do you have objections to that?

[mathbunnyru]

@mathbunnyru just as I moved old jupyterhub images to quay.io, I want to move old jupyter/docker-stacks images as well. Do you have objections to that?

I have a few:

1. We don’t have classical releases. And we build at least weekly, meaning we have much more images than other projects.
2. Some of our images are quite large. I suppose combining with first issue it will be difficult to move all of them.
3. Our current tags in Quay.io definitely clash with our previous tags with Docker Hub. We shouldn’t rewrite these tags with older images.
4. There were lots of incorrect tags in DockerHub till 2022-07-05.
5. Till that date our images were sometimes not inherited properly one from another. So, a new image was inherited from an old one, for example.

I didn’t want this mess to be present in Quay.io.

@miraculixx answering your questions.

I intentionally didn’t push old images to Quay.io, mostly for the reasons mentioned above.
All new updates will only be provided on Quay.io.
At the same time, we do not plan to shut down Docker Hub - it will definitely affect our users in a bad way and we don’t want that. At the same time I expect most of our users to move to Quay.io images in a few years, especially because of python eol.

[manics]

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

[mathbunnyru]

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

I think that’s a good idea.

[mathbunnyru]

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

@manics @yuvipanda @miraculixx done in jupyter/docker-stacks#2032
There is also a small workflow - so if we ever need to move something, we can reuse it.

In a few days I will update the table and the information about using Docker Hub for old images.

Update: done

[yuvipanda]

Can someone else move the manifests / descriptions from dockerhub to quay.io? I don't think I'll have time to do that, manually or automatically, for at least another week or so :(

[mathbunnyru]

Can someone else move the manifests / descriptions from dockerhub to quay.io? I don't think I'll have time to do that, manually or automatically, for at least another week or so :(

@yuvipanda done for 4 images in jupyterhub repo, please, take a look: jupyterhub/jupyterhub#4634

[manics]

Did you intentionally omit the arm64 images when copying the old docker-stack images?

[mathbunnyru]

Did you intentionally omit the arm64 images when copying the old docker-stack images?

Nope, that's a bug, thank you!

Do you maybe know an easy way to fix this?
Afaik, I can specify arch, when pulling, but only one, and that doesn't help much.

I know this solution:

1. Pull aarch64-<tag> and x86_64-<tag> single arch images
2. Push these images with only changing the registry
3. Use docker manifest to merge these images and create one multi-platform image <tag>. The previous step is needed otherwise docker manifest tells something about security.

Update: Implemented this solution: jupyter/docker-stacks#2035
But I don't really like it - it doesn't handle a case where an image exists only for one arch

[manics]

@yuvipanda suggested using skopeo copy --multi-arch all in jupyterhub/zero-to-jupyterhub-k8s#3254 (comment)

[mathbunnyru]

@manics thanks. I fixed the problem.

[consideRatio]

I'd like to publish to both DockerHub and Quay.io instead of switching over fully, is that okay?

Both of these container registries are free services under no obligation to stay functional and free for us to publish to and users to pull from, so it could make sense to publish images to more than one container registry to allow us to switch over if there are issues for some or all users.

Publishing to both seems like a kinder transition than to stop publishing to docker hub, as that doesn't break workflows like building and using a custom image for each release.

As an example, here is a PR for configurable-http-proxy that accomplishes this, its a minimal change that will only add the extra upload time.

[consideRatio]

If there are no objections, I'd like to help all repos publish to Docker Hub alongside Quay.io.

Repos to publish to both Docker Hub and Quay.io

• z2jh Publish to Docker Hub alongside Quay.io zero-to-jupyterhub-k8s#3272
• configurable-http-proxy Publish to Docker Hub alongside Quay.io configurable-http-proxy#514
• jupyterhub Publish to Docker Hub alongside Quay.io jupyterhub#4641
• binderhub Publish to Docker Hub alongside Quay.io binderhub#1796
• repo2docker (ignore, been at Quay.io for a long time)
• docker-image-cleaner (ignore, been at Quay.io for a long time)
• mybinder.org-deploy ([ignore, not important - currently using docker hub)

[manics]

I think it makes sense to publish JupyterHub and CHP to both since they're the base images that people use directly. Z2JH and perhaps BinderHub makes sense from the perspective of redundancy.

I don't think we should bother with repo2docker though since we made the switch a long time ago and we're beyond the point where people still expect to find it on Docker Hub.

I also don't see the point of pushing mybinder.org-deploy to both registries. It's our production deployment which we fully control, and although it's a good example of how to go about deploying a mybinder type service there's no expectation that it'll work for anyone else. The registry serves the purpose of being an internal container distribution service amongst the mybinder federation members which just happens to be public.

[consideRatio]

👍 thanks for the feedback on this @manics and @minrk! There are now four open PRs opened to review about this.

[minrk]

I agree with @manics on most points. I don't really see the benefit for the helm chart images, since those aren't images that users use directly, the URLs are internal to the chart. So my inclination would be just the standalone images that haven't migrated long ago, (i.e. just jupyterhub and chp, I think).

[consideRatio]

I'd like to retain publishing to z2jh because the jupyterhub/k8s-hub image is not seldom built from, for example 2i2c is maintaining a self-managed image we bump to and rebuuild whenever we bump the chart. Its simple to migrate of course but still also easy to publish to both still.

Also, its a failsafe for the future. With charts relying on access to various images, if quay.io is down but we have published all images to docker hub as well, the fix is just a config update for all users with issues.

---

From 2i2c's meta chart depending on JupyterHub, but building its own custom hub pod image:

[yuvipanda]

+1 to what @minrk and @manics said. I think it will help move people away from dockerhub, towards quay.io. For people who are inheriting from one of the k8s images, to bump up version numbers they have to change the FROM line anyway to specify new tags, so I think it's fine that they have to put a quay.io on front.

I'm also personally kinda pissed off and disappointed at dockerhub, given this is like the 3rd or 4th time (since this) I've been part of a community effort scrambling to try to figure out if we need to move off it or not, and it eventually 'working out' that an imminent move is not exactly needed - feels a little bit like hostage taking, and I don't wanna do that again. So, from an ideological perspective, I would like us to maintain as minimal a presence on dockerhub as possible, and make sure that our primary registry is quay.io.

[manics]

How about if we only push Z2JH releases to Docker Hub? I think that's a nice compromise between not supporting Docker Hub, whilst still allowing some flexibility and redundancy for admins.

[yuvipanda]

Given @consideRatio has already done the work in a bunch of PRs, I want to respect that. Perhaps what we can do is:

1. Merge those, so we simultaneously publish to both dockerhub and quay.io
2. Set a 1 year deadline - don't renew the dockerhub open source program
3. Write a blog post about moving our primary host from dockerhub to quay.io, and recommend people move. Describe that in 1 year, there will be pull limits from dockerhub, and new images may not be published there. Describe how to move.
4. In all public materials, refer to quay.io as our primary host, and link to Dockerhub as a 'secondary host' available as backup.
5. Push to more secondary hosts before the end of the year as redundancy. I think this would mean GitHub to start with.
6. Once the year is up, publish another blog post about how users should be using images on quay.io instead.

I think this eases users out, which I think is @consideRatio's original goal in making these PRs. I think any org that allows pulling from dockerhub will also allow pulling from quay.io (or GitHub registry), so I think the long term flexibility is not that limited.

How does this sound?

[minrk]

That all sounds great and sensible, thanks @consideRatio and @yuvipanda!

[consideRatio]

5. Push to more secondary hosts before the end of the year as redundancy. I think this would mean GitHub to start with.

👍 for use GitHub's ghcr.io as the secondary container registry, I've already used it a bit in the dask org for dask-gateway, its worked out fine and has some nice features related to permission control for example. There are some relevant notes about this in this long post, copied here for future reference:

• ghcr.io release preparation
• BLOCKING UPSTREAM: It appears that the instructions at https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#inheriting-access-for-a-container-image-from-a-repository are outdated, and I've failed to manage to push a public image to ghcr.io. I will investigate this further.
  UPDATE: This was resolved, the UI had changed, but there was a "package settings" after all but a bit tricky to find.
• Publish a dummy ghcr.io/dask/dask-gateway and ghcr.io/dask-gateway-server container manually so we can configure settings.
  1. You must be a Dask organization admin
  2. You must create a personal access token (PAT) for your account with write:packages permissions at https://github.com/settings/tokens/new.
  3. Login to the ghcr.io container registry with the PAT:

     docker login ghcr.io -u your-username
     

  4. Push a dummy image with a label to couple it with a github repo and enable us to configure permissions

     echo "FROM alpine" > Dockerfile
     docker build . --tag ghcr.io/dask/dask-gateway --label org.opencontainers.image.source=https://github.com/dask-gateway
     docker build . --tag ghcr.io/dask/dask-gateway-server --label org.opencontainers.image.source=https://github.com/dask-gateway-server
     docker push ghcr.io/dask/dask-gateway
     docker push ghcr.io/dask/dask-gateway-server
     

  5. Declare permissions/access for each container package
     For each container package: a) grant permissions to github actions to push to it, b) declare that access should be inherited, c) declare that it should be public. All settings should be found in the same page.
     • Visit https://github.com/orgs/dask/packages/container/dask-gateway/settings and do the steps above.
     • Visit https://github.com/orgs/dask/packages/container/dask-gateway-server/settings and do the steps above.

[yuvipanda]

I got this today:

No idea what that actually means.

[consideRatio]

Sigh... I don't get whats going on, but maybe incorrectly triggered automation? So far, it seems we are sponsored still. The new "scout" plan seen below was something new i think.

[manics]

My feeling is we shouldn't spend too much time investigating this:

• Keep promoting quay.io as the primary source of containers
• Keep pushing to both Quay.io and Docker Hub where it's already configured and working
• Move other repos from Docker Hub to quay.io if we want
• If Docker Hub breaks at any point remove it rather than spending time trying to fix it