How to setup Dragonfly as a pull-through cache proxy for non-central-hub registries

[everpeace]

Hi, Thanks for sharing the great project.

I would like to create dragonfly cluster as a pull-through cache proxy for private registies. But I'm confused in how to configure properly

Question

I set up my experimental environment with docker-compose(https://github.com/everpeace/test-dragonfly-proxy) by reading the document, especially for two documents below:

• https://github.com/dragonflyoss/Dragonfly/blob/master/docs/user_guide/proxy.md
• https://github.com/dragonflyoss/Dragonfly/blob/master/docs/user_guide/container_runtimes/docker_proxy.md

My environment consists of:

• 1 supernode
• 2 dfdaemons (with docker daemons)
  • each dfdaemon is configured to cache quay.io image blobs
  • each docker daemon is configured to use the dfdaemon as a proxy and specify quay.io as an insecure registry for dfdaemon's hijacking.

docker pull in dfdaemon nodes looks like pulling images through dragonfly successfully. However, it does NOT seem to cache images in dragonfly. The second docker pull doesn't make it faster at all (see the next section). I'm wondering how to configure properly?? I'm very happy if I got the community's help.

How to reproduce

$ git clone https://github.com/everpeace/test-dragonfly-proxy.git
$ docker-compose up
...
# press Ctrl-C to shutdown

In another terminal, please try to pull a sample image in dfdaemon1:

$ docker-compose exec dfdaemon1 time docker pull quay.io/everpeace/alpine:latest
latest: Pulling from everpeace/alpine
188c0c94c7c5: Pull complete
Digest: sha256:d7342993700f8cd7aba8496c2d0e57be0666e80b4c441925fc6f9361fa81d10e
Status: Downloaded newer image for quay.io/everpeace/alpine:latest
quay.io/everpeace/alpine:latest
real    0m 58.95s
user    0m 0.04s
sys     0m 0.02s

you'll see below dfdaemon logs (it can see dfdaemon invokes dfget):

dfdaemon1_1  | 2020-11-18 04:42:15.992 INFO sign:17 : start download url:https://quay.io/v2/everpeace/alpine/blobs/sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0 to f8dc35f1-1726-46f8-ae54-775d7a5431d9 in repo
dfdaemon1_1  | 2020-11-18 04:42:16.001 INFO sign:17 : start download url:https://quay.io/v2/everpeace/alpine/blobs/sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 to abb094a6-f76e-4230-a51b-39e5d25eb661 in repo
dfdaemon1_1  | 2020-11-18 04:42:27.733 INFO sign:17 : dfget url:https://quay.io/v2/everpeace/alpine/blobs/sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0 [SUCCESS] cost:11.762s
dfdaemon1_1  | 2020-11-18 04:43:10.345 INFO sign:17 : dfget url:https://quay.io/v2/everpeace/alpine/blobs/sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 [SUCCESS] cost:54.386s
dfdaemon1_1  | 2020-11-18 04:43:21.467 INFO sign:17 : scan repo and clean expired files
...

It is expected that docker pull in dfdeamon2 makes it faster because dragonfly already cached. However, it takes almost the same time. And it can't appear any downloading piece dfdaemon logs described in https://github.com/dragonflyoss/Dragonfly/tree/master/docs/quick_start#step-5-validate-dragonfly .

$ docker-compose exec dfdaemon2 time docker pull quay.io/everpeace/alpine:latest
latest: Pulling from everpeace/alpine
188c0c94c7c5: Pull complete
Digest: sha256:d7342993700f8cd7aba8496c2d0e57be0666e80b4c441925fc6f9361fa81d10e
Status: Downloaded newer image for quay.io/everpeace/alpine:latest
quay.io/everpeace/alpine:latest
real    0m 58.25s
user    0m 0.03s
sys     0m 0.02s

[poblahblahblah]

We're currently rolling out our own dragonfly deployment. Our set up is slightly different as we are using an internal private registry rather than quay.io or docker.io, so YMMV with this being helpful or not. We've been meaning to update some of the docs to make some of this more clear. Let me know if this is helpful, hopefully it is!

One important thing to note about your set up is that docker will not send credential headers to a docker repo if you set it to be an insecure repo[0], so if you need to eventually pull down private images you will need to set up your own internal proxies that either inject the proper auth headers or you will need to MITM yourself. I realize it's not great to recommend that someone MITM themselves, but here we are.

Overall our dragonfly deployment is pretty straight forward. We have a central docker registry, multiple caching nginx proxies that sit in front of the registry, and in each datacenter we have 2 dragonfly supernodes and n dragonfly clients per datacenter.

The supernode configuration is very straight forward so it's not really worth linking to a config example. The only thing to note is that we bumped up the GC time for cached objects so the supernodes would hold on to the objects for a longer period of time.

The client configuration is a bit less straight forward[1], but you can see that we chose to MITM ourselves. We chose this because it would have required a massive rolling restart of our entire docker-based infrastructure, which would have meant a rolling restart of every service running in docker. In an ideal world we would just pave over the world again, but we're not there... yet.

Looking at my notes, the decided path of least resistance for having clients reach out to docker.io or quay.io through dragonfly would be to go the proxy route where we would have an internal nginx deployment that accepted requests for quay-io-internal.foo.example.com and would from there forward all requests to quay.io. You may want to look into this. You could even have your supernodes perform this functionality. Again, this was our path of least resistance.

We also bump up the GC time on the client via the expiretime flag to keep objects in the p2p network a bit longer.

Again, I hope this helps.

0 - https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry
1 - https://gist.github.com/poblahblahblah/cd26e7410d7ed4ed6ed989257142fbe7