dask-gateway image far behind mainline dask version.

[thomafred]

What happened:

dask-gateway docker image includes dask 2.30.0, while the latest dask-version is 2021.6.2.

What you expected to happen:

dask-gateway docker image should include dash version 2021.6.0 or greater

Minimal Complete Verifiable Example:

Simply create new daskhub-deployment based on daskhub helm-chart, then spawn a new cluster:

### [1]
from dask_gateway import GatewayCluster
import sys
import dask, dask_gateway

from IPython.display import display

display(sys.version_info)
display(dask.__version__)
display("Dask GW version: {}".format(dask_gateway.__version__))

### [2]
# Expect this line to fail
cluster = GatewayCluster()
cluster

### [3]
client = cluster.get_client()
client

Anything else we need to know?:

This issue causes helm-chart daskhub to to fail when spawning new clusters.

Environment:

• Dask version: 2021.6.0
• Python version: 3.8.10
• Operating System: Linux
• Install method (conda, pip, source): helm

[TomAugspurger]

Simply create new daskhub-deployment based on daskhub helm-chart,

Are you matching your worker environment to your singleuser pod, as documented at https://github.com/dask/helm-chart/tree/main/daskhub#matching-the-user-environment? That's usually the safest way to ensure you don't have mismatches between your singleuser and worker pods.

[thomafred]

Followed the docs your refered to. The scheduler pod seems to throw an exception at startup:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/bin/dask-scheduler", line 11, in <module>
    sys.exit(go())
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/cli/dask_scheduler.py", line 226, in go
    main()
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/preloading.py", line 39, in validate_preload_argv
    for name in ctx.params.get("preload")
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/preloading.py", line 40, in <dictcomp>
    if not is_webaddress(name)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/preloading.py", line 108, in _import_module
    import_module(name)
  File "/srv/conda/envs/notebook/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dask_gateway.scheduler_preload'

dask_gateway.scheduler_preload seem to be missing.

Changing the option handler from pangeo/base-notebook:2020.07.28 to pangeo/base-notebook:latest solved the problem:

# config.yaml
jupyterhub:
 singleuser:
   extraEnv:
     DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE: '{JUPYTER_IMAGE_SPEC}'

dask-gateway:
  gateway:
    extraConfig:
      optionHandler: |
        from dask_gateway_server.options import Options, Integer, Float, String
        def option_handler(options):
            if ":" not in options.image:
                raise ValueError("When specifying an image you must also provide a tag")
            return {
                "image": options.image,
            }
        c.Backend.cluster_options = Options(
            String("image", default="pangeo/base-notebook:latest", label="Image"),
            handler=option_handler,
        )

However, things need not be this complicated. This also works:

dask-gateway:
  gateway:
    backend:
      image:
        name: pangeo/base-notebook
        tag: "latest"

[jacobtomlinson]

xref dask/dask-docker#144

It would be good to build an image as part of the Dask org which works with Dask gateway instead of using the one from the Pangeo community.

[aktech]

I believe this was fixed in #413

[martindurant]

The dockerfile was updated, but is there a new image? (I don't know if CI builds this dockerfile, or still pulls the latest from dockerhub)

[aktech]

Ah, yes that's true. Seems like the last image was created about 9 months ago probably @jcrist did it manually. We can write a GitHub Action to build and push image to dockerhub on every merge to master?

[martindurant]

That seems reasonable; although that could be a lot of images, so maybe should be controlled by the commit message, or only for tags? @jacobtomlinson , you must have done this sort of thing.

[jacobtomlinson]

There are two images right?

The dask-gateway image which contains the server and is used for launching the gateway processes on Kubernetes. This is the image we are discussing here. I'm not sure that having an out of date Dask version here really matters, but it would be good to keep this image up to date.

Then there are the user environment containers that are configurable, and the default is to use the Pangeo image.

For the dask-gateway image we could do something like dask/dask-docker#149 which pushes new images with GitHub actions. My preference here would be to only push on tags. We could also consider moving the image to the daskdev dockerhub org, having two orgs is a little confusing.

We should also publish an image for the user environment, this is covered by dask/dask-docker#144.

[consideRatio]

I'll go ahead and close this as resolved as we are now at 2021.7.1 and if #423 is merged even newer. We still need a release though, but that is represented by #381