Debugging Dagster inside Docker container

[C-Loftus]

Background

I did not see this anywhere so I wanted to write this up. My team is using dagster inside of docker swarm. We could not use dagster dev locally since we needed to be inside the container to get all our proper dependencies.

If you have any insight or feedback, feel free to let me know.

All code is found at: https://github.com/internetofwater/scheduler If you like this, please star our project

Setup

In order to get this working you need to

• install debugpy so you can expose a python module for remote debugging
• launch dagster dev from inside the container wrapping it with debugpy
  • You will need to have a custom entrypoint where it launches dagster with something like this

Example Dockerfile and entrypoint

Dockerfile

FROM python:3.10-slim

COPY Docker/user_code_requirements.txt user_code_requirements.txt
RUN pip install -r user_code_requirements.txt

ARG DAGSTER_DEBUG
RUN if [ "$DAGSTER_DEBUG" = "true" ]; then pip install debugpy dagster-webserver; fi
RUN echo "User code 'DAGSTER_DEBUG' value is: $DAGSTER_DEBUG"

# Pass the build argument as an environment variable
ENV DAGSTER_DEBUG=$DAGSTER_DEBUG

# configs and runtime code
WORKDIR /opt/dagster/app
COPY code /opt/dagster/app/code
COPY templates /opt/dagster/app/templates

# Expose the necessary ports
EXPOSE 4000
EXPOSE 5678

ENV DAGSTER_HOME=/opt/dagster/app

COPY Docker/entrypoint.sh /opt/dagster/app/entrypoint.sh
ENTRYPOINT ["/opt/dagster/app/entrypoint.sh"]

Entrypoint

#!/bin/sh
set -x
if [ "$DAGSTER_DEBUG" != "true" ] && [ "$DAGSTER_DEBUG" != "false" ]; then
  echo "DAGSTER_DEBUG env var must be set to one of 'true' or 'false'"
  exit 1
fi

if [ "$DAGSTER_DEBUG" = "true" ]; then
  exec python -m debugpy --configure-subProcess true --wait-for-client --listen 0.0.0.0:5678 -m dagster dev -h 0.0.0.0 -p 5482 --python-file /opt/dagster/app/code/main.py -d /opt/dagster/app/code
else
  exec dagster code-server start -h 0.0.0.0 -p 4000 --python-file /opt/dagster/app/code/main.py -d /opt/dagster/app/code
fi

• Forward the port used for debugpy to your local host via docker compose or another method
  • in the example above I mapped "5482:5482"
• Make sure all your other services are not being spun up / interfering with dagster dev
  • You pretty much have to use dagster dev to be able to debug. If you separate the services with grpc, debugpy will only work with the one you wrapped it with, not all of them at once (webserver & daemon) like if you use dagster dev.
• Create some sort of config to conditionally spin up dagster dev vs your typically multicontainer deployment
  • docker compose profiles are a good option for this but they don't work in swarm, only compose, so you might need to use separate compose yaml files and then just use multiple -f specifiers
• Create an associated launch.json for vscode that connects to the local debugpy port. (I'm sure an alternative exists in pycharm, I just don't use it so I am not sure there)

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Dagster inside Docker Container",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/code",
                    "remoteRoot": "/opt/dagster/app/code"
                }
            ],
            "justMyCode": true,
            "subProcess": true,
            "preLaunchTask": "Run Dagster in Debug Mode"
        }
    ]
}