The repository includes a Docker image for training on an NVIDIA server. It downloads the dataset from Hugging Face and writes checkpoints to a host directory, so neither data nor training state is stored inside the container.
The remote machine needs:
- Docker;
- an NVIDIA GPU and driver compatible with the CUDA version bundled by the locked PyTorch build;
- the NVIDIA Container Toolkit, which provides
docker run --gpus.
The current PyTorch build uses CUDA 13 and requires NVIDIA driver 580 or newer. Check the host with:
nvidia-smi
docker run --rm --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smiFrom the local repository, replace user@host with the server login:
rsync -av --relative \
pyproject.toml uv.lock README.md Dockerfile .dockerignore src \
user@host:pixl/
ssh user@hostOn the server:
cd pixl
docker build -t pixl .The image contains the locked runtime dependencies and the installed pixl package. It does not contain datasets or checkpoints.
First verify the image with the short training run:
docker run --gpus all --rm \
-v pixl-hf:/root/.cache/huggingface \
pixl --quickStart the full run in the background:
mkdir -p checkpoints
docker run --gpus all --rm -d \
--name pixl-train \
-v "$PWD/checkpoints:/app/checkpoints" \
-v pixl-hf:/root/.cache/huggingface \
pixlArguments after the image name are passed to python -m pixl.train. For example, replace the final pixl with pixl --run experiment-2 --epochs 2 --batch 64.
The bind mount keeps checkpoints under pixl/checkpoints/. The pixl-hf volume keeps the downloaded and encoded dataset between containers.
Follow the training log:
docker logs -f pixl-trainCtrl-C stops following the log; it does not stop training. Stop the run with:
docker stop pixl-trainBecause the container uses --rm, it is removed after stopping. Running the same command again resumes from checkpoints/<run>/checkpoint.pt. Use a new --run name when changing training settings.
Copy checkpoints back from the local machine:
rsync -a user@host:pixl/checkpoints/ checkpoints/From the local machine, stop the run, copy the changed files and rebuild:
ssh user@host 'docker stop pixl-train'
rsync -av --relative \
pyproject.toml uv.lock README.md Dockerfile .dockerignore src \
user@host:pixl/
ssh user@host 'cd pixl && docker build -t pixl .'Restart with the previous training command to resume. A code change that alters the model configuration, data fingerprints or saved settings requires a new run.
Run Docker with sudo, or add the remote user to the docker group.
Verify the NVIDIA driver and Container Toolkit installation with the CUDA test command under Requirements. This failure occurs before Pixl starts.
The dataset is being converted to palette tokens and stored in the pixl-hf volume. This is CPU- and memory-intensive on the first run only. Monitor it with:
docker stats pixl-trainConfirm that the training command mounts the host checkpoint directory:
-v "$PWD/checkpoints:/app/checkpoints"
Without that mount, --rm deletes the container together with its checkpoints.