Repository navigation
[BraTS 2021/PyTorch] Model not properly training #1304
Description
When running the BraTS 2021 notebook (located at PyTorch/Segmentation/nnUNet/notebooks/BraTS21.ipynb) training section, the model is not properly training even though it is going through the steps, as seen in the image below. The Dice is stuck at an extremely low value and neither that nor the loss changes at all over the epochs. The "DALI iterator does not support resetting while epoch is not finished" warning comes up on every epoch but that is not something that I have touched.
To Reproduce
Steps to reproduce the behavior:
- Clone the DeepLearningExamples repo and Install the dependencies
- Download the BraTS 2021 dataset
- Change paths in the BraTS 2021 notebook to point to file locations
- Run all of the steps up to and including the training stage
Expected behavior
I expected the model to train and have at least a Dice of 70 after 5 epochs
Environment
Please provide at least:
- PyTorch version: 1.13.1+cu116
- GPUs in the system: 2x Tesla V100-SXM2-16GB:
- CUDA driver version 515.86.01:
Activity
michal2409 commentedon Jun 20, 2023
Are you running notebook inside docker container? It looks like a dependency issue (running notebook with different versions of dependency). Please see https://ploomber.io/blog/notebook-to-docker/ for reference how to run Jupyter Notebook inside container.
DanielNajarian commentedon Jun 20, 2023
I'm running it through command line and built the environment based on their requirements files.
michal2409 commentedon Jun 20, 2023
What versions for PyTorch and NVIDIA DALI are you using?
DanielNajarian commentedon Jun 20, 2023
I am using torch 1.13.1+cu116 and nvidia-dali-cuda110 1.26.0. Looking at it now, DALI should be cuda116, correct? But there doesnt seem to be a cuda116 version of it.
michal2409 commentedon Jun 20, 2023
22.11 container has 1.18.0 DALI version (see here). Were you manually reinstalling it?
DanielNajarian commentedon Jun 20, 2023
I had to manually reinstall a few packages since the torch and torchvision CUDA versions weren't lined up and I had trouble getting 117 to work on both, so I went down to 116 and changed some stuff as a result.
Should I be focusing on 22.02 container since it lines up with CUDA 11.6, which is my torch version? This would be DALI 1.10.
michal2409 commentedon Jun 21, 2023
You can experiment with different versions. I would start with DALI 1.18.0 (or not reinstalling it inside container).
What error log you had during running container without any modification?
Luffy03 commentedon Oct 17, 2023
Hi, have you figured it out?