Very unusual loss behaviour during model training

[drsonny]

Hello! I have trained YOLOv8l on a custom dataset for detecting icebergs in sea ice on SAR images, and have used augmentation to improve the results. Despite this process, I have acquired some weird looking behaviour on the loss plots. Does anyone have any idea what this might be? My first instinct was some double descent behaviour on the validation losses. The model is trained for 50 epochs at batch size 32 and all other default parameters.

[UltralyticsAssistant]

👋 Hello @drsonny, thank you for bringing this to our attention and for sharing your training results! 🚀 Your use case of detecting icebergs in SAR images sounds fascinating 🌊❄️.

We recommend checking out our comprehensive Docs, especially the sections on model training tips and loss function details, which may help you better understand and diagnose loss behaviors like the one you're experiencing.

To assist us in identifying the cause of the unusual loss patterns, could you provide a minimum reproducible example (MRE)? This would help us debug the issue more effectively. For example:

• The exact code snippet or training configuration used.
• Details about your dataset, such as size, class distribution, or any preprocessing steps.
• Additional graphs or insights from training logs if available.

If the behavior is related to overfitting or regularization, you might also want to explore augmentations, learning rate scheduling, or monitoring your validation dataset's consistency. There are some helpful tips in the Training Results Guide.

Additionally, make sure you're using the latest version of the ultralytics package and have dependencies up-to-date. You can upgrade using:

pip install -U ultralytics

Collaborative Support

For real-time discussion and community insights, feel free to join our Discord 🎧. Alternatively, check out our Discourse forum or engage in discussions on our Subreddit. These platforms are great for sharing your unique scenarios and learning from others' experiences.

Verified Environments

While debugging, consider trying your training in one of our verified environments:

• Notebooks with free GPUs:
  
  
  

• Other options include Google Cloud, AWS, or using a Docker Image (Docker Guide) .

Status

Lastly, ensure the Ultralytics CI tests are passing on your framework version:
.

Thank you for your patience while we look into this! This is an automated response, but an Ultralytics engineer will review your discussion and respond soon 👨‍💻😊.

[glenn-jocher]

@drsonny unusual loss behavior can result from several factors, including dataset quality, augmentation settings, and learning rate schedules. I recommend ensuring your dataset is well-labeled, verifying that augmentations are appropriate for SAR images, and experimenting with learning rate adjustments or different optimizers. If the issue persists, providing details like the data.yaml file or specific loss spikes could help diagnose further.

[drsonny]

I have definitely labelled the dataset well (I have only labelled icebergs that appear in SAR images and the corresponding optical ones), and the augmentations I performed were Horizontal Flip, Vertical Flip, and Horizontal-Vertical Flip.
I haven't been able to change the learning rate (lr0) because when I tried that it just defaulted back, the optimiser picked could only give me its default learning rate

[glenn-jocher]

@drsonny thank you for the clarification. If the optimizer resets the learning rate, you can explicitly set lr0 in the training command or configuration file to override defaults. Additionally, consider using a learning rate scheduler or testing with a different optimizer to observe the impact on training stability. For more details on hyperparameter tuning, refer to this guide.

[drsonny]

Hi Glenn, is there an example code that shows me how to do this? I believe from my intuition that YOLO v8 uses an AdamW optimizer as default because it is better suited to smaller datasets. But I would like to fiddle with the learning rate lr0 in the training command to override the defaults.

[glenn-jocher]

@drsonny to explicitly set the learning rate (lr0) in YOLOv8, you can pass it directly in your training command like this:

from ultralytics import YOLO
model = YOLO('yolov8n.yaml').train(data='your_data.yaml', lr0=0.01, epochs=100)

The optimizer (AdamW/SGD) is auto-selected based on iterations, but you can force it with optimizer='AdamW'. For more details, see the optimizer selection logic in our Trainer docs.

[Y-T-G]

What's the size of your validation set?

[drsonny]

Hi. I have a 70-20-10 split for training, validation and testing.

There are 376 images and labels for training
There are 108 images and labels for validation
There are 56 images and labels for testing.

These are actually image slices of a much larger SAR image. It is difficult to acquire other SAR images of the study location. So I had to augment the data. It is the icebergs inside each image that I am training the model on, using just one class.

[glenn-jocher]

@drsonny thank you for the clarification. A validation set of 108 images could result in noisy validation loss curves, especially for complex tasks or small datasets. This variability may be amplified by data augmentation if the augmented training set significantly differs from the validation set. Consider experimenting with k-fold cross-validation or increasing the validation set size if possible to achieve more stable metrics. Let us know if you need further assistance!

[drsonny]

Thanks Glenn. I have never tried k-fold cross validation as it doesn't form part of our research proposal/scope, is there a way to implement this using YOLO v8?
We still have quite a high accuracy for the model (0.789) for the data, but we could consider further methods for future work or upgrades should there be a need. Still interested to know either way!

[glenn-jocher]

You're welcome! While YOLOv8 does not have built-in support for k-fold cross-validation, you can implement it manually by splitting your dataset into k folds and training/validating the model on different folds iteratively. This requires scripting outside the YOLOv8 framework. Let us know if you have further questions!