Evaluation, Reproducibility, Benchmarks Meeting 33

Minutes of Meeting 33

Date: 26th March, 2025

Present

The MICCAI deadline is coming up, so we might have fewer today.

• Carole
• Annika
• Anne
• Olivier
• Nick

Website Feedback

• Sat together to work through changes on the Google Document
• Some discussion about the word "recommendations" but eventually we decided to keep it
• We should make sure to look at and update this website periodically
• Send last call for website feedback after the meeting today -- ask if people want to join the next iteration of BIAS
• Send our final draft to Michal at end of this week or beginning of next

Benchmarking Task Force Ideas

• Nick to send LH challenge criteria related to datasets to Carole
• Restrict this scope to human data only? No, we want to support all types of biomedical data

Confidence Interval Project

• Given MICCAI submission, how do we want to extend this?
• Long term, we should think about consensus with a larger group
• Shorter term, it would be good to write up a more extensive version of our results in a journal
• Would be nice to extend to something else besides semantic segmentation -- how about classification?
• Could be a bit technical for many journals
• Hierarchical topics should be left out of this paper, to use for another paper in the future
• So, three papers
  • Extension of MICCAI submission
  • One that explores hierarchical topics
  • A consensus paper engaging a larger group
• Is there enough out there in instance segmentation/object detection?
  • There isn't much. Maybe leave out for now