How to Handle Corrupted Images

[samadejacobs]

ImageNet21k (14 million image dataset) has a number of "corrupted" images, LBANN need to provide a way to handle (possibly skip) corrupted images. Thoughts?

[ndryden]

How are they corrupted? Is it in the sense that the images do not load, or that they load fine but are otherwise corrupted (image artifacts, etc.)?

Incidentally, I thought ImageNet was supposed to be pretty clean, since everything is human-annotated-- is this a broadly recognized issue or maybe it's just due to our download?

[samadejacobs]

The images do not load i.e.,
LBANN: caught error message: /usr/workspace/wsa/jacobs32/lbann.git/src/data_readers/data_reader_imagenet.cpp 62ImageNet: image_utils::loadJPG failed to load - /p/lscratche/brainusr/datasets/ImageNetALL_extracted/n04257684/n04257684_9033.JPEG

Manually opening the image file (gnome-open) also failed. This happened several hours into program execution which could mean that there are probably more good than bad images.