Can I use images in training set that don't have any boxes?

[AceMcAwesome]

I'm trying to train a model to detect a single type of object, a damaged blood vessel, and I have a bunch of labeled training images with their corresponding boxes around the damaged vessels. However, I also want to include images of healthy vessels, with no bounding boxes, so that the model doesn't simply become a general vessel detector. I included these images along with label data in json format that just has an empty dictionary where the box coordinates would normally be, but when I tried training I got a loss of infinite and it didn't train. Has anyone had success including images with no boxes as negatives in a yolo-v3 training dataset?