When using the imglist file to load the data,If the number of data set label isn't the integer multiple batch size,the loss become nan

[canteen-man]

Description

I regression the label and the label width is just a float number. I use the imglist to load my image and label.When the total number of label aren't the integer multiple of the batch size ，the label become the strange value,I don’t sure if it is a random number.I print the label in the data iter,some of them are so big，others are so small,I‘m sure these number are't my label in lst file.And due to these number,the loss become nan.

I don't shuffle the label and the image.I find when it iter to the last of the data set, at the first part of the last batch in this epoch are the correct number，and the second half label should be the head of the label of all data set,but they are become the strange value.

When I set the batch size which the total number of label are the integer multiple of the batch size ，the train loss iteration turn normal.

I research the code，maybe the question is in the detection.py，the next_sample function and the next function are about load the imglist.

It seem like declare a batch label which size is same as the batch size length.Then go through the lable of imglist file to assignment the batch label,erery step is the batch size length.So the last of label don’t get correct number.

I don't sure whether this question is because of this，this mistake is too simple to feel possible.But this question real occur.

I have already slove this problem by set the number of the batch size which the total number of data set label is the integer multiple of batch size.

Environment info (Required)

mxnet 1.2.1
CUDA 9.1
UBUNTU

Package used (Python/R/Scala/Julia):
python 3.7

[lanking520]

@canteen-man Could you please provide some code that we can reproduce your problem?

[canteen-man]

@lanking520 ,the lst file is such as :
0 0.2083763 picture_root/1.png
1 0.1911981 picture_root/2.png
2 0.3475569 picture_root/3.png
And I load the image by use
training_iter=mx.image.ImageIter(batch_size =batch_size, data_shape= (3,256,256), path_root=
xxx,path_imglist= xxx)

mod=mx.mod.Module(symbol= xxx,context = xxx,data_names=['xxx'],label_names=['xxx'])

mod.bind(data_shapes=training_iter.provide_data,label_shapes=training_iter.provide_label)

mod.init_params(mx.initializer.Xavier())

lr_sch = mx.lr_scheduler.FactorScheduler(step=2000, factor=0.5)

mod.fit(train_data=training_iter,optimizer='sgd',optimizer_params=(('learning_rate', 0.1), ('lr_scheduler', lr_sch)),eval_metric='mse',num_epoch=500,epoch_end_callback=checkpoint)

And I print the label in the fit function of the base_module.py by use:
print("label",data_batch.label)
I add this print function in the "while not end_of_batch" loop.
And the print label are't normal when the last batch in a epoch if the number of data set label isn't the integer multiple of batch size.
Like the number of date set label is 10,and the batch size is 6,and the loss is nan.
And if the number of date set is 10 ,and the batch size is 5 ,the print label is normal again.

###########################################################################
不行，我受不了了，我打中文了，你也是中国人吧。
就是用这个imglst载入数据的时候我发现这数据集总数不是batch size的整数倍时，1个epoch中前边的batch size都正常，然后1个epoch结束loss就变成nan了。就正好一个epoch结束变成的nan。然后我就打印的源码中的变量，发现最后一个epoch中最后一个batch打印出的label是奇怪的数，肯定不是我lst文件中的数，而且就在最后一个batch出现。然后我没有打乱数据集排序，就正常顺序的读，发现1个epoch中最后一个batch的前几个label都是正常的，最后一个batch中的label不正常就是从数据集最后一个label结束后开始的。
按理来说最后一个label载入后不应该是从头继续载入吗，直到填充满最后这个batch size，但这里最后却是很奇怪的数，要不就很大，e的10几，要不就很小，e的负10几。
在源码这块我只改了多少个epoch存储一次模型和打印一些变量，没有改什么重要的源码，用的训练代码就上边这些。