Small possible bug

[hsilva664]

In the evaluate() function, on agents/base.py, after line 171:

_, pred_label = dists.min(1)

Shouldn't there be a:

pred_label = torch.tensor(self.old_labels, dtype=batch_y.dtype, device=batch_y.device)[pred_label]

Right after 171? Otherwise the error_analysis part of this function, which uses pred_label to see which are the examples assigned to old classes vs new classes, will use a pred_label value that will make no sense in that case.

Then, to make the rest of the code consistent after my suggested addition, lines 175, 176:
correct_cnt = (np.array(self.old_labels)[ pred_label.tolist()] == batch_y.cpu().numpy()).sum().item() / batch_y.size(0)

Would have to be changed to:

correct_cnt = (pred_label == batch_y).sum().item() / batch_y.size(0)

[Hiroid]

Hi there, I found the same bug when running one class per task using iCaRL for example. The bug here seems to happen because of the function .squeeze(), line 170, which change the dists.shape from (batch_size, n_classes) to (batch_size). Just keep the dists a 2-D tensor is OK !

Try the follow code again ! :)

# _, pred_label = dists.min(1) # "IndexError: Dimension out of range" when n_classes is 1
_, pred_label = dists.min(1) if len(dists.shape) == 2 else dists.unsqueeze(1).min(1)

[hsilva664]

No, this is not the same bug. I will give an example.

Suppose self.old_classes = [1,2,3,26,24,22], meaning these were the classes seen before the evaluate() call. When you use ncm_trick (or any nearest mean based classifier, such as iCaRL), you have a mean tensor containing the feature means of each class in self.old_classes (mean in this case is of shape (6, n_features)). These means are used to calculate the distances of the test example features to each class mean, which is given in the tensor dists, with shape (batch_size, 6).

Then, what I tried to explain in the other message will happen: _, pred_label = dists.min(1) will return a pred_label array of (batch_size), shape, but with each entry in range(6), instead of in [1,2,3,26,24,22]. The pred_label has to be compared against batch_y, and this last one's entries are in [1,2,3,26,24,22]. Therefore, in line 175, this conversion of sets happen, so that the comparison can happen correctly.

What I'm pointing out is that this conversion does not change the variable pred_label, which is still in range(6). Furthermore, in the error_analysis part of the code (line 182 of the same file), which is only run if you pass error_analysis = True in the args, this same pred_label is used and compared against batch_y one more time, but now without conversion, despite the entries being in different "ranges". This comparison is used to get information such as "how many old classes were classified as new classes" (section 8.3 from their paper). In this case, there will be inconsistent error_analysis results