makemore/lecture4/backprop/dC

[we89]

dC = torch.zeros_like(C)
for k in range(Xb.shape[0]):
    dC[Xb[k]] += demb[k]

I don't know why it is `wrong.
And the correct answer is:

dC = torch.zeros_like(C)
for k in range(Xb.shape[0]):
    for j in range(Xb.shape[1]):
        ix = Xb[k,j]
        dC[ix] += demb[k,j]

[Mountagha]

What is your question. Can you elaborate more clearly ?

[junqi-lu]

It's an issue about how PyTorch (or numpy?) manages in-place operations and advanced indexing with repeating indices, it seems PyTorch only adds the last corresponding row from src to the specified index.

So if Xb[0]=[1,1,1], dC[1] will add the last row of demb[0], which is demb[0][2], not to accumulate demb[0][0], demb[0][1] and demb[0][2].

You can find more in this.