d_neural_network derivative is off

[DOCgould]

def d_neural_network_dx(W, x, k=1):
return np.dot(np.dot(W[1].T, W[0].T**k), sigmoid_grad(x))

I believe that the derivative ought to be
return np.dot(np.multiply(W[1], W[0].T), sigmoid_grad(np.dot(x, W[0]) )
to remain consistent with chain rule,

Though, I am intrigued that you were able to get the same amount of convergence in spite of this.

[tranvohuy]

I have the same question.

[tranvohuy]

I've replaced d_neural_network_dx(W,x) by different things:

def d_neural_network_dx_prodrul(W,x):
 #using product rule
 return np.dot(np.multiply(W[0][0], sigmoid_grad(np.dot(x, W[0][0]))), W[1])

def d_neural_network_dx_ep(W, x):
  #x is scalar
  #W is np.array
  
  return (neural_network(W, x + epsilon) - neural_network(W, x - epsilon))/2/epsilon

def neural_network_a(x):
 return neural_network(W, x)
def d_neural_network_a(x):
 return grad(neural_network_a)(x)

and replace x_space to x_space = np.linspace(0, 2, nx) (interval [0,2] instead of [0,1])

Then compare results. The graph shows that the "wrong derivative" is really wrong. So the author got luck on the interval [0,1].