Can too many connection for the symbol make unintended computation?

[gkyh0805]

when too many connection for the single symbol occur, can the some grads for relu_new_1 be skipped due to the MXNET_EXEC_INPLACE_GRAD_SUM_CAP?

I saw "graph_executor.cc" and there is a comment
[
// Note: For a symbol like the following:
// data = mx.sym.Variable('data')
// sym = data + data + data + data + data + data + data
// the node entries v passed in here are of the same node of
// op _identity_with_attr_like_rhs. We should skip adding a node
// to its own control_deps.
]

I cannot understand the command perfectly, however, relu_new_1 in the following code is duplicated a lots and may have a dependency each other.

Then, can the gradient be different according to MXNET_EXEC_INPLACE_GRAD_SUM_CAP?
(the small connection accept all duplicate gradient and the many connection choose only one gradient among them?)

In my case, the usage of the identity symbol for reducing the direct connection show the quite different training loss even though the seed is fixed.

How does this code execute in mxnet through the graph executor?

conv_new_1 = mx.sym.Convolution(data=relu1, kernel=(1, 1), num_filter=1024, name="conv_new_1", lr_mult=3.0)
relu_new_1 = mx.sym.Activation(data=conv_new_1, act_type='relu', name='relu_new_1')

sym1 = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, name="sym1")
sym2 = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, name="sym2")
...
symn = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, name="symn")

syma1 = relu_new_1 + sym1
syma2 = relu_new_1 + sym2
...
syman = relu_new_1 + symn

final_out = syma1 + syma2 + ... + syman

[rajanksin]

@eric-haibin-lin : Please label: Question, Operator