Saved model and then looked at the output of the same batch after loading, got different results!

[KhurramPirov]

Description

(Brief description of the problem in no more than 2 sentences.)
Get different results from model which is saved and from loaded model.
I tried differen methods for saving and loading, all showed wrong output.

           model.forward(batch)
           out1 = model.get_outputs()[0].asnumpy()
           print('saving')
           arg, aux = model.get_params()
           all_layers = model.symbol.get_internals()
          _sym = all_layers['fc1_output']
          _arg = {}
          for k in arg:
              _arg[k] = arg[k]
          model.save_checkpoint(prefix, 1, save_optimizer_states=True)
          sym, arg_params, aux_params = mx.model.load_checkpoint(os.path.join(args.pretrained, "model"), args.pretrained_epoch)
          model.set_params(arg_params, aux_params)
          model.forward(batch)
          arg_ckpt,_ = model.get_params()
          out2 = model.get_outputs()[0].asnumpy()
          print(f"diff_output:{np.abs(out1-out2)}") 

[frankfliu]

@mxnet-label-bot add [question, training]

[frankfliu]

@KhurramPirov
It the code just pseudo code or it's complete code?

It looks you like you are running all batched data from data_iter for the initial forward,
But you only forward once for the last batch after you saved. If you are not processing the same input, the result will not be the same.

[KhurramPirov]

@frankfliu
It is a sample of the test code. I am checking for each batch once and then look at the output, it is all in one loop with keyboard interruption. Not the last batch, sorry did not insert interruption and right indents. As input I took the same batch, and also I checked the difference of weights and it was zero. The difference between output embeddings is not negligible, for each row at the scale of one tenth.

[braindotai]

Make sure you are saving and loading the model from the same path. I've tested the code below and it is working fine.

for batch in data_iter:
    model.forward(batch)
    out1 = model.get_outputs()[0]
    break
model.save_checkpoint('model', epoch = 0)
sym, arg_params, aux_params = mx.model.load_checkpoint('model', epoch = 0)
model.set_params(arg_params, aux_params)
model.forward(batch)
out2 = model.get_outputs()[0]
print(f'diff_output:{nd.abs(out1 - out2)}') 

Output is all zeros.

[KhurramPirov]

I checked the path, I do it inside batch_callback and model.fit, can it be the case(seems no), but nevetheless. пт, 12 июл. 2019 г. в 13:03, Rishik Mourya <[email protected]>:

…

Make sure you are saving and loading the model from the same path. I've tested the code below and it is working fine. for batch in data_iter: model.forward(batch) out1 = model.get_outputs()[0] break model.save_checkpoint('model', epoch = 0) sym, arg_params, aux_params = mx.model.load_checkpoint('model', epoch = 0) model.set_params(arg_params, aux_params) model.forward(batch) out2 = model.get_outputs()[0]print(f'diff_output:{nd.abs(out1 - out2)}') Output is all zeros. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-mxnet/issues/15504?email_source=notifications&email_token=AH2IALW3CR6MZP532CKQ5ALP7BJFXA5CNFSM4H7SKOHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZZKIOA#issuecomment-510829624>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH2IALTCG3VBAW5DDH2PFTLP7BJFXANCNFSM4H7SKOHA> .

-- Khurram Pirov skype: matrixrextended mob: +7-916-439-53-76

[KhurramPirov]

It doesn't work inside batch_callback. пт, 12 июл. 2019 г. в 13:03, Rishik Mourya <[email protected]>:

…

Make sure you are saving and loading the model from the same path. I've tested the code below and it is working fine. for batch in data_iter: model.forward(batch) out1 = model.get_outputs()[0] break model.save_checkpoint('model', epoch = 0) sym, arg_params, aux_params = mx.model.load_checkpoint('model', epoch = 0) model.set_params(arg_params, aux_params) model.forward(batch) out2 = model.get_outputs()[0]print(f'diff_output:{nd.abs(out1 - out2)}') Output is all zeros. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-mxnet/issues/15504?email_source=notifications&email_token=AH2IALW3CR6MZP532CKQ5ALP7BJFXA5CNFSM4H7SKOHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZZKIOA#issuecomment-510829624>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH2IALTCG3VBAW5DDH2PFTLP7BJFXANCNFSM4H7SKOHA> .

-- Khurram Pirov skype: matrixrextended mob: +7-916-439-53-76