Dynamic hops

[zh4ngx]

No description provided.

[zh4ngx]

Sample run with 3 num_hops, otherwise default hyperparams.
LR Anneal default seems wrong (swapped?)

Haven't inspected weights for 'stopping' unit, but doesn't seem to be training it (so basically randomly stopping with 0.5 prob.

Significantly worse than default on full 3 hops

/Users/andy/anaconda/bin/python /Users/andy/code/dynamic-memory-networks/dmn_train.py --babi_task_id 2
Training DMN plus on babi task 2
==> Loading test from /Users/andy/code/dynamic-memory-networks/data/en-10k/qa2_two-supporting-facts_train.txt
==> Loading test from /Users/andy/code/dynamic-memory-networks/data/en-10k/qa2_two-supporting-facts_test.txt
==> get train inputs
==> get test inputs
==> get question representation
==> get input representation
==> build episodic memory
==> generating episode
==> testing loop condition for whether softmax is greater than 0
==> generating episode
Starting run 0
==> initializing variables
==> starting training
Epoch 0
Training loss: 199.04620361328125
Validation loss: 167.33151245117188
Training accuracy: 0.1991111111111112
Vaildation accuracy: 0.34299999999999997
Saving weights
Total time: 99.06030297279358
Epoch 1
Training loss: 155.4452667236328
Validation loss: 136.3482208251953
Training accuracy: 0.40000000000000013
Vaildation accuracy: 0.48600000000000004
Saving weights
Total time: 87.96639800071716
Epoch 2
Training loss: 125.5321044921875
Validation loss: 103.54203033447266
Training accuracy: 0.5333333333333334
Vaildation accuracy: 0.6169999999999999
Saving weights
Total time: 96.0941698551178
Epoch 3
Training loss: 92.33892059326172
Validation loss: 72.00099182128906
Training accuracy: 0.6906666666666668
Vaildation accuracy: 0.7870000000000001
Saving weights
Total time: 87.98938417434692
Epoch 4
Training loss: 65.75812530517578
Validation loss: 56.747642517089844
Training accuracy: 0.7984444444444446
Vaildation accuracy: 0.834
Saving weights
Total time: 85.12621593475342
Epoch 5
Training loss: 50.40780258178711
Validation loss: 48.174991607666016
Training accuracy: 0.8527777777777774
Vaildation accuracy: 0.8640000000000001
Saving weights
Total time: 90.51080679893494
Epoch 6
Training loss: 40.055259704589844
Validation loss: 42.80059051513672
Training accuracy: 0.8873333333333336
Vaildation accuracy: 0.881
Saving weights
Total time: 89.36993503570557
Epoch 7
Training loss: 33.46493148803711
Validation loss: 39.412899017333984
Training accuracy: 0.9061111111111112
Vaildation accuracy: 0.901
Saving weights
Total time: 96.95701909065247
Epoch 8
Training loss: 28.340728759765625
Validation loss: 31.627817153930664
Training accuracy: 0.9220000000000004
Vaildation accuracy: 0.917
Saving weights
Total time: 88.39212989807129
Epoch 9
Training loss: 23.67264747619629
Validation loss: 31.556476593017578
Training accuracy: 0.935
Vaildation accuracy: 0.9199999999999999
Saving weights
Total time: 102.27093815803528
Epoch 10
Training loss: 19.92367172241211
Validation loss: 29.430988311767578
Training accuracy: 0.9448888888888883
Vaildation accuracy: 0.9299999999999999
Saving weights
Total time: 99.13410902023315
Epoch 11
Training loss: 17.867671966552734
Validation loss: 26.295700073242188
Training accuracy: 0.9533333333333328
Vaildation accuracy: 0.938
Saving weights
Total time: 95.85293078422546
Epoch 12
Training loss: 15.191685676574707
Validation loss: 28.73358726501465
Training accuracy: 0.9606666666666664
Vaildation accuracy: 0.9369999999999999
Total time: 54.645631074905396
Epoch 13
Training loss: 13.2013521194458
Validation loss: 30.400623321533203
Training accuracy: 0.964111111111111
Vaildation accuracy: 0.9199999999999999
Total time: 54.92138385772705
Epoch 14
Training loss: 12.548670768737793
Validation loss: 30.075927734375
Training accuracy: 0.9673333333333326
Vaildation accuracy: 0.933
Total time: 52.489410161972046
Epoch 15
Training loss: 12.412619590759277
Validation loss: 29.89286231994629
Training accuracy: 0.9690000000000001
Vaildation accuracy: 0.9379999999999997
Total time: 54.677321910858154
Epoch 16
Training loss: 9.951630592346191
Validation loss: 26.146759033203125
Training accuracy: 0.9755555555555552
Vaildation accuracy: 0.945
Saving weights
Total time: 92.82357406616211
Epoch 17
Training loss: 9.349241256713867
Validation loss: 26.12112045288086
Training accuracy: 0.9781111111111106
Vaildation accuracy: 0.9510000000000002
Saving weights
Total time: 90.21525406837463
Epoch 18
Training loss: 8.88101577758789
Validation loss: 24.18722152709961
Training accuracy: 0.9797777777777771
Vaildation accuracy: 0.9469999999999998
Saving weights
Total time: 89.68267393112183
Epoch 19
Training loss: 8.541062355041504
Validation loss: 25.11202621459961
Training accuracy: 0.9805555555555547
Vaildation accuracy: 0.9480000000000001
Total time: 52.820664167404175
Epoch 20
Training loss: 7.961450099945068
Validation loss: 22.37687110900879
Training accuracy: 0.9825555555555555
Vaildation accuracy: 0.9509999999999998
Saving weights
Total time: 87.7144730091095
Epoch 21
Training loss: 7.394680976867676
Validation loss: 23.667743682861328
Training accuracy: 0.985555555555555
Vaildation accuracy: 0.9540000000000001
Total time: 52.080361127853394
Epoch 22
Training loss: 7.149799823760986
Validation loss: 25.538795471191406
Training accuracy: 0.9827777777777772
Vaildation accuracy: 0.9489999999999998
Total time: 51.290579080581665
Epoch 23
Training loss: 6.991062641143799
Validation loss: 22.696842193603516
Training accuracy: 0.9837777777777776
Vaildation accuracy: 0.954
Total time: 53.48832702636719
Epoch 24
Training loss: 6.49598503112793
Validation loss: 21.40328025817871
Training accuracy: 0.9872222222222219
Vaildation accuracy: 0.9570000000000001
Saving weights
Total time: 87.63241291046143
Epoch 25
Training loss: 6.50925350189209
Validation loss: 25.58407974243164
Training accuracy: 0.9844444444444442
Vaildation accuracy: 0.9559999999999998
Total time: 51.189826011657715
Epoch 26
Training loss: 7.021908760070801
Validation loss: 23.536022186279297
Training accuracy: 0.9866666666666661
Vaildation accuracy: 0.9559999999999998
Total time: 51.75995898246765
Epoch 27
Training loss: 5.791140556335449
Validation loss: 25.807369232177734
Training accuracy: 0.989333333333333
Vaildation accuracy: 0.958
Total time: 51.23908090591431
Epoch 28
Training loss: 6.0674824714660645
Validation loss: 23.508174896240234
Training accuracy: 0.9883333333333328
Vaildation accuracy: 0.952
Total time: 50.99159598350525
Epoch 29
Training loss: 6.028339862823486
Validation loss: 20.448293685913086
Training accuracy: 0.9876666666666659
Vaildation accuracy: 0.9569999999999999
Saving weights
Total time: 88.28481602668762
Epoch 30
Training loss: 4.941899299621582
Validation loss: 28.63985824584961
Training accuracy: 0.9911111111111107
Vaildation accuracy: 0.958
Total time: 51.49487900733948
Epoch 31
Training loss: 5.850871562957764
Validation loss: 26.28609848022461
Training accuracy: 0.9884444444444439
Vaildation accuracy: 0.9570000000000001
Total time: 51.92403483390808
Epoch 32
Training loss: 4.91590690612793
Validation loss: 22.008039474487305
Training accuracy: 0.9922222222222219
Vaildation accuracy: 0.96
Total time: 51.349725008010864
Epoch 33
Training loss: 5.384566307067871
Validation loss: 24.298364639282227
Training accuracy: 0.9906666666666668
Vaildation accuracy: 0.962
Total time: 51.87458086013794
Epoch 34
Training loss: 4.58192777633667
Validation loss: 22.0777645111084
Training accuracy: 0.9924444444444444
Vaildation accuracy: 0.9609999999999997
Total time: 51.32881498336792
Epoch 35
Training loss: 4.511839389801025
Validation loss: 25.890085220336914
Training accuracy: 0.9932222222222218
Vaildation accuracy: 0.96
Total time: 51.393450021743774
Epoch 36
Training loss: 4.808272838592529
Validation loss: 23.245569229125977
Training accuracy: 0.9926666666666668
Vaildation accuracy: 0.9629999999999999
Total time: 50.579740047454834
Epoch 37
Training loss: 5.48853063583374
Validation loss: 19.434913635253906
Training accuracy: 0.9907777777777773
Vaildation accuracy: 0.966
Saving weights
Total time: 83.30758714675903
Epoch 38
Training loss: 4.386177062988281
Validation loss: 25.92519187927246
Training accuracy: 0.9927777777777772
Vaildation accuracy: 0.9559999999999998
Total time: 50.20597314834595
Epoch 39
Training loss: 4.409369468688965
Validation loss: 17.584972381591797
Training accuracy: 0.9939999999999998
Vaildation accuracy: 0.9639999999999999
Saving weights
Total time: 84.56739783287048
Epoch 40
Training loss: 4.935416221618652
Validation loss: 17.36130714416504
Training accuracy: 0.9918888888888885
Vaildation accuracy: 0.9730000000000002
Saving weights
Total time: 84.48695015907288
Epoch 41
Training loss: 5.091924667358398
Validation loss: 21.63187026977539
Training accuracy: 0.9909999999999998
Vaildation accuracy: 0.9650000000000001
Total time: 50.48800206184387
Epoch 42
Training loss: 4.483287334442139
Validation loss: 19.801252365112305
Training accuracy: 0.9931111111111106
Vaildation accuracy: 0.961
Total time: 50.89587998390198
Epoch 43
Training loss: 4.2647294998168945
Validation loss: 21.87394905090332
Training accuracy: 0.993111111111111
Vaildation accuracy: 0.9669999999999999
Total time: 49.89055395126343
Epoch 44
Training loss: 4.430873394012451
Validation loss: 18.228694915771484
Training accuracy: 0.9933333333333326
Vaildation accuracy: 0.9650000000000001
Total time: 50.12391805648804
Epoch 45
Training loss: 5.236786365509033
Validation loss: 17.964778900146484
Training accuracy: 0.9911111111111104
Vaildation accuracy: 0.9709999999999999
Total time: 49.87518501281738
Epoch 46
Training loss: 4.540221691131592
Validation loss: 19.72925567626953
Training accuracy: 0.9933333333333326
Vaildation accuracy: 0.9640000000000001
Total time: 49.46612310409546
Epoch 47
Training loss: 4.074179649353027
Validation loss: 23.967390060424805
Training accuracy: 0.9937777777777776
Vaildation accuracy: 0.9640000000000001
Total time: 50.0403368473053
Epoch 48
Training loss: 4.482390880584717
Validation loss: 19.35105323791504
Training accuracy: 0.9935555555555554
Vaildation accuracy: 0.9620000000000001
Total time: 49.86299014091492
Epoch 49
Training loss: 4.692705154418945
Validation loss: 21.8217830657959
Training accuracy: 0.9925555555555555
Vaildation accuracy: 0.9629999999999999
Total time: 50.47045302391052
Epoch 50
Training loss: 4.2504191398620605
Validation loss: 20.052335739135742
Training accuracy: 0.9935555555555553
Vaildation accuracy: 0.9700000000000001
Total time: 49.93616604804993
Epoch 51
Training loss: 4.713943004608154
Validation loss: 16.04507064819336
Training accuracy: 0.9922222222222217
Vaildation accuracy: 0.968
Saving weights
Total time: 82.80339312553406
Epoch 52
Training loss: 4.126282215118408
Validation loss: 16.90555191040039
Training accuracy: 0.9945555555555556
Vaildation accuracy: 0.9629999999999999
Total time: 50.2987699508667
Epoch 53
Training loss: 4.198649883270264
Validation loss: 14.489843368530273
Training accuracy: 0.9947777777777774
Vaildation accuracy: 0.9720000000000001
Saving weights
Total time: 83.70088505744934
Epoch 54
Training loss: 3.8231048583984375
Validation loss: 13.964970588684082
Training accuracy: 0.9951111111111106
Vaildation accuracy: 0.9810000000000001
Saving weights
Total time: 84.24891090393066
Epoch 55
Training loss: 3.482072591781616
Validation loss: 18.016647338867188
Training accuracy: 0.9959999999999996
Vaildation accuracy: 0.969
Total time: 50.45484685897827
Epoch 56
Training loss: 4.163816928863525
Validation loss: 14.935145378112793
Training accuracy: 0.9938888888888886
Vaildation accuracy: 0.975
Total time: 50.0498480796814
Epoch 57
Training loss: 3.789604663848877
Validation loss: 18.94182014465332
Training accuracy: 0.995555555555555
Vaildation accuracy: 0.9739999999999999
Total time: 50.3101019859314
Epoch 58
Training loss: 5.081748962402344
Validation loss: 16.687408447265625
Training accuracy: 0.9917777777777773
Vaildation accuracy: 0.969
Total time: 50.12720203399658
Epoch 59
Training loss: 4.402705192565918
Validation loss: 14.933670043945312
Training accuracy: 0.9935555555555555
Vaildation accuracy: 0.9750000000000002
Total time: 49.79584789276123
Epoch 60
Training loss: 3.553560733795166
Validation loss: 16.63455581665039
Training accuracy: 0.9957777777777775
Vaildation accuracy: 0.976
Total time: 49.98001003265381
Epoch 61
Training loss: 3.6404049396514893
Validation loss: 15.045092582702637
Training accuracy: 0.996222222222222
Vaildation accuracy: 0.976
Total time: 49.618205070495605
Epoch 62
Training loss: 3.5773489475250244
Validation loss: 14.44677734375
Training accuracy: 0.9956666666666661
Vaildation accuracy: 0.9790000000000001
Total time: 49.708503007888794
Epoch 63
Training loss: 3.950178623199463
Validation loss: 14.634023666381836
Training accuracy: 0.9947777777777776
Vaildation accuracy: 0.977
Total time: 50.22454595565796
Epoch 64
Training loss: 3.648160219192505
Validation loss: 19.92352867126465
Training accuracy: 0.9948888888888884
Vaildation accuracy: 0.974
Total time: 50.382654905319214
Epoch 65
Training loss: 4.959195137023926
Validation loss: 15.278950691223145
Training accuracy: 0.9927777777777774
Vaildation accuracy: 0.9750000000000002
Total time: 50.48412203788757
Epoch 66
Training loss: 4.266040325164795
Validation loss: 17.983768463134766
Training accuracy: 0.9938888888888889
Vaildation accuracy: 0.9760000000000002
Total time: 50.30915594100952
Epoch 67
Training loss: 3.8262503147125244
Validation loss: 14.038446426391602
Training accuracy: 0.9955555555555553
Vaildation accuracy: 0.9789999999999999
Total time: 50.31680417060852
Epoch 68
Training loss: 3.3532612323760986
Validation loss: 14.419880867004395
Training accuracy: 0.9958888888888885
Vaildation accuracy: 0.9750000000000002
Total time: 49.89346885681152
Epoch 69
Training loss: 4.325474262237549
Validation loss: 15.837652206420898
Training accuracy: 0.9938888888888885
Vaildation accuracy: 0.9690000000000001
Total time: 49.37493896484375
Epoch 70
Training loss: 3.8574538230895996
Validation loss: 12.929789543151855
Training accuracy: 0.9958888888888885
Vaildation accuracy: 0.9800000000000001
Saving weights
Total time: 82.63506603240967
Epoch 71
Training loss: 3.677373170852661
Validation loss: 15.108114242553711
Training accuracy: 0.9955555555555553
Vaildation accuracy: 0.976
Total time: 49.91562795639038
Epoch 72
Training loss: 4.313438415527344
Validation loss: 11.134689331054688
Training accuracy: 0.9947777777777772
Vaildation accuracy: 0.9800000000000001
Saving weights
Total time: 83.43994402885437
Epoch 73
Training loss: 3.650719165802002
Validation loss: 10.829253196716309
Training accuracy: 0.9955555555555552
Vaildation accuracy: 0.9790000000000001
Saving weights
Total time: 83.31515216827393
Epoch 74
Training loss: 3.4483885765075684
Validation loss: 16.882917404174805
Training accuracy: 0.9963333333333331
Vaildation accuracy: 0.9720000000000001
Total time: 49.70207500457764
Epoch 75
Training loss: 4.472114562988281
Validation loss: 18.27253532409668
Training accuracy: 0.9927777777777778
Vaildation accuracy: 0.9720000000000001
Total time: 50.01022696495056
Epoch 76
Training loss: 4.65834903717041
Validation loss: 16.064048767089844
Training accuracy: 0.9926666666666663
Vaildation accuracy: 0.9760000000000002
Total time: 49.37656807899475
Epoch 77
Training loss: 3.555063486099243
Validation loss: 11.689934730529785
Training accuracy: 0.9958888888888885
Vaildation accuracy: 0.9750000000000002
Total time: 49.481266021728516
Epoch 78
Training loss: 3.6655359268188477
Validation loss: 13.993217468261719
Training accuracy: 0.995444444444444
Vaildation accuracy: 0.9760000000000002
Total time: 49.76642298698425
Epoch 79
Training loss: 4.054218769073486
Validation loss: 12.01845932006836
Training accuracy: 0.9946666666666661
Vaildation accuracy: 0.9770000000000001
Total time: 49.617154121398926
Epoch 80
Training loss: 3.5529096126556396
Validation loss: 12.659688949584961
Training accuracy: 0.9961111111111108
Vaildation accuracy: 0.977
Total time: 49.375205993652344
Epoch 81
Training loss: 3.501506805419922
Validation loss: 13.506021499633789
Training accuracy: 0.9961111111111108
Vaildation accuracy: 0.978
Total time: 50.00975513458252
Epoch 82
Training loss: 2.7968130111694336
Validation loss: 12.129981994628906
Training accuracy: 0.9978888888888887
Vaildation accuracy: 0.983
Total time: 50.19576096534729
Epoch 83
Training loss: 3.4639041423797607
Validation loss: 15.84648323059082
Training accuracy: 0.9965555555555554
Vaildation accuracy: 0.9760000000000002
Total time: 49.601863861083984
Epoch 84
Training loss: 4.088804244995117
Validation loss: 14.518104553222656
Training accuracy: 0.9957777777777779
Vaildation accuracy: 0.975
Total time: 50.283888816833496
Epoch 85
Training loss: 3.584038257598877
Validation loss: 13.608121871948242
Training accuracy: 0.9955555555555553
Vaildation accuracy: 0.9760000000000002
Total time: 50.4201180934906
Epoch 86
Training loss: 2.912271499633789
Validation loss: 15.470728874206543
Training accuracy: 0.9977777777777775
Vaildation accuracy: 0.9760000000000002
Total time: 49.864295959472656
Epoch 87
Training loss: 4.526463985443115
Validation loss: 13.22993278503418
Training accuracy: 0.9927777777777778
Vaildation accuracy: 0.9739999999999999
Total time: 49.66288995742798
Epoch 88
Training loss: 3.673863410949707
Validation loss: 13.944592475891113
Training accuracy: 0.9956666666666664
Vaildation accuracy: 0.9729999999999999
Total time: 50.7358181476593
Epoch 89
Training loss: 3.4181735515594482
Validation loss: 12.872227668762207
Training accuracy: 0.9968888888888887
Vaildation accuracy: 0.974
Total time: 50.28430700302124
Epoch 90
Training loss: 4.146879196166992
Validation loss: 12.866670608520508
Training accuracy: 0.995222222222222
Vaildation accuracy: 0.976
Total time: 49.52800512313843
Epoch 91
Training loss: 3.148224115371704
Validation loss: 11.879186630249023
Training accuracy: 0.9973333333333332
Vaildation accuracy: 0.9730000000000002
Total time: 50.027369022369385
Epoch 92
Training loss: 3.200453996658325
Validation loss: 11.769929885864258
Training accuracy: 0.9974444444444441
Vaildation accuracy: 0.9770000000000001
Total time: 50.18730401992798
Epoch 93
Training loss: 3.1462228298187256
Validation loss: 15.242090225219727
Training accuracy: 0.9967777777777774
Vaildation accuracy: 0.975
Total time: 50.176302909851074
Epoch 94
Training loss: 3.7191219329833984
Validation loss: 11.667328834533691
Training accuracy: 0.995222222222222
Vaildation accuracy: 0.9780000000000001
Best validation accuracy: 0.979

Process finished with exit code 0