Making evaluation after training work

[wailoktam]

Hi, I have no problem getting evaluation during training work by setting evaluation mode to all in config. But my supervisor suggests me that:

1. I should not see the evaluation result during training
2. I should not use any of the test sets during development.

To achieve this:

1. I have to put back the commented out lines for evaluation:

evaluator.load_epoch(model, 54) evaluator.get_mrr(model, evaluate_all=True)

1. I also need to add a line to change the evaluation set during training to dev

#self._eval_sets = dict([(s, self.load(s)) for s in ['dev', 'test1', 'test2']]) self._eval_sets = dict([('dev', self.load('dev')) ])

1. I also need to add a line to set the evaluation set to test 1 and/or test 2 when doing the evaluation in 1. I try:

evaluator.eval_sets = dict([('dev', evaluator.load('test1')) ]) evaluator.load_epoch(model, 54) evaluator.get_mrr(model, evaluate_all=True)

But it gives me the error:

TypeError: 'dict' object is not callable

What should I do? Thanks in advance.

[codekansas]

I think _eval_sets should have a leading underscore, as in evaluator._eval_sets = dict([('dev', evaluator.load('test1')) ]) evaluator.load_epoch(model, 54) evaluator.get_mrr(model, evaluate_all=True). The way I wrote it is so that class variables have a leading underscore while the access methods do not. I think this is causing the issue.