Logging issue when using custom metrics #6698
Unanswered
cemde
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
Replies: 1 comment 1 reply
-
Can you share the complete stacktrace? In my case the |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am training a NN with pytorch lightning and would like to calculate the BrierScore at every step. I implemented this using the
torchmetrics.Metric
class. This breaks the logging.With the
Module
, I initialise a dictionaryself.metrics_dict
in which I store the metric functions. It looks like thisAt the end of each step I call
I have a
ModelCheckpoint
configured:The ModelCheckpoint works fine when I only use metrics that come with the
torchmetrics
package, e.g.Accuracy
andF1
(subset of the dict shown above). As soon as I add my own metricBrierScore
(or any other custom for that matter), the ModelCheckpoint raises an Exception:(line 495 of
p_l/callbacks/model_checkpoint.py
)My own metric works fine during debugging:
The odd thing is that this error arrises after 5 training steps in epoch 0. Further, the metric functions work themselves.
What am I doing wrong?
Snippets:
Creating the
self.metrics_dict
Beta Was this translation helpful? Give feedback.
All reactions