Confusion matrix? #188
-
|
The documentation at trex.run shows me how to see the confusion matrix but I don't know how to get any quantitative answers from it. I can eyeball how well the IDs look but is there a way to get interpretation and quantitative numbers on how good the IDs are? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Hey! It was meant to be more of a visual confirmation that everything is working as intended. So essentially like you state, to eyeball it. Unfortunately it is quite problematic to try to estimate exactly how well the training worked. At least in a real sense. Of course you can plot your own confusion matrix, the calculated accuracy and validation accuracy on the dataset - but that does not tell you if the IDs are actually being assigned correctly. It could also mean that the dataset is suboptimal (i.e. auto correlation), and you're overfitting to it. That's what the "uniqueness" was meant to mitigate at least in part. This is simply a measure of "of a subset of all frames, distributed across the entire video, in how many (%) of those frames does the network predict unique IDs. Meaning that within a frame, there is only one instance of ID 0, one instance of ID 1, etc. Obviously this lacks the part about whether identities are the correct identities, but there is a good chance that if you have >95% uniqueness, it's predicting correct IDs. Cool thing is that it's independent of everything else that is being calculated. Uniqueness is displayed during training (also in the terminal, so you could look at the logs to figure this out), as well as in this file: history_path = output_path+"_"+str(accumulation_step)+"_history.npz" # output_dir/filename_<N>_history.npz
TRex.log("saving histories at '"+history_path+"'")which is also printed out in the terminal. The other metrics like per-class-accuracy are also in there. I will also say that I do not trust confusion matrices anyway. Or maybe I am just the one who's confused here, who knows. If you really want it though, you can edit the python script if you like, which is called learn_static.py and is in the application folder. Hope that helped! |
Beta Was this translation helpful? Give feedback.
-
|
Ok thank you! That helps! |
Beta Was this translation helpful? Give feedback.
Hey! It was meant to be more of a visual confirmation that everything is working as intended. So essentially like you state, to eyeball it. Unfortunately it is quite problematic to try to estimate exactly how well the training worked. At least in a real sense. Of course you can plot your own confusion matrix, the calculated accuracy and validation accuracy on the dataset - but that does not tell you if the IDs are actually being assigned correctly. It could also mean that the dataset is suboptimal (i.e. auto correlation), and you're overfitting to it. That's what the "uniqueness" was meant to mitigate at least in part. This is simply a measure of "of a subset of all frames, distributed ac…