Artifacts lineage is tracked but the Dashboard and Artifacts tabs are empty

[polyaxon-team]

From slack

I'm currently facing a problem regarding the polyaxon tracking module. When running a job, I can see the logged metrics, parameters, artifacts in the lineage tab, although on the Dashboard tab, I can see only the chart's name and no values being tracked.

Have anyone encountered this issue before? Any help will be much appreciated.

I have tried a basic example of MNIST, when training it on CPU it tracks the metrics, but on GPU it doesn't.

Also, when I'm running a training job (MNIST) on the CPU it tracks the metrics and I can see the charts in the Dashboard, but when I configure the polyaxonfile to use GPU, the logs are indeed saying that the GPU is being used, but the metrics charts are empty.

[polyaxon-team]

You are probably deploying Polyaxon with the default artifacts store and you are using multiple nodes.

If that's the case you will need to upgrade your deployment with one if the artifacts stores that supports multi-node deployment: https://polyaxon.com/integrations/#Artifacts

More info about the artifacts store default behavior: https://polyaxon.com/docs/setup/connections/#artifactsstore

Also, when I'm running a training job (MNIST) on the CPU it tracks the metrics and I can see the charts in the Dashboard, but when I configure the polyaxonfile to use GPU, the logs are indeed saying that the GPU is being used, but the metrics charts are empty.

Your GPU is on different node that's why it shows the metrics and artifacts when running with CPU, note that even with CPU if it's on a different node nothing will show up, so you will need a PVC / blob storage accessible to all nodes.