Sending Databricks Token to Workers

[samuel-gerstein]

Hi,

I am trying to use MLFlow with Spark distributed training. For TorchDistributor to log metrics however, it needs my Databricks token. In TorchDistributor's run function you could just initialize HOST_TOKEN, however there is no functionality to pass this in with Nixtla.

Is there a way to work around this?

[jmoralez]

I think you can set it as a cluster env variable