[<Ray component: Data] - inconsistent URL handling in Ray's Databricks integration

[leibovitzgil]

What happened + What you expected to happen

The current Databricks integration in Ray Data requires providing the Databricks host URL without the "https://" prefix. However, this creates compatibility issues when using Ray Data alongside MLflow, as MLflow's Databricks integration (which uses the same DATABRICKS_HOST environment variable) expects the URL to include the "https://" prefix.

Versions / Dependencies

ray[data]==2.40.0
mlflow==2.10.2

Reproduction script

class SomeModel:
def init(self):
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
fr_model_path = "models:/SOME_ML_FLOW_PATH"
model = mlflow.pytorch.load_model(model_uri=fr_model_path)
model.to(self.device)
model.eval()

ds = read_databricks_tables(
warehouse_id="xxxx",
catalog="xxx",
schema="xxx",
query=SOME_SQL_QUERY
)

df = (
ds
.map(
SomeModel,
concurrency=5
))

Issue Severity

High: It blocks me from completing my task.