Made sqlalchemy dependency optional for Databricks provider

[justinpakzad]

Made sqlalchemy an optional dependency for the Databricks provider. AirflowOptionalProviderFeatureException() gets raised if it's not installed. Just a quick note - Databricks uses the databricks-sqlalchemy package, since it's now an optional dependency, I needed to add it to the dev dependencies to ensure the tests that rely on it work properly.

closes: #59900

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

I think, the compat tests are failing because of an interesting side-effect of making databricks-sqlalchemy optional. What happens is that we we are installing "dev" dependencies with airflow 3 -> so latest databrics-sqlalchemy is installed (2.0.8) - because it supports sqlalchemy 2 (and airflow 3.2 already supports it). Then when later airflow 2 or 3 versions are installed, they bring the sqlalchemy to 1 (because they do not support 2) but they do not bring databricks-sqlalchemy down, when databricks provider is installed then (without sqlalchemy extra) - it also will not bring the databricks-sqlalchemy down :(

[potiuk]

I wil send a workaround patch shortly for that "optional sqlalchemy extra" soon

[justinpakzad]

Thanks for chiming in. Will update the PR w the changes once you send the workaround patch over.

[potiuk]

Fixup pushed

[justinpakzad]

Fixup pushed

Thanks for that. I wasn't sure if I was meant to make the change myself, so I was waiting to update the "databricks-sqlalchemy>=1.0.2" dependency. Let me update that now.

[jscheffl]

Looks good for me! Restarted CI - hopefully will turn green in a moment.