Add Python 3.12 support (#36755)

[potiuk]

Finally after a number of dependency upgrades we seem to be able to upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion)

Several providers are excluded from being installed and wait for Python 3.12, but it should not block Airlfow's general 3.12 support.

Co-authored-by: dirrao [email protected]
(cherry picked from commit 76dee0b)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[dirrao]

Awesome. I am eagerly waiting for the initial release. Thank you for the amazing work @potiuk.

[Taragolis]

Oh.. we need conditional for Pendulum2 tests since pendulum 2 not supported in Python 3.12 due to removed parts

[potiuk]

Oh.. we need conditional for Pendulum2 tests since pendulum 2 not supported in Python 3.12 due to removed parts

Good point. We also need to have it in release notes.

[potiuk]

Oh.. we need conditional for Pendulum2 tests since pendulum 2 not supported in Python 3.12 due to removed parts

Good point. We also need to have it in release notes.

Added both

[Taragolis]

Ahh this one is regression of #38027 in case of Python 3.12 🤕 , seems like botocore/boto3 internally use deprecated in stuff 🙄

      def add_auth(self, request):
          if self.credentials is None:
              raise NoCredentialsError()
  >       datetime_now = datetime.datetime.utcnow()
  E       DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: 

[potiuk]

Ahh this one is regression of #38027 in case of Python 3.12 🤕 , seems like botocore/boto3 internally use deprecated in stuff

Yeah . Was just looking at it . What's the best fix ?

[Taragolis]

You could cherry pick #38043, locally it works fine in python 3.12.

[potiuk]

I just pushd my version :) . Almost the same

            if sys.version_info >= (3, 12):
                # Botocore generates deprecation warning on Python 3.12 connected with utcnow use
                warnings.filterwarnings("ignore", message=r".*datetime.utcnow.*", category=DeprecationWarning)

[potiuk]

OK. It seems to be quite good. I removed the "latest version only" and pushed a PR that should do the same things that the canary build will do eventually. It's ready for review in general.

🤞

I had to add a few more exclusions to make sure that we are skipping some of the imports in some of the "common" tests - when some providers are excluded - because eventually the 3.12 image will be missing some librarires and those imports will fail . I think I covered all cases and that the timeouts we saw last time were caused by these problems and by the impersonation test which is now quarantined.

Again 🤞

[potiuk]

BTW. I also added here few controls in our CI (labels) that allow to iterate on such PRs with far less resources used and faster :)

[potiuk]

Also I have found why we had the difference in tests failing for cassandra after I merged last PR - main reason was because cassandra was not yet excluded in main - so I will need to merge #38050 first and only after it will be merged, I will have to (again) rebase this PR after increasing the EPOCH.