Move airflow sources to airflow-core package (apache#47798)

This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:

* apache-airflow - becomes an empty, meta no-code distribution that
  only has dependencies to apache-airflow-core and task-sdk
  distributions and it has preinstalled provider distributions
  added in standard "wheel" distribution. All "extras" lead
  either to "apache-airflow-core" extras or to providers - the
  dependencies and optional dependencies are calculated differently
  depending on "editable" or "standard" mode - in editable mode,
  just provider dependencies are installed for preinstalled providers
  in standard mode - those preinstalled providers are dependencies.

* the apache-airflow-core distribution contains all airflow core
  sources (previously in apache-airflow) and it has no provider
  extras. Thanks to that apache-airflow distribution does not
  have any dynamically calculated dependencies.

* the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
  build hooks that add custom build target and implement custom
  cleanup in order to implement compiling assets as part of the build.

* During the move, the following changes were applied for consistency:

  * packages when used in context of distribution packages have been
    renamed to "distributions" - including all documentations and
    commands in breeze to void confusion with import packages
    (see
https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)

  * all tests in `airflow-core` follow now the same convention
    where tests are in `unit`, `system` and `integration` package.
    no extra package has been as second level, because all the
    provider tests have "<PROVIDER>" there, so we just have to avoid
    naming airflow unit."<PROVIDER>" with the same name as provider.

  * all tooling in CI/DEV have been updated to follow the new
    structure. We should always build to packages now when we
    are building them using `breeze`.

Author: potiuk

.dockerignore

@@ -27,7 +27,7 @@
 **
 
 # Allow only these directories
-!airflow
+!airflow-core
 !common
 !dags
 !dev
@@ -75,19 +75,24 @@
 !pyproject.toml
 !manifests
 !generated
+
+# Reproducible build configuration
+!reproducible_build.yaml
+
 # Now - ignore unnecessary files inside allowed directories
 # This goes after the allowed directories
 
 # Git version is dynamically generated
-airflow/git_version
+airflow-core/src/airflow/git_version
 
 # Exclude node/pmpme caches..
 **/.pnpm-store
 **/node_modules
 # Exclude link to docs
-airflow/ui/static/docs
+airflow-core/src/airflow/ui/static/docs
 
 # Legacy www exclusions (just to exclude things in case you switch to old branches and generate assets)
+# THey should not be in airflow-core as "www" never was there.
 airflow/www/static/docs
 airflow/www/static/dist
 airflow/www/node_modules
@@ -123,7 +128,7 @@ docs/**/_api/**
 **/*.egg
 
 # But ensure UI dist files are included
-!airflow/ui/dist
+!airflow-core/src/airflow/ui/dist
 !providers/fab/src/airflow/providers/fab/www/dist
 
 # Exclude temporary vi files

.github/CODEOWNERS

@@ -80,7 +80,7 @@ Dockerfile.ci @potiuk @ashb
 /dev/PROVIDER_PACKAGE_DETAILS.md @eladkal
 /dev/README.md  @kaxil
 /dev/README_RELEASE_*.md  @kaxil @pierrejeambrun
-/dev/README_RELEASE_PROVIDER_PACKAGES.md  @eladkal
+/dev/README_RELEASE_PROVIDERS.md  @eladkal
 ISSUE_TRIAGE_PROCESS.rst @eladkal
 
 # AIP-52 - Setup and Teardown

.github/ISSUE_TEMPLATE/airflow_bug_report.yml

@@ -7,7 +7,7 @@ body:
     attributes:
       # yamllint disable rule:line-length
       value: "
-        <img src='https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
+        <img src='https://raw.githubusercontent.com/apache/airflow/main/airflow-core/docs/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
         Thank you for finding the time to report the problem!
 
         We really appreciate the community's efforts to improve Airflow.

.github/ISSUE_TEMPLATE/airflow_doc_issue_report.yml

@@ -7,7 +7,7 @@ body:
     attributes:
       # yamllint disable rule:line-length
       value: "
-        <img src='https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
+        <img src='https://raw.githubusercontent.com/apache/airflow/main/airflow-core/docs/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
         Thank you for finding the time to report the problem!
 
         We really appreciate the community's efforts to improve Airflow.

.github/ISSUE_TEMPLATE/airflow_helmchart_bug_report.yml

@@ -7,7 +7,7 @@ body:
     attributes:
       # yamllint disable rule:line-length
       value: "
-        <img src='https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='120' height='120'>
+        <img src='https://raw.githubusercontent.com/apache/airflow/main/airflow-core/docs/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='120' height='120'>
         Thank you for finding the time to report the problem!
 
         We really appreciate the community's efforts to improve Airflow.

.github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml

@@ -7,7 +7,7 @@ body:
     attributes:
       # yamllint disable rule:line-length
       value: "
-        <img src='https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
+        <img src='https://raw.githubusercontent.com/apache/airflow/main/airflow-core/docs/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
         Thank you for finding the time to report a problem!
 
         We really appreciate the community's efforts to improve Airflow.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -7,7 +7,7 @@ body:
     attributes:
       # yamllint disable rule:line-length
       value: "
-        <img src='https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
+        <img src='https://raw.githubusercontent.com/apache/airflow/main/airflow-core/docs/img/logos/airflow_64x64_emoji_transparent.png' align='left' width='80' height='80'>
         Thank you for finding the time to propose new feature!
 
         We really appreciate the community efforts to improve Airflow.

.github/SECURITY.md

@@ -118,7 +118,7 @@ Team and how members of it are chosen can be found in the
 
 ### Does CVE in Airflow Providers impact Airflow core package ?
 
-Airflow core package is released separately from provider packages. While Airflow comes with ``constraints``
+Airflow core package is released separately from provider distributions. While Airflow comes with ``constraints``
 which describe which version of providers have been tested when the version of Airflow was released, the
 users of Airflow are advised to install providers independently of Airflow core when they want to apply
 security fixes found and released in providers. Therefore, the issues found and fixed in providers do

.github/actions/breeze/action.yml

@@ -45,7 +45,7 @@ runs:
       shell: bash
       run: breeze ci free-space
       env:
-        AIRFLOW_SOURCES_ROOT: "${{ github.workspace }}"
+        AIRFLOW_ROOT_PATH: "${{ github.workspace }}"
     - name: "Get Python version"
       shell: bash
       run: >
@@ -56,4 +56,4 @@ runs:
       shell: bash
       run: breeze setup config --no-cheatsheet --no-asciiart
       env:
-        AIRFLOW_SOURCES_ROOT: "${{ github.workspace }}"
+        AIRFLOW_ROOT_PATH: "${{ github.workspace }}"

.github/boring-cyborg.yml

@@ -301,11 +301,11 @@ labelPRBasedOnFilePath:
     - providers/zendesk/**
 
   area:API:
-    - airflow/api/**/*
-    - airflow/api_fastapi/**/*
+    - airflow-core/src/airflow/api/**/*
+    - airflow-core/src/airflow/api_fastapi/**/*
     - clients/**/*
-    - docs/apache-airflow/stable-rest-api-ref.rst
-    - tests/api_fastapi/**/*
+    - airflow-core/docs/stable-rest-api-ref.rst
+    - airflow-core/tests/unit/api_fastapi/**/*
 
   area:dev-tools:
     - scripts/**/*
@@ -331,94 +331,93 @@ labelPRBasedOnFilePath:
     - docs/helm-chart/**
 
   area:UI:
-    - docs/apache-airflow/ui.rst
-    - airflow/ui/**/*
+    - airflow-core/docs/ui.rst
+    - airflow-core/src/airflow/ui/**/*
 
   area:CLI:
-    - airflow/cli/**/*.py
-    - tests/cli/**/*.py
-    - docs/apache-airflow/cli-and-env-variables-ref.rst
-    - docs/apache-airflow/howto/usage-cli.rst
+    - airflow-core/src/airflow/cli/**/*.py
+    - airflow-core/tests/unit/cli/**/*.py
+    - airflow-core/docs/cli-and-env-variables-ref.rst
+    - airflow-core/docs/howto/usage-cli.rst
 
   area:Lineage:
-    - airflow/lineage/**/*
-    - tests/lineage/**/*
-    - docs/apache-airflow/administration-and-deployment/lineage.rst
+    - airflow-core/src/airflow/lineage/**/*
+    - airflow-core/tests/unit/lineage/**/*
+    - airflow-core/docs/administration-and-deployment/lineage.rst
 
   area:Logging:
-    - airflow/utils/log/**/*
-    - docs/apache-airflow/administration-and-deployment/logging-monitoring/logging-*.rst
-    - tests/utils/log/**/*
+    - airflow-core/src/airflow/utils/log/**/*
+    - airflow-core/docs/administration-and-deployment/logging-monitoring/logging-*.rst
+    - airflow-core/tests/unit/utils/log/**/*
     - providers/**/log/*
 
   area:Plugins:
-    - airflow/cli/commands/local_commands/plugins_command.py
-    - airflow/plugins_manager.py
-    - tests/cli/commands/local_commands/test_plugins_command.py
-    - tests/plugins/**/*
-    - docs/apache-airflow/administration-and-deployment/plugins.rst
+    - airflow-core/src/airflow/cli/commands/local_commands/plugins_command.py
+    - airflow-core/src/airflow/plugins_manager.py
+    - airflow-core/tests/unit/cli/commands/local_commands/test_plugins_command.py
+    - airflow-core/tests/unit/plugins/**/*
+    - airflow-core/docs/administration-and-deployment/plugins.rst
 
   area:Scheduler:
-    - airflow/jobs/scheduler_job_runner.py
-    - docs/apache-airflow/administration-and-deployment/scheduler.rst
-    - tests/jobs/test_scheduler_job.py
+    - airflow-core/src/airflow/jobs/scheduler_job_runner.py
+    - airflow-core/docs/administration-and-deployment/scheduler.rst
+    - airflow-core/tests/unit/jobs/test_scheduler_job.py
 
   area:DAG-processing:
-    - airflow/dag_processing/**/*
-    - airflow/jobs/dag_processor_job_runner.py
-    - docs/apache-airflow/administration-and-deployment/dagfile-processing.rst
-    - tests/dag_processing/**/*
+    - airflow-core/src/airflow/dag_processing/**/*
+    - airflow-core/src/airflow/jobs/dag_processor_job_runner.py
+    - airflow-core/docs/administration-and-deployment/dagfile-processing.rst
+    - airflow-core/tests/unit/dag_processing/**/*
 
   area:Executors-core:
-    - airflow/executors/**/*
-    - docs/apache-airflow/core-concepts/executor/**/*
-    - tests/executors/**/*
+    - airflow-core/src/airflow/executors/**/*
+    - airflow-core/docs/core-concepts/executor/**/*
+    - airflow-core/tests/unit/executors/**/*
 
   area:Secrets:
-    - airflow/secrets/**/*
-    - tests/secrets/**/*
+    - airflow-core/src/airflow/secrets/**/*
+    - airflow-core/tests/unit/secrets/**/*
     - providers/**/secrets/*
-    - docs/apache-airflow/security/secrets/**/*
+    - airflow-core/docs/security/secrets/**/*
 
   area:Triggerer:
-    - airflow/cli/commands/local_commands/triggerer_command.py
-    - airflow/jobs/triggerer_job_runner.py
-    - airflow/models/trigger.py
+    - airflow-core/src/airflow/cli/commands/local_commands/triggerer_command.py
+    - airflow-core/src/airflow/jobs/triggerer_job_runner.py
+    - airflow-core/src/airflow/models/trigger.py
     - providers/standard/src/airflow/providers/standard/triggers/**/*
-    - tests/cli/commands/local_commands/test_triggerer_command.py
-    - tests/jobs/test_triggerer_job.py
-    - tests/models/test_trigger.py
+    - airflow-core/tests/unit/cli/commands/local_commands/test_triggerer_command.py
+    - airflow-core/tests/unit/jobs/test_triggerer_job.py
+    - airflow-core/tests/unit/models/test_trigger.py
     - providers/standard/tests/unit/standard/triggers/**/*
 
   area:Serialization:
-    - airflow/serialization/**/*
-    - airflow/models/serialized_dag.py
-    - tests/serialization/**/*
-    - tests/models/test_serialized_dag.py
-    - docs/apache-airflow/administration-and-deployment/dag-serialization.rst
+    - airflow-core/src/airflow/serialization/**/*
+    - airflow-core/src/airflow/models/serialized_dag.py
+    - airflow-core/tests/unit/serialization/**/*
+    - airflow-core/tests/unit/models/test_serialized_dag.py
+    - airflow-core/docs/administration-and-deployment/dag-serialization.rst
 
   area:core-operators:
-    - airflow/operators/**/*
-    - airflow/hooks/**/*
-    - airflow/sensors/**/*
-    - tests/hooks/**/*
-    - tests/sensors/**/*
-    - docs/apache-airflow/operators-and-hooks-ref.rst
-    - docs/apache-airflow/howto/operator/*
+    - airflow-core/src/airflow/hooks/**/*
+    - airflow-core/src/airflow/sensors/**/*
+    - airflow-core/tests/unit/hooks/**/*
+    - airflow-core/tests/unit/sensors/**/*
+    - airflow-core/docs/operators-and-hooks-ref.rst
+    - airflow-core/docs/howto/operator/*
 
   area:production-image:
     - Dockerfile
     - docs/docker-stack/**/*
     - docker_tests/**/*
 
   area:system-tests:
-    - tests/system/**/*
+    - airflow-core/tests/system/**/*
 
   area:task-sdk:
     - task-sdk/**/*
 
   area:db-migrations:
-    - airflow/migrations/versions/*
+    - airflow-core/src/airflow/migrations/versions/*
 
   area:providers:
     - providers/**/*
@@ -445,7 +444,7 @@ firstPRWelcomeComment: >
 
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory).
   Adding a new operator? Check this short
-  [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst)
+  [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst)
   Consider adding an example DAG that shows how users should use it.
 
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst)