How should we import Task SDK items in core?

[amoghrajesh]

Now that we are winding up porting definitions and general items to task sdk that can serve as building blocks for building dags for the DAG author, one thought that pops is -- how do we import "correctly" into the core from task sdk.

For more context, "BaseOperatorLink" is a definition users would need to create their OperatorLinks, but not something like lets say "DAG" / "Params". So what would be the right way to import in core? The entire file path (from airflow.sdk.definitions.baseoperatorlink import BaseOperatorLink) or just holistically from "sdk" (from airflow.sdk import BaseOperatorLink).

Related discussion #47008 (comment)

[potiuk]

Always from airflow.sdk

[amoghrajesh]

Similar opinion too!

[potiuk]

More info: Generally speaking - the whole refactoring of packages we do is to follow the same structure:

<DISTRIBUTION>
              | pyproject.toml
              | src
                   \ airflow
                            \ <AIRFLOW_SUB_PACKAGE>
              | tests
                     \ <TEST_NAMESPACE>
                       ....

And we want "src" and "tests" to be the PYTHONPATH roots.

For example for task sdk:

• DISTRIBUTION = task-sdk
• AIRFLOW_SUB_PACKAGE = sdk
• TEST_NAMESPACE = task_sdk

For providers (say google)

• DISTRIBUTIONR = providers/google
• AIRFLOW_SUB_PACKAGE = providers.google
• TEST_NAMESPACES:
  * unit.google
  * system.google
  * integration.google

And the way how we should refer to it:

from airflow.<AIRFLOW_SUB_PACKAGE> <- imports the "production code"

from TEST_NAMESPACE <- imports tests code

Examples:

"production" code:

from airflow.sdk import BaseOperatorLink
from airlfow.providers.google.hooks import BaseHook

"test code"

from task_sdk import make_client, make_client_w_dry_run, make_client_w_responses

from unit.google.cloud.utils.gcp_authenticator import (
    GCP_GCS_KEY,
    GCP_SECRET_MANAGER_KEY,
)

We might want to turn that into part of our contributing guide.

[potiuk]

BTW. Working on airflow-core DISTRIBUTION now :)

[potiuk]

BTW. We could potentially allow relative import as well, I am abit hesitant, because it makes refactorings more difficult, also some ways of importing code (for example inporting our DAG files) might drop the information about base /relative imports (we saw it happening in the past where relative imports were not working).

For example if we are in ariflow.sdk.__init__.py we could do:

import .definitions.SomeDefinition

But IMHO that is a bit implicit and while it's shorter to write, it's also much more difficult to read, and causes extra cognitive load - because you need to know where you are. And when in doubt, I prefer optimizing for reading rather than writing.