ContainerOp alternative

[tom-dd]

Per request of @Bobgy:

Currently, when using ContainerOp you will get the following warning:

FutureWarning: Please create reusable components instead of constructing ContainerOp instances directly. Reusable components are shareable, portable and have compatibility and support guarantees. Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op) and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.load_component_from_file
  category=FutureWarning,

The alternatives requires you to specify your ContainerOp instance as either yaml or json if you want to use it inline. This option however lacks the editor/type checking support that you would get when using the Python based ContainerOp.

Proposals:

• Offer a Python based alternative for defining your ContainerOp instance in a future proof way
• Yaml schema's could offer editor support for the content of the definition, however it will probably be less convinient than native Python code.

[Bobgy]

/assign @chensun @Ark-kun

[Ark-kun]

The alternatives requires you to specify your ContainerOp instance as either yaml or json if you want to use it inline.

There are also lightweight python components created using kfp.components.create_component_from_func.

This option however lacks the editor/type checking support that you would get when using the Python based ContainerOp.

Can you describe the problems you're seeing?
The loaded components are represented as strongly typed functions.
When working in a Jupyter environment, the editor shows the help for the parameters and help(op) works as well. Does this work for you?

Yaml schema's could offer editor support for the content of the definition

Can you please help me understand this request. ComponentSpec has a schema and also when you load a component from URL, file or inline text, the component structure is validated. Is there something missing?

Offer a Python based alternative for defining your ContainerOp instance in a future proof way

It's possible to construct an instance of the kfp.components.structures.ComponentsSpec class and then .save it. I'm not sure this is more convenient than lightweight python components or manual component.yaml authoring. I'd like to hear your thoughts.

[Bobgy]

@Ark-kun you might want to check more context in the slack discussion: https://kubeflow.slack.com/archives/CE10KS9M4/p1604333477307400

[munagekar]

Comments on #4644 might be relevant.

[Bobgy]

There was a great example for component spec python class in #3748 (comment).

[tom-dd]

There are also lightweight python components created using kfp.components.create_component_from_func.

In our case, we are coming from a "legacy" environment where we have all our pipeline code in a large container, so unfortunately we are not lightweight anymore.

Can you describe the problems you're seeing?
The loaded components are represented as strongly typed functions.
When working in a Jupyter environment, the editor shows the help for the parameters and help(op) works as well. Does this work for you?

We are not using Notebooks at the moment, so I cannot judge how that would work. However, when you have pylint/mypy check your code for type safety it helps if also your ContainerOp definitions can be checked as well.

Yaml schema's could offer editor support for the content of the definition

Can you please help me understand this request. ComponentSpec has a schema and also when you load a component from URL, file or inline text, the component structure is validated. Is there something missing?

It is validated when you load them, but having validation/autocomplete while you are writing specs is very helpful

It's possible to construct an instance of the kfp.components.structures.ComponentsSpec class and then .save it. I'm not sure this is more convenient than lightweight python components or manual component.yaml authoring. I'd like to hear your thoughts.

I guess this is close to what could work. If I take the (a reduced) example from the link @Bobgy posted, I can identify a case however that would be caught by my editor tooling with the ContainerOp route:

component_spec = ComponentSpec(
    name='Dummy op',
    description='Dummy component for illustrative purposes',
    inputs=[
        InputSpec(name='input1', type='String'),
    ],
    implementation=ContainerImplementation(container=ContainerSpec(
        image="dummy-image",
        command=[
            "python", "runner.py",
            "--input1", InputValuePlaceholder('inpt1'),
        ],
    ))
)

vs

def slack_notification(input1: str) -> ContainerOp:
    return ContainerOp(
        name="dummy-image",
        image="dummy-image",
        arguments=[
                "python", "runner.py",
                "--input1", inpt1
        ],
    )

My editor tooling would find that inpt1 doesn't exist

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pbarker]

Also agree with this, maybe my editor (vscode) isn't loading components right, but having the ability to use autocomplete around operations is pivotal to developer experience. I would suggest not deprecating container_op unless necessary

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreclaudino]

@Ark-kun , you sugested using create_component_from_func, but it doen't allow creating components from containers in other technologies than Python, what in my openion is a great feature of Kubeflow, which is being lost when breaking ContainerOp.

[joesan]

I'm also seeing this warning:

FutureWarning: Please create reusable components instead of constructing ContainerOp instances directly. Reusable components are shareable, portable and have compatibility and support guarantees. Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op) and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.components.html#kfp.components.load_component_from_file
  warnings.warn(

Do I simply ignore and just proceed?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.