Retry pipeline does not work

[szymek116]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Describe your issue

After upgrade from version 1.8.1 pipeline retry from UI functionality does not work. Tested on vanila 1.9.1 installation and does not work either.

Versions:
kubeflow: 1.9.1
minikube: v1.34.0
k8s: v1.31.0

Steps to reproduce the issue

Reproduced on vanilla install

1. deploy one node minikube cluster
2. deploy kubeflow using manifests
3. Create default notebook using jupyter-scipy:v1.9.2 image
4. Create following pipeline, deploy, run and retry it from UI. Despite message from UI that retry succeeded no containers are spawned (while on older versions they have in the process of retry)

from kfp import dsl

@dsl.component
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    with open('x.py') as f: s = f.read() # this fails
    return hello_text

@dsl.pipeline
def hello_pipeline(recipient: str) -> str:
    hello_task = say_hello(name=recipient)
    return hello_task.output

from kfp import compiler

compiler.Compiler().compile(hello_pipeline, 'pipeline.yaml')

Put here any screenshots or videos (optional)

No response

[juliusvonkohout]

"Are you posting in the right repository ?" Maybe kubeflow/pipelines is the right place.

[szymek116]

no idea :) I meant at that stage didn't get to the root cause. But if that is something related to network policies I guess it might be here as well, I think they were introduced recently ?

But found recently similar issues in pipelines repo as well with no explanation yet:
kubeflow/pipelines#11288
kubeflow/pipelines#11329

[juliusvonkohout]

I think you need to provide a bit more debug info then cc @tarekabouzeid

[ntny]

@szymek116, hi could you share which kind of workflow are you using? Tekton or Argo? Also trying to determine the root cause.