Requirements to run AI Pipeline through Elyra on ODH on Op1st (more docs?)

[pacospace]

As Data Scientist,

I would like to run AI Pipeline using Elyra. As runtime need to be added to each step and ODH currently supports Kubeflow.
There are some required information missing for the user.

How do I find out which is:

• Kubeflow Pipelines API Endpoint
• Cloud Object Storage Endpoint

do we get user credentials for Ceph? To run AI Pipeline also:

• Cloud Object Storage Username (required)
• Cloud Object Storage Password (required)

and also is there any default bucket assigned to user?

• Cloud Object Storage Bucket Name (required)

Reference: https://elyra.readthedocs.io/en/latest/user_guide/runtime-conf.html#using-elyra-runtimes-user-interfac

[4n4nd]

Will need to:

1. Deploy the Kubeflow Pipelines components from upstream Kubeflow.
2. Add the imagestream for s2i-elyra in the Jupyterhub namespace.

[tumido]

You can use the ODH's fork of Kubeflow for the best upstream experience on Openshift:

See:
https://github.com/opendatahub-io/manifests/blob/v1.0-branch-openshift/kfdef/kfctl_openshift.yaml

It'll deploy the Kubeflow Pipelines for you. It should be easy to extract only those parts that you need. I'll look into that tomorrow.

[4n4nd]

yeah i found that and made https://github.com/4n4nd/odh/tree/add-kubeflow-pipeline/kfdef/kubeflow-pipeline

[durandom]

Deploy the Kubeflow Pipelines components from upstream Kubeflow.

let's not deploy anything other upstream than ODH

[durandom]

let's put this on hold until @HumairAK deployed "all the things" (incl. KF pipelines) via operate-first/apps#10

then re-visit the UX of that deployment here

[goern]

ping

[pacospace]

What is the status of this issue? :)

[pacospace]

https://opendatahub.io/docs/kubeflow/installation.html

[durandom]

@HumairAK @tumido can we make KFPipelines a priority to be installed next?

[HumairAK]

Yep will take a look into this.

[4n4nd]

kubeflow/manifests#1711

[goern]

@pacospace will have a time-boxed look if we can help with deploying https://developer.ibm.com/blogs/kubeflow-pipelines-with-tekton-and-watson/

@vpavlin @nakfour comments? ideas?

[4n4nd]

@goern @pacospace this is next on my list after the monitoring onboarding docs (issue), should I make this a priority and work with @pacospace before the monitoring docs?

[pacospace]

@goern @pacospace this is next on my list after the monitoring onboarding docs (issue), should I make this a priority and work with @pacospace before the monitoring docs?

Don't want to make it more priority than what you have to do already :) Thanks @4n4nd

When you have time, as discussed, let's look at: https://github.com/kubeflow/kfp-tekton/blob/master/guides/kfp-tekton-openshift.md#deploy-kubeflow-pipelines-with-tekton-backend and then we can test running a simple pipelines once endpoints are available.

and thanks @tumido @goern @durandom :)

[martinpovolny]

I would like to sum up the status of this. I know that @tumido, @4n4nd and @pacospace, you all got somehow involved. I am also very interested.

Can we, please, sync and create some smaller tasks?

I have deployed kfdef for Kubeflow from here https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_openshift.v1.2.0.yaml to my cluster in QuickLab. But that is a lot of components. Some could be probably skipped for now.

@4n4nd : did you play with it? @durandom suggested you probably did. Did you get anything running? If so what was the size of the cluster? Did you follow what @pacospace suggested above?

I also hit into this issue: kubeflow/kfctl#460 that @HumairAK created (thanks, @tumido for the pointer).

[vpavlin]

FYI @nakfour has been working on adding KF pipelines on Tekton into a default stack kubeflow/manifests#1721

[tumido]

We've talked with @pacospace and then I've synced with @4n4nd multiple times over this. If I understood the task properly, @pacospace is more interested in deploying the Kubeflow pipelines on Tekton, not just "some" Kubeflow pipelines (by default this is backed by Argo, the Tekton backed pipelines are a new thing).

@pacospace (and @goern) pointed us to:

• https://developer.ibm.com/blogs/kubeflow-pipelines-with-tekton-and-watson/
• https://github.com/kubeflow/kfp-tekton/blob/master/guides/kfp-tekton-openshift.md#deploy-kubeflow-pipelines-with-tekton-backend
• the above lead to this kfdef: https://github.com/IBM/KubeflowDojo/blob/master/OpenShift/manifests/kfctl_openshift_pipelines.v1.1.0.yaml

This was before @nakfour's PR + also also @pacospace suggested against opening a PR to ODH since he suspected somebody was already working on it.

In the meantime I think @4n4nd already experimented with the IBM manifests a bit to gain some insight into how it all works and fits together. So he's probably the best person to ask about that bit.

[goern]

Sounds like we are good to go on the technical side, next steps are:

• wait for ODH 1.0 release to include kubeflow+tekton, see https://opendatahub.io/docs/roadmap/future.html Adding KFserving and kfp tekton to Openshift Stack kubeflow/manifests#1721 @vpavlin
• deploy ODH1.0 to Op1st @martinpovolny
• verify that we can use that environment for https://www.mlprague.com/ @vpavlin @pacospace
• deploy https://github.com/thoth-station/elyra-aidevsecops-tutorial to ODH1.0 on Op1st @pacospace
• create a walk-thru video of the whole story @pacospace anyone else?

wdyt?

[4n4nd]

keeping issue open until confirmed kf-pipelines works with elyra

[pacospace]

I was able to submit a Kubeflow pipeline with Tekton from new Elyra image for workshop!! 💯 Thanks @4n4nd @harshad16.

I'm working on making the pipeline successfull now!