Reuse struct from Kubeflow Pipelines

.github/scripts/tests/README.md

@@ -8,14 +8,17 @@ To prevent file monitoring issues in development environments (e.g., IDEs or fil
 sudo sysctl fs.inotify.max_user_instances=2280
 sudo sysctl fs.inotify.max_user_watches=1255360
 ```
+## Prerequisites
+* Kind https://kind.sigs.k8s.io/
+
 ## Create kind cluster
 ```bash
 cat <<EOF | kind create cluster --name=kubeflow --config=-
 kind: Cluster
 apiVersion: kind.x-k8s.io/v1alpha4
 nodes:
 - role: control-plane
-  image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865
+  image: kindest/node:v1.30.6@sha256:b6d08db72079ba5ae1f4a88a09025c0a904af3b52387643c285442afb05ab994
   kubeadmConfigPatches:
   - |
     kind: ClusterConfiguration
@@ -32,20 +35,20 @@ Instead of replacing your kubeconfig, we are going to set to a diff file
 kind get kubeconfig --name kubeflow > /tmp/kubeflow-config
 export KUBECONFIG=/tmp/kubeflow-config
 ```
-# docker
+## docker
 In order to by pass the docker limit issue while downloading the images. Let's use your credentials
 ```bash
-docker login
+docker login -u='...' -p='...' quay.io
 ```
 
 Upload the secret. The following command will return an error. You need to replace `to` with user `username`
 ```bash
 kubectl create secret generic regcred \
---from-file=.dockerconfigjson=/home/to/.docker/config.json \
+--from-file=.dockerconfigjson=$HOME/.docker/config.json \
 --type=kubernetes.io/dockerconfigjson
 ```
 
-# Test environment variables
+## Test environment variables
 Replace the `/path/to` in order to match the `data-science-pipelines-operator` folder
 ```bash
 export GIT_WORKSPACE=/path/to/data-science-pipelines-operator
@@ -58,7 +61,34 @@ Replace `username` with your quay user
 export REGISTRY_ADDRESS=quay.io/username
 ```
 
-# Run the test
+## Run the test
 ```bash
 sh .github/scripts/tests/tests.sh --kind
 ```
+
+# Debug a test using GoLand
+Let's say you wish to debug the `Should create a Pipeline Run` test.
+The first step is right click inside the method content and select the menu
+`Run 'TestIntegrationTestSuite'`. It will fail because you need to fill some parameters.
+Edit the configuration for `TestIntegrationTestSuite/TestPipelineSuccessfulRun/Should_create_a_Pipeline_Run in github.com/opendatahub-io/data-science-pipelines-operator/tests`
+````
+-k8sApiServerHost=https://127.0.0.1:39873
+-kubeconfig=/tmp/kubeflow-config
+-DSPANamespace=test-dspa
+-DSPAPath=/path/to/data-science-pipelines-operator/tests/resources/dspa-lite.yaml
+````
+## How to retrieve the parameters above
+* `k8sApiServerHost`: inspect the kubeconfig and retrieve the server URL from there
+* `kubeconfig`: the path where you stored the output of `kind get kubeconfig`
+* `DSPANamespace`: namespace
+* `DSPAPath`: full path for the dspa.yaml
+
+`Should create a Pipeline Run`, `DSPANamespace` and `DSPAPath` depends on the test scenario.
+
+If you wish to keep the resources, add `-skipCleanup=True` in the config above.
+
+## If you wish to rerun the test you need to delete the dspa
+```bash
+$ kubectl delete datasciencepipelinesapplications test-dspa -n test-dspa
+datasciencepipelinesapplication.datasciencepipelinesapplications.opendatahub.io "test-dspa" deleted
+```

config/internal/apiserver/sample-pipeline/sample-config.yaml.tmpl

@@ -11,7 +11,7 @@ data:
         [
           {
             "name": "[Demo] iris-training",
-            "description": "[source code](https://github.com/opendatahub-io/data-science-pipelines/tree/master/samples/iris-sklearn) A simple pipeline to demonstrate a basic ML Training workflow",
+            "description": "[source code](https://github.com/opendatahub-io/data-science-pipelines-operator/blob/main/docs/example_pipelines/iris/iris-pipeline.py) A simple pipeline to demonstrate a basic ML Training workflow",
             "file": "/samples/iris-pipeline-compiled.yaml"
           }
         ]

controllers/testdata/declarative/case_2/expected/created/sample-config.yaml.tmpl

@@ -12,7 +12,7 @@ data:
         [
           {
             "name": "[Demo] iris-training",
-            "description": "[source code](https://github.com/opendatahub-io/data-science-pipelines/tree/master/samples/iris-sklearn) A simple pipeline to demonstrate a basic ML Training workflow",
+            "description": "[source code](https://github.com/opendatahub-io/data-science-pipelines-operator/blob/main/docs/example_pipelines/iris/iris-pipeline.py) A simple pipeline to demonstrate a basic ML Training workflow",
             "file": "/samples/iris-pipeline-compiled.yaml"
           }
         ]

docs/example_pipelines/iris/iris-pipeline.py

@@ -108,15 +108,15 @@ def my_pipeline(
 
     normalize_dataset_task = normalize_dataset(
         input_iris_dataset=create_dataset_task.outputs['iris_dataset'],
-        standard_scaler=True)
+        standard_scaler=standard_scaler)
 
     train_model(
         normalized_iris_dataset=normalize_dataset_task
         .outputs['normalized_iris_dataset'],
         n_neighbors=neighbors)
 
-endpoint = 'http://ml-pipeline-ui-kubeflow.apps.rmartine.dev.datahub.redhat.com/'
 
-compiler.Compiler().compile(
-    pipeline_func=my_pipeline,
-    package_path= __file__.replace('.py', '-v2.yaml'))
+if __name__ == "__main__":
+    compiler.Compiler().compile(
+        pipeline_func=my_pipeline,
+        package_path= __file__.replace('.py', '.yaml'))

docs/example_pipelines/iris/iris-pipeline.yaml

@@ -3,8 +3,6 @@
 # Inputs:
 #    neighbors: int [Default: 3.0]
 #    standard_scaler: bool [Default: True]
-# Outputs:
-#    train-model-metrics: system.ClassificationMetrics
 components:
   comp-create-dataset:
     executorLabel: exec-create-dataset
@@ -66,7 +64,7 @@ deploymentSpec:
         - -c
         - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
           \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
-          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.7.0'\
+          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.10.0'\
           \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"'  &&\
           \  python3 -m pip install --quiet --no-warn-script-location 'pandas==2.2.0'\
           \ && \"$0\" \"$@\"\n"
@@ -99,7 +97,7 @@ deploymentSpec:
         - -c
         - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
           \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
-          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.7.0'\
+          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.10.0'\
           \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"'  &&\
           \  python3 -m pip install --quiet --no-warn-script-location 'pandas==2.2.0'\
           \ 'scikit-learn==1.4.0' && \"$0\" \"$@\"\n"
@@ -136,7 +134,7 @@ deploymentSpec:
         - -c
         - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
           \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
-          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.7.0'\
+          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.10.0'\
           \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"'  &&\
           \  python3 -m pip install --quiet --no-warn-script-location 'pandas==2.2.0'\
           \ 'scikit-learn==1.4.0' && \"$0\" \"$@\"\n"
@@ -173,12 +171,6 @@ pipelineInfo:
   name: iris-training-pipeline
 root:
   dag:
-    outputs:
-      artifacts:
-        train-model-metrics:
-          artifactSelectors:
-          - outputArtifactKey: metrics
-            producerSubtask: train-model
     tasks:
       create-dataset:
         cachingOptions:
@@ -202,8 +194,7 @@ root:
                 producerTask: create-dataset
           parameters:
             standard_scaler:
-              runtimeValue:
-                constant: true
+              componentInputParameter: standard_scaler
         taskInfo:
           name: normalize-dataset
       train-model:
@@ -234,11 +225,5 @@ root:
         defaultValue: true
         isOptional: true
         parameterType: BOOLEAN
-  outputDefinitions:
-    artifacts:
-      train-model-metrics:
-        artifactType:
-          schemaTitle: system.ClassificationMetrics
-          schemaVersion: 0.0.1
 schemaVersion: 2.1.0
-sdkVersion: kfp-2.7.0
+sdkVersion: kfp-2.10.0

go.mod

@@ -5,95 +5,106 @@ go 1.21
 require (
 	github.com/anthhub/forwarder v1.1.0
 	github.com/fsnotify/fsnotify v1.6.0
-	github.com/go-logr/logr v1.2.4
+	github.com/go-logr/logr v1.3.0
 	github.com/go-sql-driver/mysql v1.7.1
 	github.com/go-test/deep v1.1.0
-	github.com/golang/glog v1.1.0
+	github.com/golang/glog v1.2.0
+	github.com/kubeflow/pipelines v0.0.0-20241115174720-634aadf808cd
 	github.com/manifestival/controller-runtime-client v0.4.0
 	github.com/manifestival/manifestival v0.7.2
 	github.com/minio/minio-go/v7 v7.0.56
 	github.com/openshift/api v0.0.0-20231118005202-0f638a8a4705
-	github.com/prometheus/client_golang v1.15.1
-	github.com/spf13/viper v1.8.1
-	github.com/stretchr/testify v1.8.3
-	go.uber.org/zap v1.24.0
+	github.com/prometheus/client_golang v1.16.0
+	github.com/spf13/viper v1.15.0
+	github.com/stretchr/testify v1.8.4
+	go.uber.org/zap v1.26.0
 	k8s.io/api v0.27.2
-	k8s.io/apimachinery v0.27.2
+	k8s.io/apimachinery v0.27.3
 	k8s.io/client-go v0.27.2
 	sigs.k8s.io/controller-runtime v0.15.0
 )
 
 require (
+	github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
-	github.com/emicklei/go-restful/v3 v3.9.0 // indirect
-	github.com/evanphx/json-patch v4.12.0+incompatible // indirect
+	github.com/emicklei/go-restful/v3 v3.10.2 // indirect
+	github.com/evanphx/json-patch v5.8.0+incompatible // indirect
 	github.com/evanphx/json-patch/v5 v5.6.0 // indirect
 	github.com/go-errors/errors v1.0.1 // indirect
 	github.com/go-logr/zapr v1.2.4 // indirect
-	github.com/go-openapi/jsonpointer v0.19.6 // indirect
-	github.com/go-openapi/jsonreference v0.20.1 // indirect
-	github.com/go-openapi/swag v0.22.3 // indirect
+	github.com/go-openapi/analysis v0.20.1 // indirect
+	github.com/go-openapi/errors v0.20.2 // indirect
+	github.com/go-openapi/jsonpointer v0.20.2 // indirect
+	github.com/go-openapi/jsonreference v0.20.4 // indirect
+	github.com/go-openapi/loads v0.21.0 // indirect
+	github.com/go-openapi/runtime v0.21.1 // indirect
+	github.com/go-openapi/spec v0.20.4 // indirect
+	github.com/go-openapi/strfmt v0.21.1 // indirect
+	github.com/go-openapi/swag v0.22.6 // indirect
+	github.com/go-openapi/validate v0.20.3 // indirect
+	github.com/go-stack/stack v1.8.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
 	github.com/google/btree v1.0.1 // indirect
-	github.com/google/gnostic v0.5.7-v3refs // indirect
-	github.com/google/go-cmp v0.5.9 // indirect
-	github.com/google/gofuzz v1.1.0 // indirect
+	github.com/google/gnostic v0.6.9 // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
+	github.com/google/gofuzz v1.2.0 // indirect
 	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
-	github.com/google/uuid v1.3.0 // indirect
+	github.com/google/uuid v1.5.0 // indirect
 	github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect
 	github.com/hashicorp/hcl v1.0.0 // indirect
-	github.com/imdario/mergo v0.3.12 // indirect
+	github.com/imdario/mergo v0.3.13 // indirect
 	github.com/inconshreveable/mousetrap v1.0.1 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
-	github.com/klauspost/compress v1.16.5 // indirect
-	github.com/klauspost/cpuid/v2 v2.2.4 // indirect
+	github.com/klauspost/compress v1.16.7 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.5 // indirect
 	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
-	github.com/magiconair/properties v1.8.5 // indirect
+	github.com/magiconair/properties v1.8.7 // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
 	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
 	github.com/minio/md5-simd v1.1.2 // indirect
 	github.com/minio/sha256-simd v1.0.1 // indirect
-	github.com/mitchellh/mapstructure v1.4.1 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/moby/spdystream v0.2.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/pelletier/go-toml v1.9.3 // indirect
+	github.com/oklog/ulid v1.3.1 // indirect
+	github.com/pelletier/go-toml/v2 v2.0.8 // indirect
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/prometheus/client_model v0.4.0 // indirect
 	github.com/prometheus/common v0.42.0 // indirect
-	github.com/prometheus/procfs v0.9.0 // indirect
+	github.com/prometheus/procfs v0.10.1 // indirect
 	github.com/rs/xid v1.5.0 // indirect
-	github.com/sirupsen/logrus v1.9.2 // indirect
-	github.com/spf13/afero v1.9.2 // indirect
-	github.com/spf13/cast v1.3.1 // indirect
+	github.com/sirupsen/logrus v1.9.3 // indirect
+	github.com/spf13/afero v1.9.3 // indirect
+	github.com/spf13/cast v1.5.0 // indirect
 	github.com/spf13/cobra v1.6.0 // indirect
 	github.com/spf13/jwalterweatherman v1.1.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	github.com/subosito/gotenv v1.2.0 // indirect
+	github.com/subosito/gotenv v1.4.2 // indirect
 	github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca // indirect
+	go.mongodb.org/mongo-driver v1.7.5 // indirect
 	go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 // indirect
-	go.uber.org/atomic v1.7.0 // indirect
-	go.uber.org/multierr v1.6.0 // indirect
-	golang.org/x/crypto v0.21.0 // indirect
+	go.uber.org/multierr v1.10.0 // indirect
+	golang.org/x/crypto v0.22.0 // indirect
 	golang.org/x/net v0.23.0 // indirect
-	golang.org/x/oauth2 v0.7.0 // indirect
-	golang.org/x/sync v0.2.0 // indirect
-	golang.org/x/sys v0.18.0 // indirect
-	golang.org/x/term v0.18.0 // indirect
+	golang.org/x/oauth2 v0.16.0 // indirect
+	golang.org/x/sync v0.6.0 // indirect
+	golang.org/x/sys v0.19.0 // indirect
+	golang.org/x/term v0.19.0 // indirect
 	golang.org/x/text v0.14.0 // indirect
-	golang.org/x/time v0.3.0 // indirect
-	gomodules.xyz/jsonpatch/v2 v2.3.0 // indirect
-	google.golang.org/appengine v1.6.7 // indirect
+	golang.org/x/time v0.5.0 // indirect
+	gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
+	google.golang.org/appengine v1.6.8 // indirect
 	google.golang.org/protobuf v1.35.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
@@ -102,9 +113,9 @@ require (
 	k8s.io/apiextensions-apiserver v0.27.2 // indirect
 	k8s.io/cli-runtime v0.24.17 // indirect
 	k8s.io/component-base v0.27.2 // indirect
-	k8s.io/klog/v2 v2.90.1 // indirect
-	k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f // indirect
-	k8s.io/utils v0.0.0-20230209194617-a36077c30491 // indirect
+	k8s.io/klog/v2 v2.100.1 // indirect
+	k8s.io/kube-openapi v0.0.0-20230515203736-54b630e78af5 // indirect
+	k8s.io/utils v0.0.0-20230505201702-9f6742963106 // indirect
 	sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
 	sigs.k8s.io/kustomize/api v0.11.4 // indirect
 	sigs.k8s.io/kustomize/kyaml v0.13.6 // indirect