Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(CI): Use the correct image registry for replacements in integration tests #11564

Merged
merged 4 commits into from
Jan 29, 2025
Merged

fix(CI): Use the correct image registry for replacements in integration tests #11564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0650ede
Use the correct image registry for replacements in integration tests
mprahl Jan 28, 2025
10d311a
Print the pod logs when the pods fail to start in integration tests
mprahl Jan 28, 2025
4930dcb
Fix the sample compilation in the API server container build
mprahl Jan 29, 2025
f075efa
Show the output when building the container images in CI
mprahl Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/resources/manifests/argo/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ resources:
- ../../../../manifests/kustomize/env/platform-agnostic

images:
- name: gcr.io/ml-pipeline/api-server
- name: ghcr.io/kubeflow/kfp-api-server
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests are failing due to this change, I falsely assumed things were working but the old images were being used 🤦🏾 , so I'm guessing that's why the server is crashing

I think the issue is probably here

changing .[].file to .pipelines[].file should fix it, since the format of the samples json is now different

if it still fails let me know, this one probably falls on me to fix

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HumairAK yup, my other PR adds logging to the failing pods (I'll shift that over here) and the log message is this:

I0129 02:20:31.147561      13 client_manager.go:216] Client manager initialized successfully
F0129 02:20:31.147947      13 main.go:82] Failed to load samples. Err: failed to load sample [Tutorial] Data passing in python components. Error: open /samples/tutorials/Data passing in python components/Data passing in python components - Files.py.yaml: no such file or directory

goroutine 1 [running]:
github.com/golang/glog.Fatalf(...)
	/go/pkg/mod/github.com/golang/[email protected]/glog.go:690
main.main()
	/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/main.go:82 +0x2a9

SIGABRT: abort
PC=0x4098ae m=0 sigcode=18446744073709551610

newName: kind-registry:5000/apiserver
newTag: latest
- name: gcr.io/ml-pipeline/persistenceagent
- name: ghcr.io/kubeflow/kfp-persistence-agent
newName: kind-registry:5000/persistenceagent
newTag: latest
- name: gcr.io/ml-pipeline/scheduledworkflow
- name: ghcr.io/kubeflow/kfp-scheduled-workflow-controller
newName: kind-registry:5000/scheduledworkflow
newTag: latest

Expand Down
6 changes: 3 additions & 3 deletions .github/resources/manifests/tekton/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ resources:
# when application is deleted.

images:
- name: gcr.io/ml-pipeline/api-server
- name: ghcr.io/kubeflow/kfp-api-server
newName: kind-registry:5000/apiserver
newTag: latest
- name: gcr.io/ml-pipeline/persistenceagent
- name: ghcr.io/kubeflow/kfp-persistence-agent
newName: kind-registry:5000/persistenceagent
newTag: latest
- name: gcr.io/ml-pipeline/scheduledworkflow
- name: ghcr.io/kubeflow/kfp-scheduled-workflow-controller
newName: kind-registry:5000/scheduledworkflow
newTag: latest
- name: '*/aipipeline/tekton-exithandler-controller'
Expand Down
10 changes: 5 additions & 5 deletions .github/resources/scripts/build-images.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,35 +25,35 @@ EXIT_CODE=0

docker system prune -a -f

docker build -q -t "${REGISTRY}/apiserver:${TAG}" -f backend/Dockerfile . && docker push "${REGISTRY}/apiserver:${TAG}" || EXIT_CODE=$?
docker build --progress=plain -t "${REGISTRY}/apiserver:${TAG}" -f backend/Dockerfile . && docker push "${REGISTRY}/apiserver:${TAG}" || EXIT_CODE=$?
if [[ $EXIT_CODE -ne 0 ]]
then
echo "Failed to build apiserver image."
exit $EXIT_CODE
fi

docker build -q -t "${REGISTRY}/persistenceagent:${TAG}" -f backend/Dockerfile.persistenceagent . && docker push "${REGISTRY}/persistenceagent:${TAG}" || EXIT_CODE=$?
docker build --progress=plain -t "${REGISTRY}/persistenceagent:${TAG}" -f backend/Dockerfile.persistenceagent . && docker push "${REGISTRY}/persistenceagent:${TAG}" || EXIT_CODE=$?
if [[ $EXIT_CODE -ne 0 ]]
then
echo "Failed to build persistenceagent image."
exit $EXIT_CODE
fi

docker build -q -t "${REGISTRY}/scheduledworkflow:${TAG}" -f backend/Dockerfile.scheduledworkflow . && docker push "${REGISTRY}/scheduledworkflow:${TAG}" || EXIT_CODE=$?
docker build --progress=plain -t "${REGISTRY}/scheduledworkflow:${TAG}" -f backend/Dockerfile.scheduledworkflow . && docker push "${REGISTRY}/scheduledworkflow:${TAG}" || EXIT_CODE=$?
if [[ $EXIT_CODE -ne 0 ]]
then
echo "Failed to build scheduledworkflow image."
exit $EXIT_CODE
fi

docker build -q -t "${REGISTRY}/driver:${TAG}" -f backend/Dockerfile.driver . && docker push "${REGISTRY}/driver:${TAG}" || EXIT_CODE=$?
docker build --progress=plain -t "${REGISTRY}/driver:${TAG}" -f backend/Dockerfile.driver . && docker push "${REGISTRY}/driver:${TAG}" || EXIT_CODE=$?
if [[ $EXIT_CODE -ne 0 ]]
then
echo "Failed to build driver image."
exit $EXIT_CODE
fi

docker build -q -t "${REGISTRY}/launcher:${TAG}" -f backend/Dockerfile.launcher . && docker push "${REGISTRY}/launcher:${TAG}" || EXIT_CODE=$?
docker build --progress=plain -t "${REGISTRY}/launcher:${TAG}" -f backend/Dockerfile.launcher . && docker push "${REGISTRY}/launcher:${TAG}" || EXIT_CODE=$?
if [[ $EXIT_CODE -ne 0 ]]
then
echo "Failed to build launcher image."
Expand Down
13 changes: 13 additions & 0 deletions .github/resources/scripts/kfp-readiness/wait_for_pods.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@
config.load_kube_config()
v1 = client.CoreV1Api()

def log_pods():
pods = v1.list_namespaced_pod(namespace=namespace)

for pod in pods.items:
try:
logging.info(
f"---- Pod {namespace}/{pod.metadata.name} logs ----\n"
+ v1.read_namespaced_pod_log(pod.metadata.name, namespace)
)
except client.exceptions.ApiException:
continue

def get_pod_statuses():
pods = v1.list_namespaced_pod(namespace=namespace)
Expand Down Expand Up @@ -74,6 +85,8 @@ def check_pods(calm_time=10, timeout=600, retries_after_ready=5):
logging.info(f"Pods are still stabilizing. Retrying in {calm_time} seconds...")
time.sleep(calm_time)
else:
log_pods()

raise Exception("Pods did not stabilize within the timeout period.")

logging.info("Final pod statuses:")
Expand Down
4 changes: 2 additions & 2 deletions backend/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ COPY backend/src/apiserver/config/sample_config.json /samples/
# Compiling the preloaded samples.
# The default image is replaced with the GCR-hosted python image.
RUN set -e; \
< /samples/sample_config.json jq .[].file --raw-output | while read pipeline_yaml; do \
< /samples/sample_config.json jq ".pipelines[].file" --raw-output | while read pipeline_yaml; do \
pipeline_py="${pipeline_yaml%.yaml}"; \
python3 "$pipeline_py"; \
echo "Compiling: \"$pipeline_py\"" && python3 "$pipeline_py" && echo -n "Output: " && ls "$pipeline_py.yaml"; \
done

# 3. Start api web server
Expand Down