Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish images via pulse rather than dockerhub. #378

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion taskcluster/ci/docker-image/kind.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,3 @@ jobs:
definition: node
args:
NODE_VERSION: "14"
skopeo: {}
37 changes: 9 additions & 28 deletions taskcluster/ci/push-image/kind.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,43 +9,24 @@ kind-dependencies:
- k8s-image

transforms:
- shipit_taskgraph.transforms.deploy_secret:transforms
- shipit_taskgraph.transforms.docker_push:transforms
- taskgraph.transforms.job:transforms
- shipit_taskgraph.transforms.cloudops_deploy:transforms
- taskgraph.transforms.task:transforms

job-defaults:
worker-type: b-linux
worker:
taskcluster-proxy: true
docker-image: {in-tree: skopeo}
max-run-time: 3600
dependencies:
tests-js: tests-frontend-node-14
tests-api: tests-api-python-38
run:
using: run-task
checkout: false
cache-dotcache: false
command:
- /usr/local/bin/push_image.sh
fetches:
k8s-image:
- artifact: image.tar.zst
extract: false
tests-js: tests-frontend-node-14
tests-api: tests-api-python-38
project: shipitapi
image-task-id: {task-reference: "<k8s-image>"}

jobs:
shipit-admin:
description: "Push to shipit-admin repository."
dependencies:
k8s-image: build-docker-image-shipit-admin
worker:
env:
DOCKER_REPO: docker.io/mozilla/releng-shipit-admin
k8s-image: build-docker-image-shipit-admin
variant: admin
shipit-public:
description: "Push to shipit-public repository."
dependencies:
k8s-image: build-docker-image-shipit-public
worker:
env:
DOCKER_REPO: docker.io/mozilla/releng-shipit-public
k8s-image: build-docker-image-shipit-public
variant: public
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might be able to parameterize this in the k8s-image task, and use single_dep to create a push-image task per k8s-image task. I don't think that's a massive win with 2 k8s-image tasks, so I think this is a perfectly fine implementation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was definitely thinking more about the shape of the resulting tasks, than taskgraph details. Your idea of encoding it in the build task makes sense. If we do that, we could also use that as a check in cloudops-infra-deploylib, that the image was intended to be used for where the hook is telling it to use it. I suspect that would make secops happy; I know that when @ajvb was reviewing the hg stuff for ci-admin, having checks for the info received via pulse was desirable.

60 changes: 0 additions & 60 deletions taskcluster/docker/skopeo/Dockerfile

This file was deleted.

14 changes: 0 additions & 14 deletions taskcluster/docker/skopeo/policy.json

This file was deleted.

46 changes: 0 additions & 46 deletions taskcluster/docker/skopeo/push_image.sh

This file was deleted.

65 changes: 65 additions & 0 deletions taskcluster/shipit_taskgraph/transforms/cloudops_deploy.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# -*- coding: utf-8 -*-

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

"""
Deployment secret related environment variables
"""

from __future__ import absolute_import, print_function, unicode_literals

from six import text_type
from voluptuous import Required, Optional

from taskgraph.transforms.base import TransformSequence
from taskgraph.transforms.task import task_description_schema
from taskgraph.util.schema import Schema, taskref_or_string

transforms = TransformSequence()

deploy_schema = Schema(
{
Required('name'): text_type,
Required('job-from'): task_description_schema['job-from'],
Required("description"): task_description_schema['job-from'],
Required("project"): text_type,
Required("image-task-id"): taskref_or_string,
Optional("variant"): text_type,
Required("dependencies"): {text_type: text_type},
}
)

transforms.add_validate(deploy_schema)


CLOUDOPS_DEPLOY_ROUTE = "cloudops.deploy.v1.{project}.{environment}"


@transforms.add
def build_task(config, jobs):
"""Set the environment variables for the push to docker hub task."""
for job in jobs:
# skip the task if we don't need to push the image
if not config.params.get("deploy"):
continue

route = CLOUDOPS_DEPLOY_ROUTE.format(
project=job["project"], environment=config.params["deployment_branch"],
)
deploy = {
"image-task-id": job["image-task-id"],
}
if "variant" in job:
deploy["variant"] = job["variant"]

task = {
'name': job['name'],
"description": job["description"],
"worker-type": "succeed",
"dependencies": job["dependencies"],
"extra": {"cloudops-deploy": deploy},
"routes": [route],
}
yield task
42 changes: 0 additions & 42 deletions taskcluster/shipit_taskgraph/transforms/docker_push.py

This file was deleted.