Support for pushing to AWS ECR

[Niksko]

Is pushing to ECR supported? The docs say that images can be pushed to OCI compatible registries, and ECR is OCI compatible. When I try and tag and push an image to an ECR registry that I am authenticated to, I get an error with status 405 Method Not Allowed. Below is a verbose run of the wame push with the name of my registry redacted

wasme push <my-registry>:latest -c ~/.docker/config.json -v
INFO[0000] Pushing image <my-registry>:latest
WARN[0000] encountered unknown type application/vnd.module.wasm.content.layer.v1+wasm; children may not be fetched
DEBU[0000] push                                          digest="sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30" mediatype=application/vnd.module.wasm.content.layer.v1+wasm size=12527
WARN[0000] encountered unknown type application/vnd.module.wasm.config.v1+json; children may not be fetched
DEBU[0000] push                                          digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json size=166
WARN[0000] encountered unknown type application/vnd.module.wasm.config.v1+json; children may not be fetched
DEBU[0000] push                                          digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json size=166
WARN[0000] reference for unknown type: application/vnd.module.wasm.config.v1+json  digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json size=166
WARN[0000] reference for unknown type: application/vnd.module.wasm.content.layer.v1+wasm  digest="sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30" mediatype=application/vnd.module.wasm.content.layer.v1+wasm size=12527
DEBU[0000] checking and pushing to                       digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json size=166 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770"
WARN[0000] reference for unknown type: application/vnd.module.wasm.config.v1+json  digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json size=166
DEBU[0000] checking and pushing to                       digest="sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30" mediatype=application/vnd.module.wasm.content.layer.v1+wasm size=12527 url="https:/<my-registry>/v2/ns-wasm-testing/blobs/sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30"
DEBU[0000] do request                                    digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json request.header.accept="application/vnd.module.wasm.config.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD size=166 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770"
DEBU[0000] do request                                    digest="sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30" mediatype=application/vnd.module.wasm.content.layer.v1+wasm request.header.accept="application/vnd.module.wasm.content.layer.v1+wasm, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD size=12527 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30"
DEBU[0000] checking and pushing to                       digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json size=166 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770"
DEBU[0000] do request                                    digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json request.header.accept="application/vnd.module.wasm.config.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD size=166 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770"
DEBU[0000] fetch response received                       digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json response.header.connection=keep-alive response.header.content-length=166 response.header.content-type=application/vnd.docker.container.image.v1+json response.header.date="Fri, 10 Jul 2020 03:37:45 GMT" response.header.docker-content-digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" response.header.docker-distribution-api-version=registry/2.0 response.status="200 OK" size=166 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770"
DEBU[0000] fetch response received                       digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" mediatype=application/vnd.module.wasm.config.v1+json response.header.connection=keep-alive response.header.content-length=166 response.header.content-type=application/vnd.docker.container.image.v1+json response.header.date="Fri, 10 Jul 2020 03:37:45 GMT" response.header.docker-content-digest="sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770" response.header.docker-distribution-api-version=registry/2.0 response.status="200 OK" size=166 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:766ecd72b4cc55952c00650f1a77b5f138207e76b5b2a241352ef245203af770"
DEBU[0000] fetch response received                       digest="sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30" mediatype=application/vnd.module.wasm.content.layer.v1+wasm response.header.connection=keep-alive response.header.content-length=12527 response.header.content-type=application/vnd.docker.container.image.v1+json response.header.date="Fri, 10 Jul 2020 03:37:45 GMT" response.header.docker-content-digest="sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30" response.header.docker-distribution-api-version=registry/2.0 response.status="200 OK" size=12527 url="https://<my-registry>/v2/ns-wasm-testing/blobs/sha256:8b74e9b0bbc5ff674c49cde904669a775a939b4d8f7f72aba88c184d527dfc30"
DEBU[0000] push                                          digest="sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702" mediatype=application/vnd.oci.image.manifest.v1+json size=893
DEBU[0000] checking and pushing to                       digest="sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702" mediatype=application/vnd.oci.image.manifest.v1+json size=893 url="https://<my-registry>/v2/ns-wasm-testing/manifests/latest"
DEBU[0000] do request                                    digest="sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD size=893 url="https://<my-registry>/v2/ns-wasm-testing/manifests/latest"
DEBU[0000] fetch response received                       digest="sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702" mediatype=application/vnd.oci.image.manifest.v1+json response.header.connection=keep-alive response.header.content-length=79 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 10 Jul 2020 03:37:45 GMT" response.header.docker-distribution-api-version=registry/2.0 response.status="404 Not Found" size=893 url="https://<my-registry>/v2/ns-wasm-testing/manifests/latest"
DEBU[0000] do request                                    digest="sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702" mediatype=application/vnd.oci.image.manifest.v1+json request.header.content-type=application/vnd.oci.image.manifest.v1+json request.header.user-agent=containerd/1.3.0+unknown request.method=PUT size=893 url="https://<my-registry>/v2/ns-wasm-testing/manifests/latest"
DEBU[0000] fetch response received                       digest="sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702" mediatype=application/vnd.oci.image.manifest.v1+json response.header.connection=keep-alive response.header.content-length=137 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 10 Jul 2020 03:37:45 GMT" response.header.docker-distribution-api-version=registry/2.0 response.status="405 Method Not Allowed" size=893 url="https://<my-registry>/v2/ns-wasm-testing/manifests/latest"
Error: All attempts fail:
#1: oras push failed: failed commit on ref "manifest-sha256:9d4b4660f71f2714cc71e2b844e9b8460def21f6d76259140e70d447ccc7c702": unexpected status: 405 Method Not Allowed

I've also tried passing username and password via the -u and -p flags

[zortness]

I am having similar issues that I'll detail here to possibly give some additional info for the devs. All of these interactions are with a private ECR repository, which I have used with the docker command line utility without issue.

Login attempts to ECR give a 404.

$ wasme login -s <aws_ecr_url> -u AWS -p <really_long_pw>
Error: non-200 status code: 404L: 404 Not Found

Using an OCI image with a path-prefixed name for organizational purposes, eg myfilters, will error with a 404.

$ wasme push <aws_ecr_url>/myfilters/mytestfilter:0.0.1 -u AWS -p <really_long_pw>
#1: oras push failed: unexpected response: 404 Not Found

This is not ideal, as larger organizations tend to organize images by product, team, or department in the above way. Having folders is better supported in products like JFrog Artifactory, but still functions in ECR, so it is likely wasme running into an issue in this case.

Eliminating the prefix path and creating a new repo location with just the root name, and rebuilding the filter to match, we then get the above 405 error.

$ wasme push <aws_ecr_url>/mytestfilter:0.0.1 -u AWS -p <really_long_pw>
Error: All attempts fail:
#1: oras push failed: failed commit on ref "manifest-sha256:<hash>": unexpected status: 405 Method Not Allowed

I have also tried to extract more information using the -v flag, to no avail.

I will make another ticket, but it would also be nice if these commands could take the password from stdin, similar to the docker and aws tools. This would also coincide nicely with the AWS examples for using docker.

[zortness]

Some additional info. I was able to log into my ECR repository with ORAS, oras login, so it's possible that wasme (which is wrapping oras at the moment) is not providing some configuration to the library, or an upgrade may alleviate it.

Using oras push, I get an equivalent/identical error to the wasme push errors above.

I also played around with wasm-to-oci, which also wraps the oras go library. I was able to log in with the raw oras login [...] sequence, since wasm-to-oci does not provide a login method. I figured out that this works, as the oras library seems to store the credential cache in a well known location. I also figured out that this appears to work (or to at least get as far as the above 405 error) in concert with wasme push [...] (without credentials provided).

The biggest difference, however, is that wasm-to-oci can actually push successfully to ECR. I don't believe the format it creates is compatible with what Envoy/Gloo expect in this case, but it is able to push images up using the oras library. This again seems to reinforce that it's a matter of configuration parameters sent to the oras library before initiating a push that determines success or failure in this case.

[zortness]

For completeness, I was able to push a wasme built binary using oras up to an ECR repository.

After creating and building the filter with wasme, navigate to ~/.wasme/store/<some_hash>. You should have a filter.wasm and runtime-config.json file in there already (along with some others).

To use oras properly, we need to create an annotations.json file with the metadata for the OCI image. For test purposes, this is basically static, but the abi versions can be adjusted from what you find in runtime-config.json:

{
	"$manifest": {
		"module.wasm.runtime/abi_version": "v0-541b2c1155fffb15ccde92b8324f3e38f7339ba6,v0-097b7f2e4cc1fb490cc1943d0d633655ac3c522f,v0-4689a30309abf31aee9ae36e73d34b1bb182685f,v0.2.1",
		"module.wasm.runtime/type": "envoy_proxy"
	}
}

Then we log oras into our ECR repo using the aws command line tool.
aws ecr get-login-password --region <region> | oras login -u AWS --password-stdin <your_account_id>.ecr.<region>.amazonaws.com

Finally, push the repo:

oras push --manifest-annotations annotations.json \
    <your_account_id>.dkr.ecr.<region>.amazonaws.com/<filter_name>:<version> \
    runtime-config.json:application/vnd.module.wasm.config.v1+json \
    filter.wasm:application/vnd.module.wasm.content.layer.v1+wasm

This assumes you have a repository created in ECR that matches what you put above for <filter_name>.