Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFE] provider build container image #74

Open
usrbinkat opened this issue Oct 12, 2023 · 0 comments
Open

[RFE] provider build container image #74

usrbinkat opened this issue Oct 12, 2023 · 0 comments
Labels
kind/enhancement Improvements or new features

Comments

@usrbinkat
Copy link
Contributor

Problem

Friction in developing native providers is a blocker to growing beyond our 14 native providers in the registry today.

Affected area/feature

  • Native Provider Development UX

Recommendation

Create a multi-arch containerized provider build environment to satisfy the many dependencies required to build multi language providers.

Future Opportunities

Adding a containerized build image simplifies future enhancements including:

  • daily/weekly boilerplate repository ci testing
  • simpler native provider developer docs
  • improved provider build reliability

Additional Information

LOGS: Click To Expand 
~/pulumi-provider-boilerplate ❯ docker build -t provider -f docker/Dockerfile .
[+] Building 1.1s (14/14) FINISHED
 => [internal] load .dockerignore                                                                                                                                                                                                                                                         0.0s
 => => transferring context: 653B                                                                                                                                                                                                                                                         0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 4.91kB                                                                                                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                                                                                                                                           1.1s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                                                                                                         0.0s
 => => transferring context: 742B                                                                                                                                                                                                                                                         0.0s
 => [1/8] FROM docker.io/library/ubuntu:22.04@sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea                                                                                                                                                                     0.0s
 => CACHED [2/8] ADD docker/rootfs /                                                                                                                                                                                                                                                      0.0s
 => CACHED [3/8] RUN set -ex     && apt-get update     && apt-get install git curl gnupg python3 python3-pip dotnet-sdk-7.0 ca-certificates build-essential dotnet-runtime-7.0      && update-alternatives --install         /usr/bin/python         python         /usr/bin/python3 1    0.0s
 => CACHED [4/8] RUN set -ex     && export arch=$(uname -m | awk '{print ($1 == "x86_64") ? "amd64" : ($1 == "aarch64" ? "arm64" : "unknown")}')     && export goversion="$(curl -s https://go.dev/dl/?mode=json         | awk -F'[":go]' '/  "version"/{print $8}'         | head -n1)"  0.0s
 => CACHED [5/8] RUN set -ex     && export NODE_MAJOR=20     && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key         | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg     && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.co  0.0s
 => CACHED [6/8] RUN set -ex     && export arch=$(uname -m | awk '{print ($1 == "x86_64") ? "x64" : ($1 == "aarch64" ? "arm64" : "unknown")}')     && export urlPulumiRelease="https://api.github.com/repos/pulumi/pulumi/releases/latest"     && export urlPulumiVersion=$(curl -s ${ur  0.0s
 => CACHED [7/8] RUN set -ex     && export arch=$(uname -m | awk '{print ($1 == "x86_64") ? "amd64" : ($1 == "aarch64" ? "arm64" : "unknown")}')     && export urlPulumiRelease="https://api.github.com/repos/pulumi/pulumictl/releases/latest"     && export urlPulumiVersion=$(curl -s  0.0s
 => CACHED [8/8] WORKDIR /provider                                                                                                                                                                                                                                                        0.0s
 => exporting to image                                                                                                                                                                                                                                                                    0.0s
 => => exporting layers                                                                                                                                                                                                                                                                   0.0s
 => => writing image sha256:2375457dd3a19fbf32aa59226bfae190a621c4b15fa5c48fed8c659f14e6c7e9                                                                                                                                                                                              0.0s
 => => naming to docker.io/library/provider:latest                                                                                                                                                                                                                                        0.0s

~/pulumi-provider-boilerplate ❯ docker run -it --rm -v $PWD:/provider provider
(cd provider && go build -o /provider/bin/pulumi-resource-xyz -ldflags "-X github.com/pulumi/pulumi-xyz/provider.Version=0.0.1-alpha.1697065379+97813b07.dirty" github.com/pulumi/pulumi-xyz/provider/cmd/pulumi-resource-xyz)
go: downloading github.com/pulumi/pulumi-go-provider v0.11.1
go: downloading github.com/pulumi/pulumi/sdk/v3 v3.79.0
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/pulumi/pulumi/pkg/v3 v3.79.0
go: downloading google.golang.org/grpc v1.57.0
go: downloading google.golang.org/protobuf v1.31.0
go: downloading github.com/hashicorp/errwrap v1.1.0
go: downloading github.com/golang/protobuf v1.5.3
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/blang/semver v3.5.1+incompatible
go: downloading golang.org/x/net v0.14.0
go: downloading lukechampine.com/frand v1.4.2
go: downloading github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645
go: downloading github.com/hashicorp/hcl/v2 v2.18.0
go: downloading github.com/opentracing/opentracing-go v1.2.0
go: downloading github.com/rivo/uniseg v0.4.4
go: downloading github.com/spf13/cobra v1.7.0
go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible
go: downloading golang.org/x/term v0.11.0
go: downloading sourcegraph.com/sourcegraph/appdash v0.0.0-20211028080628-e2786a622600
go: downloading github.com/golang/glog v1.1.2
go: downloading github.com/pkg/term v1.1.0
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d
go: downloading github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/crypto v0.12.0
go: downloading github.com/cheggaaa/pb v1.0.29
go: downloading github.com/djherbis/times v1.5.0
go: downloading github.com/go-git/go-git/v5 v5.8.1
go: downloading github.com/rogpeppe/go-internal v1.11.0
go: downloading github.com/santhosh-tekuri/jsonschema/v5 v5.3.1
go: downloading github.com/texttheater/golang-levenshtein v1.0.1
go: downloading github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7
go: downloading github.com/agext/levenshtein v1.2.3
go: downloading github.com/apparentlymart/go-textseg/v15 v15.0.0
go: downloading github.com/mitchellh/go-wordwrap v1.0.1
go: downloading github.com/zclconf/go-cty v1.14.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading golang.org/x/sys v0.11.0
go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible
go: downloading go.uber.org/atomic v1.11.0
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/opentracing/basictracer-go v1.1.0
go: downloading github.com/mattn/go-runewidth v0.0.15
go: downloading github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06
go: downloading github.com/kevinburke/ssh_config v1.2.0
go: downloading dario.cat/mergo v1.0.0
go: downloading github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371
go: downloading github.com/go-git/go-billy/v5 v5.4.1
go: downloading github.com/sergi/go-diff v1.3.1
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/skeema/knownhosts v1.2.0
go: downloading github.com/xanzy/ssh-agent v0.3.3
go: downloading github.com/pjbgf/sha1cd v0.3.0
go: downloading github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376
go: downloading github.com/emirpasic/gods v1.18.1
go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
go: downloading golang.org/x/text v0.12.0
go: downloading github.com/cloudflare/circl v1.3.3
go: downloading gopkg.in/warnings.v0 v0.1.2
go: downloading github.com/pgavlin/goldmark v1.1.33-0.20200616210433-b5eb04559386
go: downloading github.com/natefinch/atomic v1.0.1
go: downloading github.com/edsrzf/mmap-go v1.1.0
go: downloading github.com/segmentio/encoding v0.3.6
go: downloading github.com/segmentio/asm v1.2.0
rm -rf sdk/dotnet
pulumi package gen-sdk /provider/bin/pulumi-resource-xyz --language dotnet
cd sdk/dotnet/&& \
        echo "0.0.1-alpha.1697065379+97813b07.dirty" >version.txt && \
        dotnet build /p:Version=0.0.1-alpha.1697065379+97813b07.dirty

Welcome to .NET 7.0!
---------------------
SDK Version: 7.0.112

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
-----------------------------------------------------------------------------
MSBuild version 17.4.8+6918b863a for .NET
  Determining projects to restore...
  Restored /provider/sdk/dotnet/Pulumi.Xyz.csproj (in 3.81 sec).
  Pulumi.Xyz -> /provider/sdk/dotnet/bin/Debug/net6.0/Pulumi.Xyz.dll
  Successfully created package '/provider/sdk/dotnet/bin/Debug/Pulumi.Xyz.0.0.1-alpha.1697065379.nupkg'.

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:06.02
rm -rf sdk/go
pulumi package gen-sdk /provider/bin/pulumi-resource-xyz --language go
rm -rf sdk/nodejs
pulumi package gen-sdk /provider/bin/pulumi-resource-xyz --language nodejs
cd sdk/nodejs/ && \
        yarn install && \
        yarn run tsc && \
        cp ../../README.md ../../LICENSE package.json yarn.lock bin/ && \
        sed -i.bak 's/${VERSION}/v0.0.1-alpha.1697065379+97813b07.dirty/g' bin/package.json && \
        rm ./bin/package.json.bak
yarn install v1.22.19
warning package.json: No license field
info No lockfile found.
warning @pulumi/xyz@${VERSION}: No license field
[1/4] Resolving packages...
warning @pulumi/pulumi > [email protected]: The functionality that this package provided is now in @npmcli/arborist
warning @pulumi/pulumi > @opentelemetry/instrumentation > @opentelemetry/[email protected]: Please use @opentelemetry/api >= 1.3.0
warning @pulumi/pulumi > @opentelemetry/instrumentation-grpc > @opentelemetry/[email protected]: Please use @opentelemetry/api >= 1.3.0
warning @pulumi/pulumi > read-package-tree > [email protected]: This functionality has been moved to @npmcli/fs
warning @pulumi/pulumi > read-package-tree > readdir-scoped-modules > [email protected]: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 10.55s.
yarn run v1.22.19
warning package.json: No license field
$ /provider/sdk/nodejs/node_modules/.bin/tsc
Done in 1.82s.
rm -rf sdk/python
pulumi package gen-sdk /provider/bin/pulumi-resource-xyz --language python
cp README.md sdk/python/
cd sdk/python/ && \
        python3 setup.py clean --all 2>/dev/null && \
        rm -rf ./bin/ ../python.bin/ && cp -R . ../python.bin && mv ../python.bin ./bin && \
        sed -i.bak -e 's/^VERSION = .*/VERSION = "0.0.1a1697065379+dirty"/g' -e 's/^PLUGIN_VERSION = .*/PLUGIN_VERSION = "0.0.1-alpha.1697065379+97813b07.dirty"/g' ./bin/setup.py && \
        rm ./bin/setup.py.bak && \
        cd ./bin && python3 setup.py build sdist
running clean
running build
running build_py
creating build
creating build/lib
creating build/lib/pulumi_xyz
copying pulumi_xyz/provider.py -> build/lib/pulumi_xyz
copying pulumi_xyz/__init__.py -> build/lib/pulumi_xyz
copying pulumi_xyz/random.py -> build/lib/pulumi_xyz
copying pulumi_xyz/_utilities.py -> build/lib/pulumi_xyz
copying pulumi_xyz/py.typed -> build/lib/pulumi_xyz
copying pulumi_xyz/pulumi-plugin.json -> build/lib/pulumi_xyz
running sdist
running egg_info
creating pulumi_xyz.egg-info
writing pulumi_xyz.egg-info/PKG-INFO
writing dependency_links to pulumi_xyz.egg-info/dependency_links.txt
writing requirements to pulumi_xyz.egg-info/requires.txt
writing top-level names to pulumi_xyz.egg-info/top_level.txt
writing manifest file 'pulumi_xyz.egg-info/SOURCES.txt'
reading manifest file 'pulumi_xyz.egg-info/SOURCES.txt'
writing manifest file 'pulumi_xyz.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) should be supplied

creating pulumi_xyz-0.0.1a1697065379+dirty
creating pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
creating pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
copying files to pulumi_xyz-0.0.1a1697065379+dirty...
copying README.md -> pulumi_xyz-0.0.1a1697065379+dirty
copying setup.py -> pulumi_xyz-0.0.1a1697065379+dirty
copying pulumi_xyz/__init__.py -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
copying pulumi_xyz/_utilities.py -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
copying pulumi_xyz/provider.py -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
copying pulumi_xyz/pulumi-plugin.json -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
copying pulumi_xyz/py.typed -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
copying pulumi_xyz/random.py -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz
copying pulumi_xyz.egg-info/PKG-INFO -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
copying pulumi_xyz.egg-info/SOURCES.txt -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
copying pulumi_xyz.egg-info/dependency_links.txt -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
copying pulumi_xyz.egg-info/not-zip-safe -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
copying pulumi_xyz.egg-info/requires.txt -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
copying pulumi_xyz.egg-info/top_level.txt -> pulumi_xyz-0.0.1a1697065379+dirty/pulumi_xyz.egg-info
Writing pulumi_xyz-0.0.1a1697065379+dirty/setup.cfg
creating dist
Creating tar archive
removing 'pulumi_xyz-0.0.1a1697065379+dirty' (and everything under it)
@usrbinkat usrbinkat added kind/enhancement Improvements or new features needs-triage Needs attention from the triage team labels Oct 12, 2023
@mikhailshilkov mikhailshilkov removed the needs-triage Needs attention from the triage team label Oct 12, 2023
@usrbinkat usrbinkat mentioned this issue Oct 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement Improvements or new features
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

working provider build in docker usrbinkat/pulumi-provider-boilerplate
2 participants
@mikhailshilkov @usrbinkat