Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable architecture specific dependencies #622

Open
wants to merge 1 commit into
base: v2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ require (
github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5
github.com/gabriel-vasile/mimetype v1.4.7
github.com/google/uuid v1.6.0
github.com/onsi/ginkgo v1.16.5 // indirect
github.com/onsi/gomega v1.34.1
github.com/pelletier/go-toml v1.9.5
github.com/sclevine/spec v1.4.0
Expand Down
3 changes: 3 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,7 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU=
github.com/fsnotify/fsnotify v1.5.4/go.mod h1:OVB6XrOHzAwXMpEM7uPOzcehqUV2UqJxmVXmkdnm1bU=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/fullsailor/pkcs7 v0.0.0-20190404230743-d7302db945fa/go.mod h1:KnogPXtdwXqoenmZCw6S+25EAm2MkxbG0deNDu4cbSA=
github.com/fxamacker/cbor/v2 v2.4.0/go.mod h1:TA1xS00nchWmaBnEIxPSE5oHLuJBAVvqrtAnWBwBCVo=
Expand Down Expand Up @@ -1657,6 +1658,7 @@ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLA
github.com/nwaples/rardecode v1.1.0 h1:vSxaY8vQhOcVr4mm5e8XllHWTiM4JF507A0Katqw7MQ=
github.com/nwaples/rardecode v1.1.0/go.mod h1:5DzqNKiOdpKKBH87u8VlvAnPZMXcGRhxWkRpHbbfGS0=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
Expand Down Expand Up @@ -3100,6 +3102,7 @@ gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/square/go-jose.v2 v2.3.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/square/go-jose.v2 v2.5.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
Expand Down
3 changes: 3 additions & 0 deletions postal/buildpack.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ type Dependency struct {
// StripComponents behaves like the --strip-components flag on tar command
// removing the first n levels from the final decompression destination.
StripComponents int `toml:"strip-components"`

// ARCH is the architecture of this dependency
Arch string `toml:"arch"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are several platforms, for example, pack cli has plenty of them when it comes to build and image https://github.com/buildpacks/pack/blob/5cca9c51a61cf1bb8b0b6ba479bbbf10a65f0377/internal/target/platform.go#L31-L47 For nodejs we have a subset of them https://github.com/nodejs/node/blob/main/BUILDING.md#platform-list which I think it would be good to start with. So my suggestion on this one would be to also add another variable called OS or operating-system which will indicate the operating system of the dependency

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am no expert on the whole multiarch thing, but operating-system would be wrong, wouldn't it? The operating system is for all dependencies linux - it is the same run image after all. We "only" have to differ dependencies for the architecture of the image. So I thing anything other than Arch would be quite confusing, don't you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that we might need to support generating images for macos, but currently there is no such use case so makes sense to not over-complicate it. LGTM

}

func parseBuildpack(path, name string) ([]Dependency, string, error) {
Expand Down
31 changes: 29 additions & 2 deletions postal/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"runtime"
"sort"
"strings"
"time"
Expand Down Expand Up @@ -52,15 +54,17 @@ type ErrNoDeps struct {
id string
version string
stack string
arch string
supportedVersions []string
}

// Error implements the error.Error interface
func (e *ErrNoDeps) Error() string {
return fmt.Sprintf("failed to satisfy %q dependency version constraint %q: no compatible versions on %q stack. Supported versions are: [%s]",
return fmt.Sprintf("failed to satisfy %q dependency version constraint %q: no compatible versions on %q stack with architecture %q. Supported versions are: [%s]",
e.id,
e.version,
e.stack,
e.arch,
strings.Join(e.supportedVersions, ", "),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same on this one we should add the os attribute

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we dont use the os, this comment does not apply.

)
}
Expand Down Expand Up @@ -147,6 +151,10 @@ func (s Service) Resolve(path, id, version, stack string) (Dependency, error) {
continue
}

if !isCorrectArch(dependency) {
continue
}

sVersion, err := semver.NewVersion(dependency.Version)
if err != nil {
return Dependency{}, err
Expand All @@ -160,7 +168,7 @@ func (s Service) Resolve(path, id, version, stack string) (Dependency, error) {
}

if len(compatibleVersions) == 0 {
return Dependency{}, &ErrNoDeps{id, version, stack, supportedVersions}
return Dependency{}, &ErrNoDeps{id, version, stack, archFromSystem(), supportedVersions}
}

stacksForVersion := map[string][]string{}
Expand Down Expand Up @@ -220,6 +228,25 @@ func (s Service) Resolve(path, id, version, stack string) (Dependency, error) {
return compatibleVersions[0], nil
}

func isCorrectArch(dep Dependency) bool {
systemArch := archFromSystem()

if systemArch == "amd64" && dep.Arch == "" {
return true
}

return systemArch == dep.Arch
}

func archFromSystem() string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of clean code, one function should only do one thing, I would remove from this function, the two things that is doing 1. looking if the environment variable exists 2. fetching the system architecture. Changing that function to fetching only the architecture does not make sense as there is already a function for that. Therefore, I would suggest to move this logic out in to the Parent function (Resolve) as I think this is totally normal as there is already logic for validating other things like (dependency version and stacks). Althouh, if you decide to fetch the platform, then it would be good to wrap these two lines in a function called getSystemPlatform as it will be more than one line (one for fetching the GOARCH and one for fetching the GOOS

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we dont want to default into the system that the user is using as pack currently fallbacks to linux/amd64, so I would choose this platform as default, in order to avoid any errors from macos users that currently building applications

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we dont want to default into the system

I might be completely wrong, but that is not what is done. It looks up the architecture of the runtime, meaning the buildpack that actually called getDependency(). So no other architecture could even work. The BP_ARCH is only needed for unit tests.

I tried to keep everything as similar to libak as possible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, so seems that it is not breaking the default behavior. LGTM

archFromEnv, ok := os.LookupEnv("BP_ARCH")
if !ok {
archFromEnv = runtime.GOARCH
}

return archFromEnv
}

func stringSliceContains(slice []string, str string) bool {
for _, s := range slice {
if s == str {
Expand Down
88 changes: 87 additions & 1 deletion postal/service_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
//nolint Ignore SA1019, usage of deprecated package within a deprecated test case
"github.com/paketo-buildpacks/packit/v2/paketosbom"

. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)

Expand Down Expand Up @@ -115,6 +116,12 @@ strip-components = 1
service = postal.NewService(transport).
WithDependencyMappingResolver(mappingResolver).
WithDependencyMirrorResolver(mirrorResolver)

Expect(os.Setenv("BP_ARCH", "amd64")).To(Succeed())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On this one, instead of having BP_ARCH and BP_OS, we could rename the BP_ARCH to something like BP_PLATFORM or BP_TARGET, which will include both architecture and operating system, similar to what pack cli does with the targets.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we dont use the os, this comment does not apply.

})

AfterEach(func() {
Expect(os.Unsetenv("BP_ARCH")).To(Succeed())
})

context("Resolve", func() {
Expand Down Expand Up @@ -310,6 +317,85 @@ version = "4.5.6"
})
})

context("when there architecture specific versions", func() {
it.Before(func() {
err := os.WriteFile(path, []byte(`
[metadata]
[metadata.default-versions]
some-entry = "1.2.x"

[[metadata.dependencies]]
id = "some-entry"
sha256 = "some-sha-amd64"
stacks = ["*"]
uri = "some-uri"
version = "1.2.3"
arch = "amd64"

[[metadata.dependencies]]
id = "some-entry"
sha256 = "some-sha-arm"
stacks = ["*"]
uri = "some-uri"
version = "1.2.3"
arch = "arm"

[[metadata.dependencies]]
id = "some-other-entry"
sha256 = "some-other-sha"
stacks = ["*"]
uri = "some-uri"
version = "1.2.4"

`), 0600)
Expect(err).NotTo(HaveOccurred())
})

context("BP_ARCH=amd64", func() {
it.Before(func() {
Expect(os.Setenv("BP_ARCH", "amd64")).To(Succeed())
})

it.After(func() {
Expect(os.Unsetenv("BP_ARCH")).To(Succeed())
})

it("picks the dependency with the correct arch", func() {
dependency, err := service.Resolve(path, "some-entry", "1.2.3", "some-stack")
Expect(err).NotTo(HaveOccurred())
Expect(dependency.SHA256).To(Equal("some-sha-amd64"))
})

it("picks the dependency with the no arch", func() {
dependency, err := service.Resolve(path, "some-other-entry", "1.2.4", "some-stack")
Expect(err).NotTo(HaveOccurred())
Expect(dependency.SHA256).To(Equal("some-other-sha"))
})
})

context("BP_ARCH=arm", func() {
it.Before(func() {
Expect(os.Setenv("BP_ARCH", "arm")).To(Succeed())
})

it.After(func() {
Expect(os.Unsetenv("BP_ARCH")).To(Succeed())
})

it("picks the dependency with the correct arch", func() {
dependency, err := service.Resolve(path, "some-entry", "1.2.3", "some-stack")
Expect(err).NotTo(HaveOccurred())
Expect(dependency.SHA256).To(Equal("some-sha-arm"))
})

it("fails if no dependency with the correct arch is found", func() {
_, err := service.Resolve(path, "some-other-entry", "1.2.4", "some-stack")
Expect(err).To(MatchError(ContainSubstring("failed to satisfy")))
Expect(err).To(MatchError(ContainSubstring("\"arm\"")))
})
})
})

context("when both a wildcard stack constraint and a specific stack constraint exist for the same dependency version", func() {
it.Before(func() {
err := os.WriteFile(path, []byte(`
Expand Down Expand Up @@ -429,7 +515,7 @@ version = "1.2.3"
expectedErr := &postal.ErrNoDeps{}
_, err := service.Resolve(path, "some-entry", "9.9.9", "some-stack")
Expect(errors.As(err, &expectedErr)).To(BeTrue())
Expect(err).To(MatchError(ContainSubstring("failed to satisfy \"some-entry\" dependency version constraint \"9.9.9\": no compatible versions on \"some-stack\" stack. Supported versions are: [1.2.3, 4.5.6]")))
Expect(err).To(MatchError(ContainSubstring("failed to satisfy \"some-entry\" dependency version constraint \"9.9.9\": no compatible versions on \"some-stack\" stack with architecture \"amd64\". Supported versions are: [1.2.3, 4.5.6]")))
})
})
})
Expand Down
Loading