-
Notifications
You must be signed in to change notification settings - Fork 222
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unify "Verifying Artifacts" across provenance and spec #675
Conversation
✅ Deploy Preview for slsa ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
the artifact distribution system ("package ecosystem") and artifact consumer. | ||
This page is divided into two sections. The first discusses choices system | ||
implementers must make regarding verifying provenance. The second describes | ||
the procedure for verifying an artifact and its provenance against a set of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a great change. The first part is for system implementers. Can the second part specify its audience too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, though I worry the wording is a bit awkward.
### Setting Expectations | ||
#### Consumer | ||
|
||
[Consumer]: #consumer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is in the implementer section I had expected information on what the implementer needs to do to perform the verification at the consumer point. Should there be guidance for the implementer here, such as why they should or shouldn't choose this option, or what they need to do to inform the consumer of their responsibilities? Or is that better suited elsewhere (i.e., a forthcoming "How to SLSA -- Implementers" page)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's save that discussion for the "how to SLSA" page.
|
||
#### During consumption | ||
|
||
During client-side installation/deployment of a package, the verifier |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"...the verifier can ensure..." ---flagging for potentially unclear term (slsa-verifier vs. consumer as verifier)
Edit: Hm, I see that you clarify in the next sentence that the verifier can be two types of tooling. Would it make sense to says "During client-side installation/deployment of a package, verification tooling can ensure..." (or am I over thinking this)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
|
||
<dfn>Expectations</dfn> are known provenance values that indicate the corresponding | ||
artifact is authentic. For example, a producer can define the allowed values for | ||
[`buildType`](/provenance/v1#buildType) and | ||
[`externalParameters`](/provenance/v1#externalParameters) | ||
for a given package (assuming it uses the SLSA provenance format) in order to address | ||
the [build integrity threats](threats#build-integrity-threats). | ||
> **TODO:** link to more concrete guidance once it's available. | ||
|
||
Expectations MUST be sufficient to detect | ||
or prevent this adversary from injecting unofficial behavior into the package. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this be enough information for readers to know what is sufficient and what is not?
verification is of limited benefit unless a human or another part of the system | ||
responds to the failed verification. | ||
|
||
## Setting Expectations | ||
|
||
<dfn>Expectations</dfn> are known provenance values that indicate the corresponding |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just want to check on this definition. Is "expectations" used commonly in this context outside of SLSA? If not, I'm not sure the definition will be enough for new users to know what this means. ("known provenance values" seems like it could be confusing if you didn't know what it meant already)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this meaning comes from unit testing, where "expectation" is sometimes a synonym for "test assertion". I don't know how popular that usage is, though a quick Google search makes me think it may have fallen out of favor.
I'm not sure how to improve the definition, so I rewrote the example instead to be a bit more concrete. WDYT?
might wish to go beyond SLSA's minimum requirements in order to protect against | ||
threats further up the supply chain. | ||
|
||
One possible approach is to recursively verify each entry in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to clarify, is this one possible approach to checking dependencies recursively, or to going beyond SLSA's requirements to protect against threats further up the supply chain? Or are they the same?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leaving this comment open for a follow-up PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This raises the question about how far one should go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The intention was to say this this is something that one probably wants to do, without calling it out as a requirement because, as @lehors said, there is a question of how far one should go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reordered the text a bit to emphasize that this step is optional and best-effort.
trustworthiness of the provenance. This mitigates some or all of [threats] "D", | ||
"F", "G", and "H", depending on SLSA Build level and where verification happens. | ||
|
||
Up front: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpicky, but does "up front" mean you do it once and not again? Is there a more precise phrase you can use here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
### Where to verify | ||
|
||
There are three options for where in a system to verify provenance: at the | ||
[package ecosystem], at the [consumer], or at an external [monitoring] system. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
broken link: [monitoring]
is not defined (shows up with brackets in rendered page instead of link)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
docs/provenance/v1.md
Outdated
@@ -378,7 +378,8 @@ REQUIRED for SLSA Build L1: `id` | |||
<td>string (<a href="https://github.com/in-toto/attestation/blob/main/spec/field_types.md#TypeURI">TypeURI</a>)<td> | |||
|
|||
URI indicating the transitive closure of the trusted builder. This is | |||
[intended][Step 1] to be the sole determiner of the SLSA Build level. | |||
[intended](/spec/v1.0/verifying-artifacts##step-1-check-slsa-build-level) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extra #
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
and software consumers. | ||
provenance doesn't do anything unless somebody inspects it. SLSA calls that | ||
inspection verification, and this page describes how to verify artifacts and | ||
their SLSA provenance. | ||
|
||
## Overview |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I think you can drop this heading. Also if you wrap the above paragraph in <div class="subtitle">
it will show up bigger, like the other pages
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
|
||
## Overview | ||
|
||
Artifact verification is a process that consists of several stages that involves | ||
the artifact distribution system ("package ecosystem") and artifact consumer. | ||
This page is divided into two sections. The first discusses choices system |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: this is a bit confusing since there are three following sections (##
headings)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
ecosystem because doing so provides security benefits to all of the | ||
package ecosystem's clients. | ||
|
||
#### Package ecosystem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Feel free to defer to a future PR)
nit: I'm having trouble following this many levels of heading. Can we make this inner one a bulleted list (here and for "when to verify")? Something like:
* **Package ecosystem:** The package registry verifies ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed the when/where heading levels and kept the options as third-level headers. I prefer this option to a bulleted list because each entry would have multiple paragraphs.
Consumers can trust their package ecosystem or some other third party to verify | ||
artifacts, or they can verify artifacts against their own set of expectations. | ||
System implementers must make two architectural decisions about verifying | ||
provenance: where in the system to verify and when to do it. Each option |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note for the future: I find this hard to follow since the "where" and "when" are usually tied in practice:
- "ecosystem" usually means "upload" (or at least that's what I think)
- "consumers" usually means "download"
- "monitor" means "continuously"
Are they actually independent? Can we simplify?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, I had also wondered if "where" and "when" could be clearer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks, this is a solid improvement. I added a few minor comments.
System implementers must make two architectural decisions about verifying | ||
provenance: where in the system to verify and when to do it. Each option | ||
comes with its own set of considerations, but all are valid. The options | ||
are also not mutually excluvie -- more than one component of a system |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are also not mutually excluvie -- more than one component of a system | |
are also not mutually exclusive -- more than one component of a system |
themselves using the prompts in [verifying systems](verifying-systems.md) or | ||
rely on the [SLSA certification program](certification.md) (coming soon). | ||
|
||
#### External Monitor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need the external qualifier, or can we just call this a monitor? Do you mean external to the registry? Or the package management ecosystem?
A package's <dfn>consumer</dfn> is the organization or individual that uses the | ||
package. | ||
|
||
Consumers can set their own expectations for artifacts and use client-side |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consumers can set their own expectations, but may use default expectations as stated by the package producer and/or package ecosystem. Is it worth adding something about that here to ensure that readers don't think a consumer must set their own expectations?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. I'm unclear as to how this goes with the earlier statement that "The provenance verifier sets expectations for a package". Shouldn't that earlier statement be in line with what is said here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I adopted Josh's wording. I also added a bit to the "Architecture Options" heading clarifying that verification must happen at least once, with an example.
Consumers can either audit the build systems | ||
themselves using the prompts in [verifying systems](verifying-systems.md) or | ||
rely on the [SLSA certification program](certification.md) (coming soon). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not clear from the context of this page why that's necessary to audit the build system, nor why it's only mentioned for consumers. Should we add a sentence or so here? Or are we confident that readers have sufficient context from the rest of the doc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: it's called the SLSA certification program here, but the SLSA conformance program below. Can we make this consistent?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree. I think might be worth restating that given that the consumer has to trust the build system they should make a determination one way or another as to whether it is indeed trustworthy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I deleted the paragraph. It's redundant with the verification procedure section.
`externalParameters` that record a list of commands to run is likely impractical | ||
to verify because the commands change on every build. Instead, consider a | ||
`buildType` that defines the list of commands in a configuration file in a | ||
source repository, then make put only the source repository in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
source repository, then make put only the source repository in | |
source repository, then put only the source repository in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM overall but I'm concerned by the significant change brought in with regard to who sets the expectations.
docs/spec/v1.0/terminology.md
Outdated
@@ -96,7 +96,7 @@ build system the package was built. | |||
|
|||
| Term | Description | | |||
|--------------|---- | | |||
| Expectations | A set of constraints on the package's provenance metadata. The package producer sets expectations for a package, whether explicitly or implicitly. | | |||
| Expectations | A set of constraints on the package's provenance metadata. The provenance verifier sets expectations for a package, whether explicitly or implicitly. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a major change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is, but I think it's a worthwhile change. The current division of labor makes the producer responsible for the verification tool's configuration, even if the producer and verifier are separate parties with no relationship. Shifting the responsibility to the verifier doesn't prevent the producer from influencing the expectations, but it does ensure that they aren't responsible for infrastructure they might not control.
There is precedent for this model with SSH keys. The user is ultimately responsible for the contents of their known_hosts file. There are better and worse ways to populate that file, but the remote isn't to blame if you trust an invalid key. The analogy isn't perfect, but I think the overall idea applies.
I'm open to changing the definition, but I want to be sure that the spec doesn't give any party responsibility for things they can't control and that it supports trust-on-first-use as a model for setting expectations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I understand now, and I believe what's happening here is confusion over a term meaning two things:
- The producer SETS expectations, meaning that they communicate authenticity features to help others verify their artifact;
- The consumer INPUTS (not sets) expectations, meaning that they take the expectations from the producer and feed them into a tool to verify the artifact.
Would splitting the idea of "set expectations" into two terms based on those roles help disambiguate what's going on here? Line 99 would revert to the original ("The package producer sets expectations for a package...") and we could add a sentence explaining that the consumer inputs these expectations into tooling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like @joshuagl's point of view better: the package producer and/or ecosystem sets default expectations that the consumer may use but can override at verification time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reverted this change and am double-checking that this PR is consistent throughout wrt who is responsible for setting expectations.
I'll open a separate PR separating the two meanings of "expectations" so that we can discuss that change in isolation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good. Thanks.
|
||
## Overview | ||
|
||
Artifact verification is a process that consists of several stages that involves | ||
the artifact distribution system ("package ecosystem") and artifact consumer. | ||
This page is divided into the sections. The first discusses choices system |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This page is divided into the sections. The first discusses choices system | |
This page is divided into several sections. The first discusses choices system |
I think you need to expand on what "system" refers to here. The current version talks about distribution systems/package ecosystem I take from reading the rest of the changes that you mean for this to be broader but you need to be more specific. I think "system" is too generic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was intentionally being a bit vague because verification can happen in the package ecosystem or on the consumer side, and those are built by different parties. I clarified that this section is targeting distribution and deployment systems.
verification tooling to ensure that the artifact's provenance matches the | ||
consumer's expectations for that package name before use (e.g. during |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
verification tooling to ensure that the artifact's provenance matches the | |
consumer's expectations for that package name before use (e.g. during | |
verification tooling to ensure that the artifact's provenance matches their | |
expectations for that package before use (e.g. during |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
A package's <dfn>consumer</dfn> is the organization or individual that uses the | ||
package. | ||
|
||
Consumers can set their own expectations for artifacts and use client-side |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. I'm unclear as to how this goes with the earlier statement that "The provenance verifier sets expectations for a package". Shouldn't that earlier statement be in line with what is said here?
Consumers can either audit the build systems | ||
themselves using the prompts in [verifying systems](verifying-systems.md) or | ||
rely on the [SLSA certification program](certification.md) (coming soon). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree. I think might be worth restating that given that the consumer has to trust the build system they should make a determination one way or another as to whether it is indeed trustworthy.
Verifiers SHOULD reject unrecognized fields in `externalParameters` to err on | ||
the side of caution. It is acceptable to allow a parameter to have a range of | ||
values (possibly any value) if it is known that any value in the range is safe. | ||
Implementations need not special-case the `buildType` if JSON comparisons are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what "special-case" mean here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The intention was to proactively address a question that came up: does a verification tool have to have custom logic for each buildTypes
it supports, e.g. to understand the schema of externalParameters
? The answer is "probably not - regular JSON comparisons are likely fine". Any thoughts on how to word better?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried turning it around. Instead of saying you don't need special-casing, I said that JSON comparisons are sufficient. WDYT?
might wish to go beyond SLSA's minimum requirements in order to protect against | ||
threats further up the supply chain. | ||
|
||
One possible approach is to recursively verify each entry in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This raises the question about how far one should go.
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to get this submitted sooner rather than later, so maybe we can submit something that is good enough and then iterate on wording?
might wish to go beyond SLSA's minimum requirements in order to protect against | ||
threats further up the supply chain. | ||
|
||
One possible approach is to recursively verify each entry in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The intention was to say this this is something that one probably wants to do, without calling it out as a requirement because, as @lehors said, there is a question of how far one should go.
Verifiers SHOULD reject unrecognized fields in `externalParameters` to err on | ||
the side of caution. It is acceptable to allow a parameter to have a range of | ||
values (possibly any value) if it is known that any value in the range is safe. | ||
Implementations need not special-case the `buildType` if JSON comparisons are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The intention was to proactively address a question that came up: does a verification tool have to have custom logic for each buildTypes
it supports, e.g. to understand the schema of externalParameters
? The answer is "probably not - regular JSON comparisons are likely fine". Any thoughts on how to word better?
Signed-off-by: kpk47 <[email protected]>
Co-authored-by: Mark Lodato <[email protected]> Signed-off-by: kpk47 <[email protected]>
Signed-off-by: kpk47 <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks.
docs/spec/v1.0/terminology.md
Outdated
@@ -96,7 +96,7 @@ build system the package was built. | |||
|
|||
| Term | Description | | |||
|--------------|---- | | |||
| Expectations | A set of constraints on the package's provenance metadata. The package producer sets expectations for a package, whether explicitly or implicitly. | | |||
| Expectations | A set of constraints on the package's provenance metadata. The provenance verifier sets expectations for a package, whether explicitly or implicitly. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good. Thanks.
Co-authored-by: Arnaud J Le Hors <[email protected]> Signed-off-by: kpk47 <[email protected]>
Co-authored-by: Mark Lodato <[email protected]> Signed-off-by: kpk47 <[email protected]>
…k#675) * Merge verification procedure into 'verifying artifacts' page. Signed-off-by: kpk47 <[email protected]> * add note about alternate provenance formats Signed-off-by: kpk47 <[email protected]> * Responding to review comments Signed-off-by: kpk47 <[email protected]> * lint Signed-off-by: kpk47 <[email protected]> * review comments Signed-off-by: kpk47 <[email protected]> * lint Signed-off-by: kpk47 <[email protected]> * addressing review comments Signed-off-by: kpk47 <[email protected]> * lint Signed-off-by: kpk47 <[email protected]> * Revert changes to terminology.md Signed-off-by: kpk47 <[email protected]> * Update docs/spec/v1.0/verifying-artifacts.md Co-authored-by: Mark Lodato <[email protected]> Signed-off-by: kpk47 <[email protected]> * addressing review comments Signed-off-by: kpk47 <[email protected]> * Update docs/spec/v1.0/verifying-artifacts.md Co-authored-by: Arnaud J Le Hors <[email protected]> Signed-off-by: kpk47 <[email protected]> * Update docs/spec/v1.0/verifying-artifacts.md Co-authored-by: Mark Lodato <[email protected]> Signed-off-by: kpk47 <[email protected]> --------- Signed-off-by: kpk47 <[email protected]> Signed-off-by: kpk47 <[email protected]> Co-authored-by: Mark Lodato <[email protected]> Co-authored-by: Arnaud J Le Hors <[email protected]>
Addresses #650 .