PEP 694: Abstract file upload mechanisms

[ewdurbin]

This attempts to defer the implementation details of getting the bits of a given artifact from the client to the server.

The primary motivation here is to decouple this PEP from resumable/multi-part uploads, provide flexibility to implementers of PEP 694, and allow for new upload mechanisms without a PEP cycle.

---

📚 Documentation preview 📚: https://pep-previews--4431.org.readthedocs.build/

[ewdurbin]

I spent some time to sketch this out as a Finite State Machine in pypi/warehouse#18174, and will be using what I learned to refine this a bit!

[ewdurbin]

@dstufft @warsaw I think this is ready for a proper review.

[warsaw]

I'm planning on taking a closer look later today.

[dstufft]

Couple comments, but overall the changes look good to me.

I just wanted to note a few things I came across when reading the entire PEP (with these changes incorporated-- and probably a number of these came from my original PEP so that's on me :D ).

• The PEP specifies that the endpoint for PyPI will be https://upload.pypi.org/2.0. I would probably remove that from the PEP and let PyPI decide what it's endpoint will be (in particular the /2.0 part is somewhat confusing given the PEP also uses conneg).
• The PEP calls out the inability to parallelize or resume an upload as a problem to be solved, and then later states the PEP solves all the identified problems. The TUS based approach didn't really solve parallelization to begin with, and the removal of TUS means the PEP also doesn't solve resuming an uploading. I think that's fine, but we should update the wording.
• The content type handling is kind of wonky I think. Much like PEP 691 the client can use the Accept header to request a particular content type from the server, and the server includes the full version number in the meta.api-version key in the response. However, the requests appear to only be using the meta.api-version key. I think ideally we want to have requests using a correct Content-Type for their request (and the latest wouldn't be supported here), and have the server use that for handling the request data. We probably want to also explicitly require that the meta.api-version matches the Content-Type for major version.

[ewdurbin]

• The PEP calls out the inability to parallelize or resume an upload as a problem to be solved, and then later states the PEP solves all the identified problems. The TUS based approach didn't really solve parallelization to begin with, and the removal of TUS means the PEP also doesn't solve resuming an uploading. I think that's fine, but we should update the wording.

While this PEP would no longer directly implement resumable or parallel uploads, it does solve the problem of how to address them. Individual file-upload sessions may occur in parallel if a server chooses to implement a mechanism that can support it, and similarly resumable uploads can be implemented as a mechanism. I'll clarify it in the "The new upload API...", 05d7fc2

[ewdurbin]

Realization while specifying http-post-application-octet-stream: we need to support PEP 740 style attestations... tagging @woodruffw for thoughts :) See: 2ef077c

[ewdurbin]

• The content type handling is kind of wonky I think. Much like PEP 691 the client can use the Accept header to request a particular content type from the server, and the server includes the full version number in the meta.api-version key in the response. However, the requests appear to only be using the meta.api-version key. I think ideally we want to have requests using a correct Content-Type for their request (and the latest wouldn't be supported here), and have the server use that for handling the request data. We probably want to also explicitly require that the meta.api-version matches the Content-Type for major version.

See: 924a27d

[ewdurbin]

• The PEP specifies that the endpoint for PyPI will be https://upload.pypi.org/2.0. I would probably remove that from the PEP and let PyPI decide what its endpoint will be (in particular the /2.0 part is somewhat confusing given the PEP also uses conneg).

See: f8469cf

[warsaw]

I'm bouncing between reading the PR preview and the diff. I'll capture some thoughts here based on text not touched in the PR (which I don't think GH gives me the UI to add inline comments to).

• Instead of "a standard API" I think we're now talking about "an extensible API" with some standard behavior.

[warsaw]

Okay, I've pretty much run out of gas. I think some of my comments may not make a ton of sense since I reviewed it sequentially rather than reading the whole thing and then composing my feedback. Apologies for that.

Overall, I think this is a really good simplification, and I really like the direction it's going in. I know I have a lot of musings, feedback, thoughts, comments, and suggestions sprinkled throughout, and I hope they're moderately helpful.

If it would be helpful, I can try to edit the PR locally and push changes, or I could branch your PR and push a new branch/PR, or we can just try to make it all work here. Happy to also chat about it separately!

[warsaw]

While this PEP would no longer directly implement resumable or parallel uploads, it does solve the problem of how to address them. Individual file-upload sessions may occur in parallel if a server chooses to implement a mechanism that can support it, and similarly resumable uploads can be implemented as a mechanism. I'll clarify it in the "The new upload API...",

Alternative to inline suggestions:

The new upload API proposed in this PEP provides ways to solve all of these problems, either
directly or through an extensible approach, allowing servers to implement features such as resumable
and parallel uploads.  This PEP proposes better error reporting, a more robust release testing
experience, and atomic and simultaneous publishing of all release artifacts.

[konstin]

Could you add some sentences about forward compatibility? I'm thinking e.g. about declaring all value enums and dicts as non-exhaustive, so we can add new info both on the server and client side without it being a breaking change. This allows let's say adding variant information (from the WheelNext project) to the POST upload field, or having additionally statuses in the future. (This is something I'd expect implementers to do anyway, but spelling it out helps).

New required mechanisms MUST NOT be added [...] without an update to the major version.

Is it intentional that this means that we can't have an Upload API 2.1 that requires servers that say they're version 2.1 to implement let's say an S3 endpoint? This would be compatible with 2.0 clients since they can still use regular application/octet-stream POST.

Upload authentication is also not standardized.

As a client implementer, it would be very valuable to have a standard way to pass a token. This wouldn't be about how to obtain the token or its lifetime and semantic, and it would also leave room for non-token auth, but it would solve the current user confusion about different servers using different usernames for their API tokens (we regularly get questions about that in uv).

For the session status, do I read it correctly that if the server says an upload is pending, the client has to poll until it is completed, there is no mechanism intended for subscribing to server updates?

nit: Instead of valid-for, what about a valid-until with a timestamp?

[ewdurbin]

Could you add some sentences about forward compatibility? I'm thinking e.g. about declaring all value enums and dicts as non-exhaustive, so we can add new info both on the server and client side without it being a breaking change. This allows let's say adding variant information (from the WheelNext project) to the POST upload field, or having additionally statuses in the future. (This is something I'd expect implementers to do anyway, but spelling it out helps).

I'm uncertain about this. I believe that the specific example provided is addressed by the fact that variants would already be encoded into the filename, and I assume (though I'm not fully up to date) in the METADATA file of the wheel.

Additional statuses that implementors might need would generally be part of their File Upload Mechanism, which can already document and include its own methods for retrieving detailed status in its responses.

New required mechanisms MUST NOT be added [...] without an update to the major version.

Is it intentional that this means that we can't have an Upload API 2.1 that requires servers that say they're version 2.1 to implement let's say an S3 endpoint? This would be compatible with 2.0 clients since they can still use regular application/octet-stream POST.

I consider this intentional. Requiring the addition of a mechanism is a large enough change that implementors will have real work to do. I don't forsee that we will ever require mechanisms that can't be "baked in" to an arbitrary server. Requiring S3-compatible protocols of arbitrary servers... no. While requiring implementation of a standard like tus is likely for a future major version.

Upload authentication is also not standardized.

As a client implementer, it would be very valuable to have a standard way to pass a token. This wouldn't be about how to obtain the token or its lifetime and semantic, and it would also leave room for non-token auth, but it would solve the current user confusion about different servers using different usernames for their API tokens (we regularly get questions about that in uv).

Understandable, but ultimately I think that enforcing authentication methods through PEP would likely cause trouble . With PEP authoring hat on, I think it would hamstring this PEPs progress. With my PyPI maintainer hat on, I think it would unnecessarily impede future progress on evolving PyPI's authentication mechanisms, Macaroons are already showing some unforeseen consequences and we desperately need to address authentication for more general APIs.

For the session status, do I read it correctly that if the server says an upload is pending, the client has to poll until it is completed, there is no mechanism intended for subscribing to server updates?

nit: Instead of valid-for, what about a valid-until with a timestamp?

I think that is reasonable, though I don't specifically prefer one option to the other.

[ewdurbin]

For the session status, do I read it correctly that if the server says an upload is pending, the client has to poll until it is completed, there is no mechanism intended for subscribing to server updates?

Ope, missed this. Correct there is no method for subscription in this model.

[konstin]

Understandable, but ultimately I think that enforcing authentication methods through PEP would likely cause trouble . With PEP authoring hat on, I think it would hamstring this PEPs progress. With my PyPI maintainer hat on, I think it would unnecessarily impede future progress on evolving PyPI's authentication mechanisms, Macaroons are already showing some unforeseen consequences and we desperately need to address authentication for more general APIs.

To me, a lack of auth is a major hurdle to writing a client with Upload API 2.0: Without a way to pass any sorts of credentials, it's impossible to write a client that works universally the way twine and uv publish can currently talk to any registry; You can't write a CLI interface, as we don't know that the CLI arguments would be.

[ewdurbin]

To me, a lack of auth is a major hurdle to writing a client with Upload API 2.0: Without a way to pass any sorts of credentials, it's impossible to write a client that works universally the way twine and uv publish can currently talk to any registry; You can't write a CLI interface, as we don't know that the CLI arguments would be.

Again, this is very understandable, but it is likely to mean that this PEP would bake in HTTP Basic Auth with __token__ username, and a generic "token" in as the password. Maybe this is fine, but I certainly don't prefer it as a constraint on PyPI's implementation long term.