PEP 694: Abstract file upload mechanisms

[ewdurbin]

This attempts to defer the implementation details of getting the bits of a given artifact from the client to the server.

The primary motivation here is to decouple this PEP from resumable/multi-part uploads, provide flexibility to implementers of PEP 694, and allow for new upload mechanisms without a PEP cycle.

---

📚 Documentation preview 📚: https://pep-previews--4431.org.readthedocs.build/

[ewdurbin]

I spent some time to sketch this out as a Finite State Machine in pypi/warehouse#18174, and will be using what I learned to refine this a bit!

[ewdurbin]

@dstufft @warsaw I think this is ready for a proper review.

[warsaw]

I'm planning on taking a closer look later today.

[dstufft]

Couple comments, but overall the changes look good to me.

I just wanted to note a few things I came across when reading the entire PEP (with these changes incorporated-- and probably a number of these came from my original PEP so that's on me :D ).

• The PEP specifies that the endpoint for PyPI will be https://upload.pypi.org/2.0. I would probably remove that from the PEP and let PyPI decide what it's endpoint will be (in particular the /2.0 part is somewhat confusing given the PEP also uses conneg).
• The PEP calls out the inability to parallelize or resume an upload as a problem to be solved, and then later states the PEP solves all the identified problems. The TUS based approach didn't really solve parallelization to begin with, and the removal of TUS means the PEP also doesn't solve resuming an uploading. I think that's fine, but we should update the wording.
• The content type handling is kind of wonky I think. Much like PEP 691 the client can use the Accept header to request a particular content type from the server, and the server includes the full version number in the meta.api-version key in the response. However, the requests appear to only be using the meta.api-version key. I think ideally we want to have requests using a correct Content-Type for their request (and the latest wouldn't be supported here), and have the server use that for handling the request data. We probably want to also explicitly require that the meta.api-version matches the Content-Type for major version.

[dstufft]

I wonder if, instead of treating http-post-application-octet-stream as special, if we should treat this somewhat like how content types are treated?

In other words, basically define a registry of mechanisms (and in this PEP, the only one that is defined is http-post-application-octet-stream), but leave the door open that we can add other standard mechanisms to that registry in the future through future PEPs.

It may also make sense to further take a page from how content types are handled, and say that non-standard mechanisms have to start with vnd- to prevent namespace collisions. So a PyPI specific one would be vnd-pypi-whatever.

[dstufft]

I can kind of guess what this mechanism is based on the name, but it would probably be a good idea to spell out what this mechanism entails (I assume it's just a HTTP POST with the contents of the file as the body).