Remove builds.json API endpoint

[Nemo157]

I would have exact stats, but grafana is currently down 🙃. IIRC we went from ~100 req/min to somewhere around 1 req/hour. The major consumers crates.io and shields.io have definitely migrated, lib.rs is the other known consumer and @kornelski 👍'd my comment about migrating so I hope it has updated by now.

[syphar]

So, while it's true that we now get much less traffic on this endpoint, it's not zero.

I'm not sure if we now still should remove this endpoint, where it doesn't hurt (much) apart from the maintenance effort.

We could try to get access logs from CloudFront to get more information about the user agents on the remaining calls.

From what I understand the builds.json endpoint counts as a public API of docs.rs, so just removing it would break tooling?

What do you think @rust-lang/docs-rs ? Did we have breaking API changes like this in the past?

[GuillaumeGomez]

We did quite a few times, but it was years ago when it wasn't already part of rust-lang, so not sure it counts. It would be interesting to check the actual "users" of this endpoint before taking a decision I think.

[Nemo157]

I'm not sure if we now still should remove this endpoint, where it doesn't hurt (much) apart from the maintenance effort.

I have future changes that would require breaking this endpoint, or making what it says about the crate not really match what we're storing about it.

[Nemo157]

From what I understand the builds.json endpoint counts as a public API of docs.rs, so just removing it would break tooling?

I'm not sure if it is really a public API, I don't believe there's any documentation about it, we've just used it for integrations we at least partially manage like crates.io or shields.io.

[Nemo157]

https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-cloudfront-announces-cache-and-origin-request-policies/ seems like it should support us forwarding the user-agent to the service, I'll see if I can figure out how this looks in terraform and open a PR.

[Nemo157]

We've had 20 requests so far since simpleinfra#344 was deployed, they appear to be a variety of desktop/mobile browser user-agents, so my best guess is there is some website/browser plugin that is doing the request client-side.

user agents

      3 Mozilla/5.0 (iPhone; CPU iPhone OS 17_0_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0.1 Mobile/15E148 Safari/604.1
      1 Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/118.0.5993.69 Mobile/15E148 Safari/604.1
      1 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
      2 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
      7 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
      1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
      2 Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0
      4 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0

[Nemo157]

I tried a github code search too, and didn't find anything relevant, just a bunch of docs.rs, crates.io and shields forks.

[syphar]

I had a chat with Tobias (crates.io) who in the end have a similar issue (private APIs that are publicly used).

To get more info I created rust-lang/simpleinfra#354 to be able to look into the Referer header for more information,

but currently I lean towards keeping the API alive it's not too much effort. From what I remember it would need generating the build_status boolean attribute based on our new status-enum, which is probably fine.
( I'm also happy to invest the work if needed).

[syphar]

This is blocked on

• rebase
• checking who still requests the old API based on the referrer

[syphar]

we just checked the access logs for referrers.

example line:

"GET /crate/bitfun_plugin_api/0.37.4/builds.json HTTP/1.1" 
404 
5166 
"https://rust.inhuawei.com/" 
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"

So the only service requesting this endpoint is https://rust.inhuawei.com/, some internal Huawei service (?), and it's only requesting crates / versions docs.rs doesn't have (all are 404s). cc @GuillaumeGomez ? ;)

So we will drop the endpoint.

[GuillaumeGomez]

Yep it's fine. Never heard about it so I guess it's fine.