robots.txt is not being updated

[Mark-Simulacrum]

doc.rust-lang.org/robots.txt isn't getting updated, though /stable/robots.txt (and other URLs) are; those are however useless to my knowledge because crawlers only look in the root.

This is a follow up on rust-lang/rust#68677

cc @steveklabnik

[Mark-Simulacrum]

Okay, I've tracked this down to the robots.txt file being special cased in the redirector. For the time being, I've manually replaced it with the one from stable. @pietroalbini -- looking at the lambda function we use, since it's on nodejs v8, we can't deploy updates to it without migrating to v10 or v12. I'm personally really worried about changing it in production anyway, that seems very prone to making a typo or whatever, but we should do so.

Old one:

User-agent: *
Disallow: /0.3/
Disallow: /0.4/
Disallow: /0.5/
Disallow: /0.6/
Disallow: /0.7/
Disallow: /0.8/
Disallow: /0.9/
Disallow: /0.10/
Disallow: /0.11.0/
Disallow: /0.12.0/
Disallow: /1.0.0-alpha/
Disallow: /1.0.0-alpha.2/
Disallow: /1.0.0-beta/
Disallow: /1.0.0-beta.2/
Disallow: /1.0.0-beta.3/
Disallow: /1.0.0-beta.4/
Disallow: /1.0.0-beta.5/

[pietroalbini]

I would strongly prefer to avoid changing the lambda in production, as CloudFront has a ~5 minute delay when propagating changes.

A while back I started porting doc.rlo and static.rlo to Terraform, applying the changes to dev-static.rlo and dev-doc.rlo. I think the best course of action will be to finish porting the dev- domains to Terraform, test the changes in dev-doc and then apply Terraform to the production distributions.

[Mark-Simulacrum]

I probably don't have time to finish that work and manually copying the robots.txt over once in a while doesn't seem too bad for the time being (we only really rarely update it anyway).

[pietroalbini]

This is now fixed.