Google search for 'rust tutorial' returns old content

[ericsampson]

I noticed that if you search Google for 'rust tutorial', the current top two links are for non-current versions (0.3 and 0.7 I think?). This could be pretty confusing for new users, who may not realize that they are looking at older content which may not reflect the current implementation. I know this isn't necessarily under Rust's control, but I thought I would log it anyway.

[catamorphism]

Good point. The 0.7 tutorial, at least, has the version number in the lower right-hand corner of the screen. The 0.3 tutorial was created before we started putting the version marker there, but I suppose we could always go back and edit it.

[ericsampson]

I forgot about that lower-right corner version stamp, it's pretty subtle but at least it's something.

What about for each a) adding the version number in the title header so they would read 'The Rust 0.x Language Tutorial' and b) putting some sort of link to the always-current version near the top? Are these something that could be changed in a template and then regenerate the old files? Maybe this could be useful for a lot of doc pages which will have similar issues with search engines and static links in blog posts etc.

Do you think there's anything that can be done on the search engine side?

[thestinger]

I think we should be purging any documentation from unsupported versions. It's still there in the repository history if someone wants it. Until we have a supported release with bug fixes (LLVM still doesn't!), this would mean only having the latest stable release and the master documentation on the site.

cc @brson

[bstrie]

In addition to purging the docs of unsupported versions, I think we should forbid Googlebots from crawling anything specific to any point release, including the most recent point release. Which is to say, Google searches for "rust tutorial" should never return anything that's not /doc/master. If people really want the docs for the most recent point releases, that's what the links on rust-lang.org are for. People getting routed to old docs is going to be a living hell for us if we don't take action while we can.

Though I guess first we should figure out if Github Pages even lets you provide a ROBOTS.txt file.

[brson]

I'm reluctant to just delete our old release artifacts, since these make up the history of Rust. It would be like deleting my old box of baseball cards sitting in my parents' basement - I need to know they're there for sentimental reasons.

What do I need to do to stop Google from crawling the old docs?

[brson]

I could possibly make all the old docs non-public, just keep them archived.

[thestinger]

@brson: using robots.txt would work, and wouldn't require modifying the html to mark them as noindex

[adrientetar]

@brson @thestinger robots.txt is a good idea:

User-agent: *
Disallow: /archive-folder/

OR, in the html head:
<meta name="robots" content="noindex,nofollow">

[adrientetar]

@brson Could you add a robots.txt to the website?
Whenever I google for various things about Rust the first result is tutorial but like 0.3, 0.5 or 0.6; think it could be well confusing for newcomers.

I think we can just stealthly let old doc on the website.

[thestinger]

The documentation is in the git repository history so I don't think we need to keep it around on rust-lang.org. It's quite easy to build the old tutorial/manual since it doesn't require building Rust for that revision.

[brson]

I added this file as static.rust-lang.org/robots.txt

User-agent: *
Disallow: /doc/0.3/
Disallow: /doc/0.4/
Disallow: /doc/0.5/
Disallow: /doc/0.6/
Disallow: /doc/0.7/
Disallow: /doc/0.8/