Skip to content

Start a sitemap. #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 18, 2020
Merged

Start a sitemap. #99

merged 2 commits into from
Nov 18, 2020

Conversation

JulienPalard
Copy link
Member

@JulienPalard JulienPalard commented Nov 16, 2020 •
edited
Loading

It looks like this:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:xhtml="http://www.w3.org/1999/xhtml">
  
  <url>
    <loc>https://docs.python.org/3.5/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3.5/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3.5/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3.5/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3.5/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3.5/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3.5/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3.5/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3.5/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3.5/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3.5/"/>
    <changefreq>yearly</changefreq>
  </url>
  
  <url>
    <loc>https://docs.python.org/3.6/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3.6/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3.6/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3.6/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3.6/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3.6/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3.6/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3.6/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3.6/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3.6/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3.6/"/>
    <changefreq>yearly</changefreq>
  </url>
  
  <url>
    <loc>https://docs.python.org/3.7/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3.7/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3.7/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3.7/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3.7/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3.7/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3.7/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3.7/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3.7/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3.7/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3.7/"/>
    <changefreq>yearly</changefreq>
  </url>
  
  <url>
    <loc>https://docs.python.org/3.8/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3.8/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3.8/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3.8/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3.8/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3.8/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3.8/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3.8/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3.8/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3.8/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3.8/"/>
    <changefreq>daily</changefreq>
  </url>
  
  <url>
    <loc>https://docs.python.org/3.9/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3.9/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3.9/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3.9/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3.9/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3.9/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3.9/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3.9/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3.9/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3.9/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3.9/"/>
    <changefreq>daily</changefreq>
  </url>
  
  <url>
    <loc>https://docs.python.org/3.10/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3.10/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3.10/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3.10/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3.10/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3.10/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3.10/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3.10/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3.10/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3.10/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3.10/"/>
    <changefreq>daily</changefreq>
  </url>
  
  <url>
    <loc>https://docs.python.org/3/</loc>
    <xhtml:link rel="alternate" hreflang="zh-CN" href="https://docs.python.org/zh-cn/3/"/>
    <xhtml:link rel="alternate" hreflang="en" href="https://docs.python.org/en/3/"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://docs.python.org/fr/3/"/>
    <xhtml:link rel="alternate" hreflang="es" href="https://docs.python.org/es/3/"/>
    <xhtml:link rel="alternate" hreflang="ja" href="https://docs.python.org/ja/3/"/>
    <xhtml:link rel="alternate" hreflang="ko" href="https://docs.python.org/ko/3/"/>
    <xhtml:link rel="alternate" hreflang="pl" href="https://docs.python.org/pl/3/"/>
    <xhtml:link rel="alternate" hreflang="id" href="https://docs.python.org/id/3/"/>
    <xhtml:link rel="alternate" hreflang="zh-TW" href="https://docs.python.org/zh-tw/3/"/>
    <xhtml:link rel="alternate" hreflang="pt-BR" href="https://docs.python.org/pt-br/3/"/>
    <changefreq>daily</changefreq>
  </url>
</urlset>

It also add a robots.txt:

Sitemap: https://docs.python.org/sitemap.xml

# Prevent development and old documentation from showing up in search results.
User-agent: *
Disallow: /dev
Disallow: /release

# Disallow EOL versions
Disallow: /2/
Disallow: /2.0/
Disallow: /2.1/
Disallow: /2.2/
Disallow: /2.3/
Disallow: /2.4/
Disallow: /2.5/
Disallow: /2.6/
Disallow: /2.7/
Disallow: /3.0/
Disallow: /3.1/
Disallow: /3.2/
Disallow: /3.3/
Disallow: /3.4/

to advertise the sitemap, and disallow old versions (EOL version that do not have a canonical to /3/).

Closes #94, #56

@Seluj78
Copy link

Seluj78 commented Nov 18, 2020

Closes #94, #99
Did you mean #56 instead of 99 ?

Seluj78
Seluj78 reviewed Nov 18, 2020
View reviewed changes
Copy link

@Seluj78 Seluj78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good ! I feel we should merge it and wait for the search engines to update their DBs to see how efficient this is

Seluj78
Seluj78 approved these changes Nov 18, 2020
View reviewed changes
Copy link

@Seluj78 Seluj78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I meant to approve

@JulienPalard JulienPalard merged commit f9e1344 into python:master Nov 18, 2020
@AA-Turner AA-Turner mentioned this pull request Oct 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Seluj78 Seluj78 Seluj78 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

robots.txt should steer search engines away from old docs
2 participants
@JulienPalard @Seluj78