0
0
mirror of https://github.com/wagtail/wagtail.git synced 2024-11-28 17:13:31 +01:00
wagtail/docs/robots.txt
Matt Westcott ab656424b7 Reinstate spidering of other documentation versions besides stable
These are now returning the correct canonical URL (thanks to a Cloudflare worker script), and letting search engines spider them again will hopefully boost visibility of that canonical URL.
2021-02-02 15:15:43 +00:00

6 lines
79 B
Plaintext

User-agent: *
Disallow: /*/pdf/
Sitemap: https://docs.wagtail.io/sitemap.xml