view static/robots.txt @ 486:e3a73084715a

about: update URLs and prefer https:// The URLs for other version control systems on the 'About' page needed updating. Although they pretty much all resolve to the right places via redirection, it wouldn't hurt to have the links go straight to the source anyway.
author Bradley Jones <brdjns@gmx.us>
date Fri, 07 Sep 2018 13:54:33 +1200
parents 4dfad479ee98
children
line wrap: on
line source

User-agent: *
Crawl-delay: 100
Disallow: /hg
Disallow: /repo
Disallow: /bts/issue?

User-agent: geniebot
Disallow: /

User-agent: ShopWiki
Disallow: /

User-agent: DotBot
Disallow: /