annotate static/robots.txt @ 463:4ec689699207

frontpage: use example repo from mercurial-scm.org The rendered result looks like this: """ $ hg clone https://www.mercurial- scm.org/repo/hello """ (It will be copied without any word breaks, the break is only visual.) The actual repo doesn't exist yet, but really should already, because the first thing new users will see when they try the old URL with 3.9-rc+ is a lengthy error message that Mercurial "could not negotiate a common security protocol (tls1.1+) with selenic.com..." Classy.
author Anton Shestakov <av6@dwimlabs.net>
date Fri, 22 Jul 2016 18:46:43 +0800
parents 4dfad479ee98
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
403
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
1 User-agent: *
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
2 Crawl-delay: 100
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
3 Disallow: /hg
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
4 Disallow: /repo
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
5 Disallow: /bts/issue?
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
6
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
7 User-agent: geniebot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
8 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
9
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
10 User-agent: ShopWiki
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
11 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
12
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
13 User-agent: DotBot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
14 Disallow: /