Mercurial > hg-website

annotate static/robots.txt @ 483:0978d4880577

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
downloads: skip trying to fill in button if it's not there
author Kevin Bullock <kbullock+mercurial@ringworld.org>
date Sun, 14 May 2017 21:15:12 -0500
parents 4dfad479ee98
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
403
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
1 User-agent: *
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
2 Crawl-delay: 100
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
3 Disallow: /hg
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
4 Disallow: /repo
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
5 Disallow: /bts/issue?
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
6
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
7 User-agent: geniebot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
8 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
9
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
10 User-agent: ShopWiki
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
11 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
12
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
13 User-agent: DotBot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
14 Disallow: /