Mercurial > hg-website

annotate static/robots.txt @ 441:674632ee2ceb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merged in adduc/hg-website/adduc/update-link-to-matt-mackalls-repo-to-ref-1451564991479 (pull request #10) Update link to Matt Mackall's repo to reflect use of HTTPS.
author David Soria Parra <experimentalworks@gmail.com>
date Fri, 01 Jan 2016 14:45:08 +0100
parents 4dfad479ee98
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
403
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
1 User-agent: *
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
2 Crawl-delay: 100
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
3 Disallow: /hg
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
4 Disallow: /repo
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
5 Disallow: /bts/issue?
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
6
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
7 User-agent: geniebot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
8 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
9
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
10 User-agent: ShopWiki
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
11 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
12
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
13 User-agent: DotBot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
14 Disallow: /