Mercurial > hg-website

annotate static/robots.txt @ 415:b40d4f1881cf

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
say that the editor opens on commit and note how to avoid that.
author drak@kaverne
date Thu, 18 Sep 2014 10:55:08 +0200
parents 4dfad479ee98
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
403
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
1 User-agent: *
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
2 Crawl-delay: 100
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
3 Disallow: /hg
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
4 Disallow: /repo
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
5 Disallow: /bts/issue?
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
6
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
7 User-agent: geniebot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
8 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
9
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
10 User-agent: ShopWiki
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
11 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
12
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
13 User-agent: DotBot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
14 Disallow: /