annotate static/robots.txt @ 468:5d30d7a34569

css: attach the navigation bar to top of the page Currently the navigation bar doesn't looks good as there is some space between the top and bar. This patch removes that, bring the words in the center and align the search bar with the menu options.
author Pulkit Goyal <7895pulkit@gmail.com>
date Thu, 08 Dec 2016 13:45:16 +0530
parents 4dfad479ee98
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
403
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
1 User-agent: *
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
2 Crawl-delay: 100
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
3 Disallow: /hg
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
4 Disallow: /repo
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
5 Disallow: /bts/issue?
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
6
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
7 User-agent: geniebot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
8 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
9
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
10 User-agent: ShopWiki
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
11 Disallow: /
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
12
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
13 User-agent: DotBot
74b12fc80d9e Add robots.txt
David Soria Parra <davidsp@fb.com>
parents:
diff changeset
14 Disallow: /