Skip to content

Commit 0245b11

Browse files
committed
wiki: Delay abusive scrapers
1 parent 40c2e0e commit 0245b11

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

cookbooks/wiki/templates/default/robots.txt.erb

+9
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,15 @@ Disallow: /wiki/Special:Search
1919
User-agent: Exabot
2020
Crawl-delay: 60
2121

22+
User-Agent: ImagesiftBot
23+
Crawl-delay: 60
24+
25+
User-agent: SemrushBot
26+
Crawl-delay: 60
27+
28+
User-agent: SeznamBot
29+
Crawl-delay: 60
30+
2231
Sitemap: https://wiki.openstreetmap.org/sitemap-index-wiki.xml
2332

2433
Host: wiki.openstreetmap.org

0 commit comments

Comments
 (0)