The Perl and Raku Conference 2025: Greenville, South Carolina - June 27-29 Learn more

BARBIE / CPAN-Testers-WWW-Reports-3.43 / vhost / html / robots.txt 

            

              
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

              
User-agent: Googlebot
Crawl-delay: 600
Disallow: /private/
Disallow: /*.yaml$
Disallow: /*.rss$
Disallow: /*.json$
User-agent: msnbot
Disallow: /
User-agent: DotBot
Crawl-delay: 600
Disallow: /private/
Disallow: /*.yaml$
Disallow: /*.rss$
Disallow: /*.json$
User-agent: *
Crawl-delay: 600
Disallow: /private/
Disallow: /*.yaml$
Disallow: /*.rss$
Disallow: /*.json$