nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathijs Homminga <mathijs.hommi...@kalooga.com>
Subject Re: Nutch ignores robots.txt
Date Thu, 03 Nov 2011 08:39:14 GMT
Hello Max,

(Besides the fact that the this client seems to have a broken random URL generator)

Crawlers (like Nutch clients) may not always obey robot rules. If Nutch is not configured
properly, it will not recognize your Nutch entry in your robots.txt file.
If the requests come from a limited number of IP addresses (78.46.90.27, 188.40.65.130) ,
you could try to ban those. Or try to contact the people behind this address, they seem to
live in Germany as well.

Good luck,
Mathijs 


On Nov 2, 2011, at 14:13 , Maximilian Laurenz wrote:

> Hi there,
> Because a Nutch client seems to cause errors on our web server, we changed robots.txt
for www.kinoundco.de to disallow Nutch. Unfortunately we still get requests:
>  
> 2011-11-01 05:50:35 W3SVC4 MB 62.128.28.16 GET /Rango/+(Math.random()*100000)+ - 80 -
188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 373 263 0
> 2011-11-01 05:52:23 W3SVC4 MB 62.128.28.16 GET /default.aspx aspxerrorpath=/Atemlos-Gefaehrliche-Wahrheit/+(Math.random()*100000)+
80 - 188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 200 0 32449 315 15
> 2011-11-01 05:59:15 W3SVC4 MB 62.128.28.16 GET /Kleine-wahre-Luegen/+(Math.random()*100000)+
- 80 - 188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 401 277 15
> 2011-11-01 05:59:31 W3SVC4 MB 62.128.28.16 GET /Nachtasyl/+(Math.random()*100000)+ -
80 - 188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 381 267 0
> 2011-11-01 06:35:30 W3SVC4 MB 62.128.28.16 GET /Auf-der-anderen-Seite-der-Leinwand-100-Jahre-Moviemento/+(Math.random()*100000)+
- 80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 473 313 15
> 2011-11-01 06:35:33 W3SVC4 MB 62.128.28.16 GET /Zoowaerter/+(Math.random()*100000)+ -
80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 383 268 31
> 2011-11-01 06:35:40 W3SVC4 MB 62.128.28.16 GET /default.aspx aspxerrorpath=/Sascha/+(Math.random()*100000)+
80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 200 0 32449 292 62
> 2011-11-01 06:36:35 W3SVC4 MB 62.128.28.16 GET /Beschissenheit-der-Dinge/+(Math.random()*100000)+
- 80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 411 282 31
> 2011-11-01 06:38:14 W3SVC4 MB 62.128.28.16 GET /Auf-der-Suche/+(Math.random()*100000)+
- 80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 389 271 15
> 2011-11-01 06:39:55 W3SVC4 MB 62.128.28.16 GET /Fall/+(Math.random()*100000)+ - 80 -
78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 371 262 15
> 2011-11-01 07:51:10 W3SVC4 MB 62.128.28.16 GET /Midnight-in-Paris/+(Math.random()*100000)+
- 80 - 176.9.26.236 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 397 275 0
> 2011-11-01 07:51:40 W3SVC4 MB 62.128.28.16 GET /Betty-Anne-Waters/+(Math.random()*100000)+
- 80 - 176.9.26.236 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 397 275 15
>  
> Sincerely,
> Max
> Maximilian Laurenz
> S&L Medien Gruppe GmbH
> Aidenbachstraße 54
> 81379 München
> Tel. +49 89 790862-49
> Fax +49 89 790862-55
> mlaurenz@slmedien.de
> http://www.slmedien.de
> S&L Medien Gruppe GmbH | Geschäftsführung: Maria-Theresia von Seidlein, Torsten
Weihrich, Olaf Wiehler | Sitz der Gesellschaft: München | Amtsgericht München | HRB 99977
>  


Mime
View raw message