lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <>
Subject Re: [ANNOUNCE] Web Crawler
Date Mon, 15 Jul 2013 13:30:16 GMT
Lucene does not provide any capabilities for crawling websites. You would 
have to contact the Nutch project, the ManifoldCF project, or other web 
crawling projects.

As far as bypassing robots.txt, that is a very unethical thing to do. It is 
rather offensive that you seem to be suggesting that anybody on this mailing 
list would engage in such an unethical or unprofessional activity.

-- Jack Krupansky

-----Original Message----- 
From: Ramakrishna
Sent: Monday, July 15, 2013 9:13 AM
Subject: Re: [ANNOUNCE] Web Crawler


I'm trying nutch to crawl some web-sites. Unfortunately they restricted to
crawl their web-site by writing robots.txt. By using crawl-anywhere can I
crawl any web-sites irrespective of that web-sites robots.txt??? If yes, plz
send me the materials/links to study about crawl-anywhere or else plz
suggest me which are the crawlers to use to crawl web-sites without
bothering about robots.txt of that particular site. Its urgent plz reply as
soon as possible.

Thanks in advance

View this message in context:
Sent from the Lucene - Java Users mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail: 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message