incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Frovarp <>
Subject Re: can droid crawl this site
Date Mon, 28 Dec 2009 16:39:25 GMT
ray lukas wrote:
> Well this link crashes Nutch (redirection problem I would guess but have not
> proved it).. I really just need to get my hands on the HTML and I will feed
> it into my parsing and indexing systems. For this I just need a crawling
> mechanism that will give me the HTML for these types of links. Nutch is,
> wonderful but for this overkill and is unable t crawl these links, so I am
> looking at Droid as a solution. 
> I am not archiving anything, I am directly using the html in my java
> application. Can Droid crawl this site and return me the correct html. Could
> someone try it for me on their droid installation and let me know?
> Thanks guys.. 
> &xmcid=-12026&entry_point_id=3079198> 
I'm just in the evaluation phase of using Droids myself, but from what 
I've found it is quite flexible.

I would try running the SimpleRuntime code with the URL you listed and 
see if it gives you back your expected results. If that doesn't work, it 
may require some further work.


View raw message