nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki>
Subject Re: Fwd: [Collex] application#index (ActionController::RoutingError) "no route found to match \"/nines/ escape(document.title) u,\" with {:method=>:get}"
Date Tue, 10 Jul 2007 13:36:55 GMT
Erik Hatcher wrote:
> Please reply to me directly as well, as I'm not on the nutch-dev list 
> regularly.
> I'm curious ... Googlebot, Yahoo Slurp, and now CazoodleBot (based on 

Googlebot and Slurp, too ?? Hey, we're in a pretty good company! ;)

> Nutch) are hitting our site at and I get all sorts 
> of invalid links crawled.  Is our site doing something wrong in our 
> markup?  Or are all these crawlers flawed by hitting non-sensible URLs?

This is a side-effect of an unsophisticated method for extracting URLs 
from Javascript files - the extractor takes "likely" strings and tries 
to build absolute URLs out of them. In may cases the strings have 
nothing to do with URLs.

Please see NUTCH-505 for more details.

Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration  Contact: info at sigram dot com

View raw message