lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <>
Subject Re: Apache Nutch Crawling Dynamic Products on Pages
Date Wed, 18 Sep 2013 23:58:34 GMT

: We have a page that contains some dynamic products. So when I am 
: crawling site, that page gets crawled but when searching for that 
: particular product, its showing no results. Can you guys help me, How 
: can I crawl these dynamic product on page? Is there any way to achieve 
: this and also can we use sitemaps with Nutch?

it's not clear wether your problem is that nutch isn't crawling the pages 
that you want because of the dynamic URLs, or if nutch is crawling the 
pages that you want, but then you are having trouble searching and finding 
those pages because of how your (Solr?) index is configured.

either way: i would suggest you start by addressing your question to the 
nutch user list -- but you almost certainly need to provide more details 
about the types or urls, types of content, how you are running nutch's 
crawl command, logs you see, results you see when executing searches 

In the event that you confirm nutch is crawling the documents and putting 
them -- including all the data you expect -- into a solr index, but they 
are not being returned by the search terms you expect, then please 
start a new thread on the solr-user@lucene mailing list with details about 
your solr configuration, the documents in your index (the specifics of 
how/where nutch got those documents probably isn't relevant on solr-user) 
and what results you get from your query...


View raw message