lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Rafalovitch <arafa...@gmail.com>
Subject Re: Seeking a simple way to test my index.
Date Wed, 19 Sep 2018 18:05:41 GMT
Have you looked at Apache Nutch? Seems like the direct match for your
- growing - requirements and it does integrate with Solr. Or one of
the other solutions, like http://stormcrawler.net/
http://www.norconex.com/collectors/

Otherwise, this does not really feel like a Solr question.

Regards,
   Alex.

On 19 September 2018 at 14:01, Chip Calhoun <ccalhoun@aip.org> wrote:
> I've got a Solr instance which crawls roughly 3,500 seed pages, depth of 1, at 240 institutions,
all but 1 of which I don't control. I recrawl once a month or so. Naturally if one of the
sites I crawl changes, then I need to know to update my seed URLs. I've been checking this
by hand, which was tenable when my site was smaller, but is now completely unreasonable.
>
>
> Is there a way to test my index without actually having to run a lot of manual searches?
Perhaps an output I could skim? Any suggestions would be helpful.
>
>
> Thanks,
>
> Chip

Mime
View raw message