nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Moreno Feltscher (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2496) Speed up link inversion step in crawling script
Date Fri, 12 Jan 2018 23:35:00 GMT


Moreno Feltscher commented on NUTCH-2496:

One thing I found out is that if I do the link inversion step after all the iterations are
done it takes a lot less time. Would it be feasible to move the link inversion and indexing
step out of the loop and do it only once in the end? Any thoughts about this?

> Speed up link inversion step in crawling script
> -----------------------------------------------
>                 Key: NUTCH-2496
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Moreno Feltscher
>            Assignee: Lewis John McGibbney
> While working on a project where I have to index a huge number of URLs I encountered
an issue with the link inversion step of the crawling script. A while ago Ian Lopata stumbled
upon the same issue as described here:
> {quote}
> I am running the invertlinks step in my Nutch 1.6 based crawl process on a 
> single node.  I run invertlinks only because I need the Inlinks in the 
> indexer step so as to store them with the document.  I do not need the 
> anchor text and I am not scoring.  I am finding that invertlinks (and more 
> specifically the merge of the linkdb) takes a long time - about 30 minutes 
> for a crawl of around 150K documents.  I am looking for ways that I might 
> shorten this processing time.  Any suggestions? 
> {quote}
> Back then [~wastl-nagel] suggested turning off the normalizers and filters during the
inversion step which speeds up the process a bunch.
> In my case however I kind of depend on those so this is no real solution.
> I opened this issue here in order to get some feedback on how we could improve things
in a crawl script and speed up the process.

This message was sent by Atlassian JIRA

View raw message