nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "behnam nikbakht (Commented) (JIRA)" <>
Subject [jira] [Commented] (NUTCH-1282) linkdb scalability
Date Sat, 03 Mar 2012 13:15:56 GMT


behnam nikbakht commented on NUTCH-1282:

another option is when we construct web graph for implementing advanced scoring methods, and
then, we can extract anchors from inlinks in web graph and no need to linkdb.
> linkdb scalability
> ------------------
>                 Key: NUTCH-1282
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: linkdb
>    Affects Versions: 1.4
>            Reporter: behnam nikbakht
> as described in NUTCH-1054, the linkdb is optional in solrindex and it's usage is only
for anchor and not impact on scoring. 
> as seemed, size of linkdb in incremental crawl grow very fast and make it unscalable
for huge size of web sites.
> so, here is two choises, one, ignore invertlinks and linkdb from crawl, and second, make
it scalable
> in invertlinks, there is 2 jobs, first for construct new linkdb from new parsed segments,
and second for merge new linkdb with old linkdb. the second job is unscalable and we can ignore
it with this changes in solrIndex:
> in the class IndexerMapReduce, reduce method, if fetchDatum == null or dbDatum == null
or parseText == null or parseData == null, then add anchor to doc and update solr (no insert)
> here also some changes required to NutchDocument.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message