nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-1113) Merging segments causes URLs to vanish from crawldb/index?
Date Fri, 10 Jan 2014 17:30:55 GMT


Sebastian Nagel commented on NUTCH-1113:

Hi [~markus17], I've run your unit tests and also tried again to find anything using test
data used to verify NUTCH-1520: SegmentMerger works as expected (in local mode). Strange why
it only happens in distributed mode.
# "to ignore linked crawl datums completely" applies only to crawl_fetch dir, ie. linked items
from redirects?
# after looking again at SegmentMerger: What if both fetch_success and linked datum are contained
in one segment? That may happen if we find a redirect in the same round the target is fetched
(found the link before). We do not consider this case by testing {{lastFname.compareTo(sp.segmentName)
< 0}}. If the linked datum comes first it is not overwritten by the fetch datum. I'll try
to verify this in a test unit.

> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>                 Key: NUTCH-1113
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.3
>            Reporter: Edward Drapkin
>            Priority: Blocker
>             Fix For: 1.9
>         Attachments: NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch,
NUTCH-1113-junit.patch, NUTCH-1113-trunk.patch, merged_segment_output.txt, unmerged_segment_output.txt
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene indexing
code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up in the
index vs. when I crawl without merging the segments.  Somehow the segment merger causes me
to lose ~20% of my crawl database!

This message was sent by Atlassian JIRA

View raw message