nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From thammegowda <>
Subject [GitHub] nutch pull request: NUTCH-2250 : CommonCrawlDumper : Invalid forma...
Date Thu, 14 Apr 2016 09:44:51 GMT
GitHub user thammegowda opened a pull request:

    NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts

    ### Summary
    + Reads all parts of segments
    + writes single document per dump file
    ### What was the fix?
    1. Lists all files insides segments and selects all the parts matching to pattern part-[0-9]{5}
under the content directory. 
    2. CommonCrawlFormat implementation for "JACKSON" type does not clear state, instead it
keeps appending documents. This fix uses a new instance for each document (actually it was
hard to resolve the state issue because of too many options such as compressing and archiving).

You can merge this pull request into a Git repository by running:

    $ git pull NUTCH-2250

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #103
commit 47cc4e27a41a78753ed431685010da1987bca269
Author: Thamme Gowda <>
Date:   2016-04-14T09:38:30Z

    NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts
    + Reads all parts of segements
    + FIX : writes only one document to dump file


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message