nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris A. Mattmann (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (NUTCH-1949) Dump out the Nuth data into the Common Crawl format
Date Wed, 04 Mar 2015 04:13:05 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris A. Mattmann updated NUTCH-1949:
-------------------------------------
    Component/s: tool
                 storage
                 linkdb
                 crawldb

> Dump out the Nuth data into the Common Crawl format
> ---------------------------------------------------
>
>                 Key: NUTCH-1949
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1949
>             Project: Nutch
>          Issue Type: New Feature
>          Components: crawldb, linkdb, storage, tool
>            Reporter: Giuseppe Totaro
>            Assignee: Giuseppe Totaro
>             Fix For: 1.10
>
>         Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx, CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The {{CommonCrawlDataDumper}}
is a tool able to perfom the following steps:  
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or directory including
segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support and code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message