nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <>
Subject [jira] [Commented] (NUTCH-1997) Add CBOR "magic header" to CommonCrawlDataDumper output
Date Sat, 25 Apr 2015 16:50:39 GMT


Hudson commented on NUTCH-1997:

FAILURE: Integrated in Nutch-trunk #3089 (See [])
NUTCH-1997: Fix for Add CBOR magic header to CommonCrawlDataDumper output contributed by Giuseppe
Totaro, and Luke Sh. (mattmann:
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/java/org/apache/nutch/tools/

> Add CBOR "magic header" to CommonCrawlDataDumper output
> -------------------------------------------------------
>                 Key: NUTCH-1997
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: tool
>            Reporter: Giuseppe Totaro
>            Assignee: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 1.10
>         Attachments: NUTCH-1997.patch
> For each file extracted from Nutch crawled data, {{CommonCrawlDataDumper}} wraps a single
string value, representing the JSON text, into CBOR. 
> For instance, using the Unix {{hexdump}} tool, we can see that, as expected, the first
byte of all files is "0x7F" (the first three bits are "011", that is the major type for strings,
and the following 5 bits are "11010", meaning a uint32_t encodes the length of following text),
and the following 4 bytes (single-precision float) encodes the right length of file (as described
in [RFC7049|]). Therefore, a CBOR tag is currently included
into the file (a list of cbor tags is available [here|]).
> In order to add support for CBOR detection using Apache Tika (as described in [TIKA-1610|]),
it would be great if {{CommonCrawlDataDumper}} tool is able to add the self-describing CBOR
"magic header" ([Tag 55799|]) to CBOR-encoded
output files. 
> Thanks a lot [~Lukeliush] for this great research. Thanks [~chrismattmann] for supporting
me on this work.

This message was sent by Atlassian JIRA

View raw message