nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joris Rau (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2213) CommonCrawlDataDumper saves gzipped body in extracted form
Date Wed, 10 Feb 2016 10:44:18 GMT


Joris Rau commented on NUTCH-2213:

Okay. Thanks for your fast reply.
One last question: Do you consider saving a gzipped response body in an extracted form as
compliant with the WARC specs (since I could not find anything about it in the specs)?

> CommonCrawlDataDumper saves gzipped body in extracted form
> ----------------------------------------------------------
>                 Key: NUTCH-2213
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl, dumpers
>            Reporter: Joris Rau
>            Priority: Critical
>              Labels: easyfix
> I have downloaded [a WARC file|]
from the common crawl data. This file contains several gzipped responses which are stored
plaintext (without the gzip encoding).
> I used [warctools|] from Internet Archive
to extract the responses out of the WARC file. However this tool expects the Content-Length
field to match the actual length of the body in the WARC ([See the issue on github|]).
warctools uses a more up to date version of hanzo warctools which is recommended on the [Common
Crawl website|] under "Processing the file format".
> I have not been using Nutch and can therefore not say which versions are affected by
> After reading [the official WARC draft|]
I could not find out how gzipped content is supposed to be stored. However probably multiple
WARC file parsers will have an issue with this.
> It would be nice to know whether you consider this a bug and plan on fixing this and
whether this is a major issue which concerns most WARC files of the Common Crawl data or only
a small part.

This message was sent by Atlassian JIRA

View raw message