nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2213) CommonCrawlDataDumper saves gzipped body in extracted form
Date Wed, 10 Feb 2016 15:42:18 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141024#comment-15141024
] 

ASF GitHub Bot commented on NUTCH-2213:
---------------------------------------

GitHub user jnioche opened a pull request:

    https://github.com/apache/nutch/pull/88

    NUTCH-2213 : do not store the headers verbatim if the response was compressed

    See discussion on [https://issues.apache.org/jira/browse/NUTCH-2213]. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jnioche/nutch NUTCH-2213

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nutch/pull/88.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #88
    
----
commit ba86ddfcc2431ae41accd8f56cc4ffcba16d1cc3
Author: Julien Nioche <julien@digitalpebble.com>
Date:   2016-02-10T15:38:26Z

    NUTCH-2213 : do not store the headers verbatim if the response was compressed

----


> CommonCrawlDataDumper saves gzipped body in extracted form
> ----------------------------------------------------------
>
>                 Key: NUTCH-2213
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2213
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl, dumpers
>            Reporter: Joris Rau
>            Assignee: Julien Nioche
>            Priority: Critical
>              Labels: easyfix
>
> I have downloaded [a WARC file|https://aws-publicdatasets.s3.amazonaws.com/common-crawl/crawl-data/CC-MAIN-2015-40/segments/1443738099622.98/warc/CC-MAIN-20151001222139-00240-ip-10-137-6-227.ec2.internal.warc.gz]
from the common crawl data. This file contains several gzipped responses which are stored
plaintext (without the gzip encoding).
> I used [warctools|https://github.com/internetarchive/warctools] from Internet Archive
to extract the responses out of the WARC file. However this tool expects the Content-Length
field to match the actual length of the body in the WARC ([See the issue on github|https://github.com/internetarchive/warctools/pull/14#issuecomment-182048962]).
warctools uses a more up to date version of hanzo warctools which is recommended on the [Common
Crawl website|https://commoncrawl.org/the-data/get-started/] under "Processing the file format".
> I have not been using Nutch and can therefore not say which versions are affected by
this.
> After reading [the official WARC draft|http://archive-access.sourceforge.net/warc/warc_file_format-0.9.html]
I could not find out how gzipped content is supposed to be stored. However probably multiple
WARC file parsers will have an issue with this.
> It would be nice to know whether you consider this a bug and plan on fixing this and
whether this is a major issue which concerns most WARC files of the Common Crawl data or only
a small part.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message