nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2393) 2.x patch for MD5 duplication issue addressed in NUTCH-2391
Date Sat, 10 Jun 2017 08:46:18 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045461#comment-16045461
] 

Sebastian Nagel commented on NUTCH-2393:
----------------------------------------

I don't know what happens if data is read and deserialized from any store supported by Gora.
It could be that the byte[] backing up the ByteBuffer contains other fields or other elements/cells
of the "content" column. [ByteBuffers are used because they are efficient|http://www.kdgregory.com/index.php?page=java.byteBuffer]
and that means to avoid copying data around. The underlying byte array may be, e.g., a mapped
file on disk or an Avro-serialized buffer but not a byte[] in the Java heap. Of course, in
ParserChecker it points to a byte[] with exactly the size of the page content. Or am I wrong,
[~lewismc]?

If no objections I'll commit after the weekend. Thanks!

> 2.x patch for MD5 duplication issue addressed in NUTCH-2391
> -----------------------------------------------------------
>
>                 Key: NUTCH-2393
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2393
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl
>    Affects Versions: 2.3.1
>            Reporter: Kaidul Islam
>            Assignee: Kaidul Islam
>            Priority: Minor
>             Fix For: 2.4
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Equivalent patch for 2.x for issue addressed in NUTCH-2391



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message