nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2393) 2.x patch for MD5 duplication issue addressed in NUTCH-2391
Date Sat, 10 Jun 2017 08:46:18 GMT


Sebastian Nagel commented on NUTCH-2393:

I don't know what happens if data is read and deserialized from any store supported by Gora.
It could be that the byte[] backing up the ByteBuffer contains other fields or other elements/cells
of the "content" column. [ByteBuffers are used because they are efficient|]
and that means to avoid copying data around. The underlying byte array may be, e.g., a mapped
file on disk or an Avro-serialized buffer but not a byte[] in the Java heap. Of course, in
ParserChecker it points to a byte[] with exactly the size of the page content. Or am I wrong,

If no objections I'll commit after the weekend. Thanks!

> 2.x patch for MD5 duplication issue addressed in NUTCH-2391
> -----------------------------------------------------------
>                 Key: NUTCH-2393
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl
>    Affects Versions: 2.3.1
>            Reporter: Kaidul Islam
>            Assignee: Kaidul Islam
>            Priority: Minor
>             Fix For: 2.4
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> Equivalent patch for 2.x for issue addressed in NUTCH-2391

This message was sent by Atlassian JIRA

View raw message