hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15822) zstd compressor can fail with a small output buffer
Date Mon, 08 Oct 2018 16:33:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642104#comment-16642104
] 

Jason Lowe commented on HADOOP-15822:
-------------------------------------

bq. do you think it's related? Or is it something different, maybe MR-specific? 

I do not think it is related.  The MapOutput buffer code is miscalculating how much buffer
space is remaining before it forces a spill.  In this failure case the buffer involved is
not dealing with compressed data, so it should not matter what codec is being used.  Have
you tried reproducing it with lz4 or no codec at all?

I'll dig a bit into the Jenkins test failures to see if they are somehow related. 

> zstd compressor can fail with a small output buffer
> ---------------------------------------------------
>
>                 Key: HADOOP-15822
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15822
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.9.0, 3.0.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Major
>         Attachments: HADOOP-15822.001.patch, HADOOP-15822.002.patch
>
>
> TestZStandardCompressorDecompressor fails a couple of tests on my machine with the latest
zstd library (1.3.5).  Compression can fail to successfully finalize the stream when a small
output buffer is used resulting in a failed to init error, and decompression with a direct
buffer can fail with an invalid src size error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message