hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6803) ZlibCompressor hangs on close
Date Thu, 03 Jun 2010 00:11:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874874#action_12874874
] 

Eli Collins commented on HADOOP-6803:
-------------------------------------

Your right, I forgot that methods not fields have the "native" modifier. The "finish" field
is read from the native code and "finished" is updated there so it looks like it should work.
  What's strange is if I revert the change to ZlibCompressor it passes now, earlier when I
first added the test and ran with the native lib it hung with jstack showing the following:

{code}
"main" prio=10 tid=0x0000000040b19000 nid=0x36f0 runnable [0x00007f1257588000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.hadoop.io.compress.zlib.ZlibCompressor.deflateBytesDirect(Native Method)
        at org.apache.hadoop.io.compress.zlib.ZlibCompressor.compress(ZlibCompressor.java:359)
        - locked <0x00007f12495063e0> (a org.apache.hadoop.io.compress.GzipCodec$GzipZlibCompressor)
        at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:76)
        at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:86)
        at org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:97)
        at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:301)
        at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
        - locked <0x00007f12495167c8> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
        at java.io.BufferedWriter.close(BufferedWriter.java:248)
        - locked <0x00007f12495167c8> (a java.io.OutputStreamWriter)
        at org.apache.hadoop.io.compress.TestCodec.testGzipCodecWrite(TestCodec.java:653)
{code}


> ZlibCompressor hangs on close
> -----------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is
therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close
loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and
confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's
been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message