flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vinoyang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9812) SpanningRecordSerializationTest fails on travis
Date Mon, 23 Jul 2018 07:06:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552412#comment-16552412
] 

vinoyang commented on FLINK-9812:
---------------------------------

Hi [~till.rohrmann] I have not analyzed the core problem of this issue. I just have a look
at this. The reason seems that we reused (write data) a closed spilling channel (FileChannel). 
the exception point is here : 
{code:java}
if (spillingChannel != null) {
   // spill to file
   ByteBuffer toWrite = segment.wrap(segmentPosition, toCopy);
   this.spillingChannel.write(toWrite);   //exception point
}{code}
There are two places may close the channel : 
{code:java}
spillingChannel.close();
{code}
one is : SpanningWrapper#clear method:
{code:java}
if (spillingChannel != null) {
   try {
      spillingChannel.close();
   }
   catch (Throwable t) {
      // ignore
   }
   spillingChannel = null;
}
{code}
this way, I think it's OK, because the close and set method come in pairs. 

another is : addNextChunkFromMemorySegment
{code:java}
if (accumulatedRecordBytes == recordLength) {
   // we have the full record
   if (spillingChannel == null) {
      this.serializationReadBuffer.setBuffer(buffer, 0, recordLength);
   }
   else {
      spillingChannel.close();

      BufferedInputStream inStream = new BufferedInputStream(new FileInputStream(spillFile),
2 * 1024 * 1024);
      this.spillFileReader = new DataInputViewStreamWrapper(inStream);
   }
}
{code}
this way may be a potential problem point, the spillingChannel is closed but is not been
set to null here.

So, if in other place the spillingChannel is not been re-initialized(there are some condition
to call the init method "createSpillingChannel"), it will happen this exception.

> SpanningRecordSerializationTest fails on travis
> -----------------------------------------------
>
>                 Key: FLINK-9812
>                 URL: https://issues.apache.org/jira/browse/FLINK-9812
>             Project: Flink
>          Issue Type: Bug
>          Components: Network, Tests, Type Serialization System
>    Affects Versions: 1.6.0
>            Reporter: Chesnay Schepler
>            Assignee: vinoyang
>            Priority: Critical
>             Fix For: 1.6.0
>
>
> https://travis-ci.org/zentol/flink/jobs/402744191
> {code}
> testHandleMixedLargeRecords(org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest)
 Time elapsed: 6.113 sec  <<< ERROR!
> java.nio.channels.ClosedChannelException: null
> 	at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
> 	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:199)
> 	at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer$SpanningWrapper.addNextChunkFromMemorySegment(SpillingAdaptiveSpanningRecordDeserializer.java:529)
> 	at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer$SpanningWrapper.access$200(SpillingAdaptiveSpanningRecordDeserializer.java:431)
> 	at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.setNextBuffer(SpillingAdaptiveSpanningRecordDeserializer.java:76)
> 	at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest.testSerializationRoundTrip(SpanningRecordSerializationTest.java:149)
> 	at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest.testSerializationRoundTrip(SpanningRecordSerializationTest.java:115)
> 	at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest.testHandleMixedLargeRecords(SpanningRecordSerializationTest.java:104)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message