hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] kishendas opened a new pull request #2949: HADOOP-17657: implement StreamCapabilities in SequenceFile.Writer and fall back to flush, if hflush is not supported
Date Fri, 23 Apr 2021 21:14:55 GMT

kishendas opened a new pull request #2949:
URL: https://github.com/apache/hadoop/pull/2949


   Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on S3 from Tez,
which internally calls org.apache.hadoop.io.SequenceFile$Writer.hflush ->  org.apache.hadoop.fs.FS
DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not implemented and throws
java.lang.UnsupportedOperationException. 
   
   
   bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" level="ERROR" thread="HistoryEventHandlingThread"]
Thread Thread[HistoryEventHandlingThread, 5,main] threw an Exception.^Mjava.lang.UnsupportedOperationException:
S3A streams are not Syncable^M at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M
at org.apache.hadoop.fs.FS DataOutputStream.hflush(FSDataOutputStream.java:136)^M at org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M
at org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr iter.java:64)^M
at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M
at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han dleEvent(ProtoHistoryLoggingService.java:198)^M
at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M
at java.lang.Thread.r
 un(Thread.java:748)^M
   
   In order to fix this issue we should implement StreamCapabilities in SequenceFile.Writer.
Also, we should fall back to flush(), if hflush() is not supported. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message