spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <andrew.row...@thomsonreuters.com>
Subject EOFException on History server reading in progress lz4
Date Thu, 03 Sep 2015 16:32:47 GMT
I'm trying to solve a problem of the history server spamming my logs with
EOFExceptions when it tries to read a history file from HDFS that is both
lz4 compressed and incomplete. The actual exception is:

java.io.EOFException: Stream ended prematurely
	at
net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
	at
net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:192)
	at
net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at
scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67
)
	at
org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:
55)
	at
org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$hi
story$FsHistoryProvider$$replay(FsHistoryProvider.scala:443)
	at
org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistor
yProvider.scala:278)
	at
org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistor
yProvider.scala:275)
	at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.sc
ala:251)
	at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.sc
ala:251)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:5
9)
	at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
	at
scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
	at
org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$hi
story$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:275
)
	at
org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$a
non$2.run(FsHistoryProvider.scala:209)
	at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
	at java.lang.Thread.run(Thread.java:745)

The bit I'm struggling with is handling this in ReplayListenerBus.scala - I
tried adding the following to the try/catch:

case eof: java.io.EOFException =>
    logWarning(s"EOFException (probably due to incomplete lz4) at
$sourceName", eof)

but this never seems to get triggered - it still dumps the whole exception
out to the log.

I feel like there's something basic I'm missing for the exception not to be
caught by the try/catch in ReplayListenerBus. Can anyone point me in the
right direction?

Thanks,

Andrew

Mime
View raw message