spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hster Geguri <hster.investiga...@gmail.com>
Subject kinesis batches hang after YARN automatic driver restart
Date Mon, 02 Nov 2015 19:26:21 GMT
Hello Wonderful Sparks Peoples,

We are testing AWS Kinesis/Spark Streaming (1.5) failover behavior with
Hadoop/Yarn 2.6 and 2.71 and want to understand expected behavior.

When I manually kill a yarn application master/driver with a linux kill -9,
YARN will automatically relaunch another master that successfully reads in
the previous checkpoint.

However- more than half the time, the kinesis executors (5 second batches)
don't continue processing immediately.  I.e. batches of 0 events are queued
for  5-9 minutes before it starts reprocessing the stream again. When I
drill down to the current job which is hanging- it shows all stages/tasks
are complete. I would expect the automatically relaunched behavior to be
similar to as if I had manually done a resubmit with spark-submit where the
stream processing continues within a minute of launch.

Any input is highly appreciated.

Thanks much,
Heji

Mime
View raw message