spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Ki Kim <>
Subject KafkaReceiver executor in spark streaming job on YARN suddenly killed by ResourceManager
Date Fri, 02 Jan 2015 16:13:43 GMT
Hi, guys

I tried to run job of spark streaming with kafka on YARN.
My business logic is very simple.
Just listen on kafka topic and  write dstream to hdfs on each batch iteration.
After launching streaming job few hours, it works well. However suddenly died by ResourceManager.
ResourceManager send SIGTERM to kafka receiver executor and other executors also died without
any log error messages.
No Outofmemory exception, no other exceptions…. just died.

Do you know about this issue??
Please help me to resolve this problem.

My environments are below described.

* Scala : 2.10
* Spark : 1.1
* YARN : 2.5.0-cdh5.2.1
* CDH : 5.2.1 version

Thanks in advance.

Have a nice day~
View raw message