spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From M Singh <>
Subject Apache Spark Structured Streaming - How to keep executor alive.
Date Fri, 23 Mar 2018 16:19:01 GMT
I am working on spark structured streaming (2.2.1) with kafka and want 100 executors to be
alive. I set spark.executor.instances to be 100.  The process starts running with 100 executors
but after some time only a few remain which causes backlog of events from kafka.  
I thought I saw a setting to keep the executors from being killed.  However, I am not able
to find that configuration in spark docs.  If anyone knows that setting, please let me know.

View raw message