Burak KÖSE created SPARK-28230:
----------------------------------
Summary: Fix Spark Streaming gracefully shutdown on Hadopp 2.8.x
Key: SPARK-28230
URL: https://issues.apache.org/jira/browse/SPARK-28230
Project: Spark
Issue Type: Bug
Components: Spark Core, Structured Streaming
Affects Versions: 2.4.3
Reporter: Burak KÖSE
Gracefully shutdown is not properly working on Hadoop 2.8.x. Hadoop introduces 10 seconds
timeout by default. This is hardcoded and not something that we can configure with Hadoop
settings.
[https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L51]
However, if we use
{code:java}
public void addShutdownHook(Runnable shutdownHook, int priority, long timeout, TimeUnit unit){code}
in [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180,]
problem might be solved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|