spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Burak KÖSE (JIRA) <j...@apache.org>
Subject [jira] [Created] (SPARK-28230) Fix Spark Streaming gracefully shutdown on Hadopp 2.8.x
Date Tue, 02 Jul 2019 09:24:00 GMT
Burak KÖSE created SPARK-28230:
----------------------------------

             Summary: Fix Spark Streaming gracefully shutdown on Hadopp 2.8.x
                 Key: SPARK-28230
                 URL: https://issues.apache.org/jira/browse/SPARK-28230
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, Structured Streaming
    Affects Versions: 2.4.3
            Reporter: Burak KÖSE


Gracefully shutdown is not properly working on Hadoop 2.8.x. Hadoop introduces 10 seconds
timeout by default. This is hardcoded and not something that we can configure with Hadoop
settings.

 

[https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L51]

 

However, if we use 

 
{code:java}
public void addShutdownHook(Runnable shutdownHook, int priority, long timeout, TimeUnit unit){code}
 

in [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180,]

problem might be solved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message