spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18994) worker clean up app directory block the heartbeat sending
Date Tue, 03 Jan 2017 04:28:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794100#comment-15794100
] 

Apache Spark commented on SPARK-18994:
--------------------------------------

User 'liujianhuiouc' has created a pull request for this issue:
https://github.com/apache/spark/pull/16456

> worker clean up app directory block the heartbeat sending 
> ----------------------------------------------------------
>
>                 Key: SPARK-18994
>                 URL: https://issues.apache.org/jira/browse/SPARK-18994
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 1.6.1
>            Reporter: liujianhui
>
> h2. scene
> # when an application stop, it will trigger master send ApplicationFinish event to worker
> # worker will clean up the directories for the application, but it takes a long time
to delete them, as a result of the worker endpoint is Thread safe, and then it block the heartbeat
sending to master
> # master find the heartbeat expired for that worker, and then remove the worker, it will
re-launching the driver belong to that worker, actually that driver doesn't exit, finally
this will exist two app with same driver-id
>  the worker log
> {code}
> 2016-12-23 11:39:33,033  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Executor app-20161208222600-0020/200 finished with state KILLED exitStatus 137 | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:39:33,033  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Cleaning up local directories for application app-20161208222600-0020 | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.network.shuffle.ExternalShuffleBlockResolver
| Application app-20161208222600-0020 removed, cleanupLocalDirs = true | org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.applicationRemoved(ExternalShuffleBlockResolver.java:206)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect.
| org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | worker-register-master-threadpool-2 | org.apache.spark.deploy.worker.Worker
| Connecting to master bigdata-sirius-ser02.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | worker-register-master-threadpool-3 | org.apache.spark.deploy.worker.Worker
| Connecting to master bigdata-sirius-ser03.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect.
| org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | shuffle-client-1 | org.apache.spark.deploy.worker.Worker
| Successfully registered with master spark://bigdata-sirius-ser02.gz01:8090 | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | shuffle-client-1 | org.apache.spark.deploy.worker.Worker
| Worker cleanup enabled; old application directories will be deleted in: /data1/spark/work
| org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Not spawning another attempt to register with the master, since there is an attempt scheduled
already. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect.
| org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect.
| org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | worker-register-master-threadpool-3 | org.apache.spark.deploy.worker.Worker
| Connecting to master bigdata-sirius-ser03.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker
| Not spawning another attempt to register with the master, since there is an attempt scheduled
already. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021  |INFO | worker-register-master-threadpool-2 | org.apache.spark.deploy.worker.Worker
| Connecting to master bigdata-sirius-ser02.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> {code}
> *master* log
> {code}
> 2016-12-23 11:40:46,046  |WARN | dispatcher-event-loop-20 | org.apache.spark.deploy.master.Master
| Removing worker-20161208233029-100.69.146.57-26874 because we got no heartbeat in 60 seconds
| org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021  |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master
| Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking
it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message