spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
Date Mon, 23 Oct 2017 03:06:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214569#comment-16214569
] 

Saisai Shao commented on SPARK-22272:
-------------------------------------

Setting "spark.file.transferTo" to "false" will potentially affect the performance, that's
why we enabled this by default and leave an undocumented configurations if users has some
issues. The original JIRA is SPARK-3948, which is a kernel issue. I don't think we should
disable this by default, since it is JDK/Kernel specific. If you encountered such problems,
you can disable it in your cluster, but generally we don't want to disable this to hurt the
performance.

> killing task may cause the executor progress hang because of the JVM bug
> ------------------------------------------------------------------------
>
>                 Key: SPARK-22272
>                 URL: https://issues.apache.org/jira/browse/SPARK-22272
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.2
>         Environment: java version "1.7.0_75"
> hadoop version 2.5.0
>            Reporter: roncenzhao
>         Attachments: 26883.jstack, screenshot-1.png, screenshot-2.png
>
>
> JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693
> We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to merge all
partitions files when 'spark.file.transferTo' is true(default), so it may cause the jvm bug.
> When the driver send one task to this bad executor, the task will never run and as a
result the job will hang forever without handling.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message