spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StanZhai <m...@zhaishidan.cn>
Subject Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0
Date Tue, 07 Feb 2017 14:00:57 GMT
>From thread dump page of Executor of WebUI, I found that there are about 1300
threads named  "DataStreamer for file
/test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_000069_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
in TIMED_WAITING state like this:
<http://apache-spark-developers-list.1001551.n3.nabble.com/file/n20881/QQ20170207-212340.png>


The exceed off-heap memory may be caused by these abnormal threads. 

This problem occurs only when writing data to the Hadoop(tasks may be killed
by Executor during writing).

Could this be related to  https://issues.apache.org/jira/browse/HDFS-9812
<https://issues.apache.org/jira/browse/HDFS-9812>  ?

It's may be a bug of Spark when killing tasks during writing data. What's
the difference between Spark 1.6.x and 2.1.0 in killing tasks?

This is a critical issue, I've worked on this for days.

Any help?



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20881.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message