spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christophe Préaud (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-15401) Spark Thrift server creates empty directories in tmp directory on the driver
Date Thu, 19 May 2016 09:44:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290804#comment-15290804
] 

Christophe Préaud edited comment on SPARK-15401 at 5/19/16 9:43 AM:
--------------------------------------------------------------------

I suspect the origin of the problem may come from the [createTempDir|https://github.com/apache/spark/blob/v1.6.1/core/src/main/scala/org/apache/spark/util/Utils.scala#L231#L241]
method: directories created by this method are automatically deleted when the VM shuts down,
however the Spark thrift server (at least the one on our cluster) is never shut down.


was (Author: preaudc):
I suspect the origin of the problem may come from the [createTempDir|https://github.com/apache/spark/blob/v1.6.1/core/src/main/scala/org/apache/spark/util/Utils.scala#L231#L241]
method: directories create by this method are automatically deleted when the VM shuts down,
however the Spark thrift server (at least the one on our cluster) is never shut down.

> Spark Thrift server creates empty directories in tmp directory on the driver
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-15401
>                 URL: https://issues.apache.org/jira/browse/SPARK-15401
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.1
>            Reporter: Christophe Préaud
>            Priority: Minor
>
> Each connection to the Spark thrift server (e.g. using beeline) creates two empty directories
in the tmp directory on the driver which are never removed:
> cd <tmp directory>
> ls -ltd *_resources | wc -l && /opt/spark/bin/beeline -u jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
-n kookel -e '!quit' && ls -ltd *_resources | wc -l
> 9080
> Connecting to jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
> Connected to: Spark SQL (version 1.6.1)
> Driver: Spark Project Core (version 1.6.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Closing: 0: jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
> Beeline version 1.6.1 by Apache Hive
> 9082
> Those directories accumulates over time and are not removed:
> ls -ld *_resources | wc -l
> 9064
> And they are indeed empty:
> find *_resources -type f | wc -l
> 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message