spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christophe Préaud (JIRA) <j...@apache.org>
Subject [jira] [Updated] (SPARK-15401) Spark Thrift server creates empty directories in tmp directory on the driver
Date Thu, 19 May 2016 09:37:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Christophe Préaud updated SPARK-15401:
--------------------------------------
    Description: 
Each connection to the Spark thrift server (e.g. using beeline) creates two empty directories
in the tmp directory on the driver which are never removed:

cd <tmp directory>
ls -ltd *_resources | wc -l && /opt/spark/bin/beeline -u jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
-n kookel -e '!quit' && ls -ltd *_resources | wc -l
9080
Connecting to jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
Connected to: Spark SQL (version 1.6.1)
Driver: Spark Project Core (version 1.6.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Closing: 0: jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
Beeline version 1.6.1 by Apache Hive
9082

Those directories accumulates over time and are not removed:
ls -ld *_resources | wc -l
9064

And they are indeed empty:
find *_resources -type f | wc -l
0

  was:
Each connection to the Spark thrift server (e.g. using beeline) creates two empty directories
in the tmp directory which are never removed:

cd <tmp directory>
ls -ltd *_resources | wc -l && /opt/spark/bin/beeline -u jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
-n kookel -e '!quit' && ls -ltd *_resources | wc -l
9080
Connecting to jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
Connected to: Spark SQL (version 1.6.1)
Driver: Spark Project Core (version 1.6.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Closing: 0: jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
Beeline version 1.6.1 by Apache Hive
9082

Those directories accumulates over time and are not removed:
ls -ld *_resources | wc -l
9064

And they are indeed empty:
find *_resources -type f | wc -l
0


> Spark Thrift server creates empty directories in tmp directory on the driver
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-15401
>                 URL: https://issues.apache.org/jira/browse/SPARK-15401
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.1
>            Reporter: Christophe Préaud
>            Priority: Minor
>
> Each connection to the Spark thrift server (e.g. using beeline) creates two empty directories
in the tmp directory on the driver which are never removed:
> cd <tmp directory>
> ls -ltd *_resources | wc -l && /opt/spark/bin/beeline -u jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
-n kookel -e '!quit' && ls -ltd *_resources | wc -l
> 9080
> Connecting to jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
> Connected to: Spark SQL (version 1.6.1)
> Driver: Spark Project Core (version 1.6.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Closing: 0: jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
> Beeline version 1.6.1 by Apache Hive
> 9082
> Those directories accumulates over time and are not removed:
> ls -ld *_resources | wc -l
> 9064
> And they are indeed empty:
> find *_resources -type f | wc -l
> 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message