hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12538) After set spark related config, SparkSession never get reused
Date Thu, 03 Dec 2015 19:13:11 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038373#comment-15038373
] 

Jimmy Xiang commented on HIVE-12538:
------------------------------------

Thought about it again, and chatted with Xuefu too. It seems to me there is no good fix for
this issue. Due to asynchronous, if a query is going on, if someone changes the spark setting
then submits another query, the previous query could be killed. Not sure how we can avoid
such thing properly.

One solution is to prevent any spark related setting changes once an async query is already
submitted in a session. [~xuefuz], [~szehon], what do you think?

> After set spark related config, SparkSession never get reused
> -------------------------------------------------------------
>
>                 Key: HIVE-12538
>                 URL: https://issues.apache.org/jira/browse/HIVE-12538
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 1.3.0
>            Reporter: Nemon Lou
>            Assignee: Nemon Lou
>         Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, HIVE-12538.3.patch, HIVE-12538.4.patch,
HIVE-12538.patch
>
>
> Hive on Spark yarn-cluster mode.
> After setting "set spark.yarn.queue=QueueA;" ,
> run the query "select count(*) from test"  3 times and you will find  3 different yarn
applications.
> Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING &
UNDEFINED state waiting for next work.
> And if you submit one more "select count(*) from test" ,the third one will be in FINISHED
& SUCCEEDED state and a new yarn application will start up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message