hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharathkrishna Guruvayoor Murali (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-19814) RPC Server port is always random for spark
Date Wed, 15 Aug 2018 18:24:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581448#comment-16581448
] 

Bharathkrishna Guruvayoor Murali edited comment on HIVE-19814 at 8/15/18 6:23 PM:
----------------------------------------------------------------------------------

[~stakiar] , Added an additional test to verify this, added list of ports to HiveConf and
check if the port assignment is done accordingly when the SparkSessionManager creates the
client.
I'm exposing the server port as VisibleForTesting from SparkClientFactory.


was (Author: bharos92):
[~stakiar] , Added an additional test to verify this, added list of ports to HiveConf and
check if the port assignment is done accordingly when the SparkSessionManager creates the
client.

> RPC Server port is always random for spark
> ------------------------------------------
>
>                 Key: HIVE-19814
>                 URL: https://issues.apache.org/jira/browse/HIVE-19814
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 2.3.0, 3.0.0, 2.4.0, 4.0.0
>            Reporter: bounkong khamphousone
>            Assignee: Bharathkrishna Guruvayoor Murali
>            Priority: Major
>         Attachments: HIVE-19814.1.patch, HIVE-19814.2.patch
>
>
> RPC server port is always a random one. In fact, the problem is in RpcConfiguration.HIVE_SPARK_RSC_CONFIGS
which doesn't include SPARK_RPC_SERVER_PORT.
>  
> I've found this issue while trying to make hive-on-spark running inside docker.
>  
> HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf > SparkSessionManagerImpl.setup
and the latter call SparkClientFactory.initialize(conf) which initialize the rpc server. This
RPCServer is then used to create the sparkClient which use the rpc server port as --remote-port
arg. Since initiateSparkConf ignore SPARK_RPC_SERVER_PORT, then it will always be a random
port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message