hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <>
Subject [jira] [Commented] (HIVE-17111) Add TestLocalSparkCliDriver
Date Fri, 06 Oct 2017 21:10:01 GMT


Sahil Takiar commented on HIVE-17111:


{quote} 1. Other test driver class is generated, so I'm wondering why we actually have a nongenerated
class for TestLocalSparkCliDriver. {quote} I think that was the old way of making these classes,
now you can just directly create a new Java class to make a new CLI Driver. Unless I am misunderstanding
the question.

{quote} 2. With this change, are we able to run any .q test using this test driver class?
{quote}. Yes, the idea is that you can run any .q file with these new CLI Driver, which should
help developers debug any HoS issues.

> Add TestLocalSparkCliDriver
> ---------------------------
>                 Key: HIVE-17111
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-17111.1.patch
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but the HoS still
uses decides to use the RemoteHiveSparkClient rather than the LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
>     if (master.equals("local") || master.startsWith("local[")) {
>       // With local spark context, all user sessions share the same spark context.
>       return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
>     } else {
>       return new RemoteHiveSparkClient(hiveconf, sparkConf);
>     }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and sees that
it doesn't start with {{local[}} and then decides to use the RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed up some
of the tests, and also makes qtests easier to debug since everything will now be run in the
same process.

This message was sent by Atlassian JIRA

View raw message