spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Shankar <r...@unifisoftware.com>
Subject Re: Spark hive build and connectivity
Date Thu, 22 Oct 2020 16:47:26 GMT
Hello Mitch,
I am just trying to access hive tables from my hive 3.2.1 cluster using
spark. Basically i just want my spark-jobs to be able to access these hive
tables. I want to understand how spark jobs interact with hive to access
these tables.

- I see that whenever i build spark with hive support (-Phive
-Phive-thriftserver) , it gets built with hive 2.3.7 jars. So , will it be
ok if i access tables created using my hive 3.2.1 cluster ?
- Do i have to add hive 3.2.1 jars to spark's (SPARK_DIST_CLASSPATH) ?



On Thu, Oct 22, 2020 at 11:20 AM Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> Hi Ravi,
>
> What exactly are you trying to do?
>
> You want to enhance Spark SQl or you want to run Hive on Spark engine?
>
> HTH
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 22 Oct 2020 at 16:36, Ravi Shankar <ravi@unifisoftware.com> wrote:
>
>> Hello all,
>> I am trying to understand how the Spark SQL integration with hive works.
>> Whenever i build spark with -Phive -P hive-thriftserver options, i see that
>> it is packaged with hive-2.3.7*.jars and spark-hive*.jars. And the
>> documentation claims that spark can talk to different versions of hive. If
>> that is the case , what should i do if i have a hive 3.2.1 running on my
>> instance and i want my spark application to talk to that hive cluster.
>>
>> Does this mean i have to build spark with hive version 3.2.1 or like
>> the documentation states, is it enough if i just add the metastore jars to
>> spark-defaults.conf ?
>>
>> Should i add my hive 3.2.1 lib to the SPARK_DIST_CLASSPATH as well ? Will
>> there be conflicts between the hive 2.3.7 jars and the hive 3.2.1 jars i
>> will have in this case ?
>>
>>
>> Thanks !
>>
>

Mime
View raw message