spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradyumn Agrawal <coderbond...@gmail.com>
Subject Re: Spark 3.0.1 not connecting with Hive 2.1.1
Date Sun, 10 Jan 2021 06:59:28 GMT
Hi DB Tsai,

Thanks for the JIRA link. I think this blocks me to the Hive end instead of
Spark.

Regards
Pradyumn Agrawal
Media.net (India)

On Sun, Jan 10, 2021 at 10:43 AM DB Tsai <d_tsai@apple.com> wrote:

> Hi Pradyumn,
>
> I think it’s because of a HMS client backward compatibility issue
> described here, https://issues.apache.org/jira/browse/HIVE-24608
>
> Thanks,
>
> DB Tsai | ACI Spark Core |  Apple, Inc
>
> On Jan 9, 2021, at 9:53 AM, Pradyumn Agrawal <coderbond007@gmail.com>
> wrote:
>
> Hi Michael,
> Thanks for references, although I had a hard time translating the 3rd one
> as Google Translate
> <https://translate.google.com/translate?sl=auto&tl=en&u=https://blog.csdn.net/Young2018/article/details/108871542>
> of the csdn blog, it didn't work correctly and already went through the 1st
> and 2nd earlier.
> But I can see the CDH distribution is different in my case, it is
> CDH-6.3.1.
>
> <image.png>
>
> As you can see here in the screenshot, it is saying that *Invalid Methods
> Name: get_table_req*
> I am guessing that CDH Distribution has some changes on Hive Metastore
> Client which is conflicting with Shim implementations of Spark. Although, I
> couldn't debug a lot, it's totally a guesswork.
> Would certainly like to know your and other views on this?
>
> Thanks and Regards
> Pradyumn Agrawal
> Media.net (India)
>
> On Sat, Jan 9, 2021 at 8:01 PM michael.yang <yangrong.jxufe@gmail.com>
> wrote:
>
>> Hi Pradyumn,
>>
>> We integrated Spark 3.0.1 with hive 2.1.1-cdh6.1.0 and it works fine to
>> use
>> spark-sql to query hive tables.
>>
>> Make sure you config spark-defaults.conf and spark-env.sh well and copy
>> hive/hadoop related config files to spark conf folder.
>>
>> You can refer to below refrences for detail.
>>
>> https://spark.apache.org/docs/latest/building-spark.html
>> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html
>> https://blog.csdn.net/Young2018/article/details/108871542
>>
>> Thanks
>> Michael Yang
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>

Mime
View raw message