spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradyumn Agrawal <coderbond...@gmail.com>
Subject Re: Spark 3.0.1 not connecting with Hive 2.1.1
Date Sat, 09 Jan 2021 17:53:35 GMT
Hi Michael,
Thanks for references, although I had a hard time translating the 3rd one
as Google Translate
<https://translate.google.com/translate?sl=auto&tl=en&u=https://blog.csdn.net/Young2018/article/details/108871542>
of the csdn blog, it didn't work correctly and already went through the 1st
and 2nd earlier.
But I can see the CDH distribution is different in my case, it is CDH-6.3.1.

[image: image.png]

As you can see here in the screenshot, it is saying that *Invalid Methods
Name: get_table_req*
I am guessing that CDH Distribution has some changes on Hive Metastore
Client which is conflicting with Shim implementations of Spark. Although, I
couldn't debug a lot, it's totally a guesswork.
Would certainly like to know your and other views on this?

Thanks and Regards
Pradyumn Agrawal
Media.net (India)

On Sat, Jan 9, 2021 at 8:01 PM michael.yang <yangrong.jxufe@gmail.com>
wrote:

> Hi Pradyumn,
>
> We integrated Spark 3.0.1 with hive 2.1.1-cdh6.1.0 and it works fine to use
> spark-sql to query hive tables.
>
> Make sure you config spark-defaults.conf and spark-env.sh well and copy
> hive/hadoop related config files to spark conf folder.
>
> You can refer to below refrences for detail.
>
> https://spark.apache.org/docs/latest/building-spark.html
> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html
> https://blog.csdn.net/Young2018/article/details/108871542
>
> Thanks
> Michael Yang
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message