spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <d_t...@apple.com.INVALID>
Subject Re: Spark 3.0.1 not connecting with Hive 2.1.1
Date Sun, 10 Jan 2021 05:13:41 GMT
Hi Pradyumn,

I think it’s because of a HMS client backward compatibility issue described here, https://issues.apache.org/jira/browse/HIVE-24608

Thanks,

DB Tsai | ACI Spark Core |  Apple, Inc

> On Jan 9, 2021, at 9:53 AM, Pradyumn Agrawal <coderbond007@gmail.com> wrote:
> 
> Hi Michael, 
> Thanks for references, although I had a hard time translating the 3rd one as Google Translate
<https://translate.google.com/translate?sl=auto&tl=en&u=https://blog.csdn.net/Young2018/article/details/108871542>
of the csdn blog, it didn't work correctly and already went through the 1st and 2nd earlier.

> But I can see the CDH distribution is different in my case, it is CDH-6.3.1.
> 
> <image.png>
> 
> As you can see here in the screenshot, it is saying that Invalid Methods Name: get_table_req
> I am guessing that CDH Distribution has some changes on Hive Metastore Client which is
conflicting with Shim implementations of Spark. Although, I couldn't debug a lot, it's totally
a guesswork.
> Would certainly like to know your and other views on this?
> 
> Thanks and Regards
> Pradyumn Agrawal
> Media.net (India)
> 
> On Sat, Jan 9, 2021 at 8:01 PM michael.yang <yangrong.jxufe@gmail.com <mailto:yangrong.jxufe@gmail.com>>
wrote:
> Hi Pradyumn,
> 
> We integrated Spark 3.0.1 with hive 2.1.1-cdh6.1.0 and it works fine to use
> spark-sql to query hive tables.
> 
> Make sure you config spark-defaults.conf and spark-env.sh well and copy
> hive/hadoop related config files to spark conf folder.
> 
> You can refer to below refrences for detail.
> 
> https://spark.apache.org/docs/latest/building-spark.html <https://spark.apache.org/docs/latest/building-spark.html>
> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html <https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html>
> https://blog.csdn.net/Young2018/article/details/108871542 <https://blog.csdn.net/Young2018/article/details/108871542>
> 
> Thanks
> Michael Yang
> 
> 
> 
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ <http://apache-spark-user-list.1001560.n3.nabble.com/>
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org <mailto:user-unsubscribe@spark.apache.org>
> 


Mime
View raw message