Thanks Jorn.

Just to be clear they get Hive working with Spark 1.6 out of the box (binary download)? The usual work-around is to build your own package and get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.


Cheers



On 26 March 2016 at 22:08, Jörn Franke <jornfranke@gmail.com> wrote:
If you check the newest Hortonworks distribution then you see that it generally works. Maybe you can borrow some of their packages. Alternatively it should be also available in other distributions.

On 26 Mar 2016, at 22:47, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

Hi,

I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign that Hive can utilise a Spark engine higher than 1.3.1

My understanding was that there were miss-match on Hadoop assembly Jar files that cause Hive not being able to run on Spark using the binary downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it crashed.

I do not know the development state of this cross-breed but will be very desirable if we could manage to sort out this spark-assembly-1.x.1-hadoop2.4.0.jar for once.

Thanks