spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Cannot instantiate hive context
Date Sun, 02 Nov 2014 16:41:10 GMT
Adding the libthrift jar
<http://mvnrepository.com/artifact/org.apache.thrift/libthrift/0.9.0> in
the class path would resolve this issue.

Thanks
Best Regards

On Sat, Nov 1, 2014 at 12:34 AM, Pala M Muthaia <mchettiar@rocketfuelinc.com
> wrote:

> Hi,
>
> I am trying to load hive datasets using HiveContext, in spark shell. Spark
> ver 1.0.1 and Hive ver 0.12.
>
> We are trying to get Spark work with hive datasets. I already have
> existing Spark deployment. Following is what i did on top of that:
> 1. Build spark using 'mvn -Pyarn,hive -Phadoop-2.4 -Dhadoop.version=2.4.0
> -DskipTests clean package'
> 2. Copy over spark-assembly-1.0.1-hadoop2.4.0.jar into spark deployment
> directory.
> 3. Launch spark-shell with the spark hive jar included in the list.
>
> When i execute *'*
>
> *val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)*
>
> i get the following error stack:
>
> java.lang.NoClassDefFoundError: org/apache/thrift/TBase
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         ....
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 55 more
>
> I thought that building with -Phive option should include all the
> necessary hive packages into the assembly jar (according to here
> <https://spark.apache.org/docs/1.0.1/sql-programming-guide.html#hive-tables>).
> I tried searching online and in this mailing list archive but haven't found
> any instructions on how to get this working.
>
> I know that there is additional step of updating the assembly jar across
> the whole cluster, not just client side, but right now, even the client is
> not working.
>
> Would appreciate instructions (or link to them) on how to get this working
> end-to-end.
>
>
> Thanks,
> pala
>

Mime
View raw message