spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anusha Shamanur <anushas...@gmail.com>
Subject Re: Spark SQL Thrift Server start exception : java.lang.ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory
Date Wed, 04 Mar 2015 00:52:01 GMT
I downloaded different versions of the jars and it worked.

Thanks!

On Tue, Mar 3, 2015 at 4:45 PM, Cheng, Hao <hao.cheng@intel.com> wrote:

>  Which version / distribution are you using? Please references this blog
> that Felix C posted if you’re running on CDH.
>
>
> http://eradiating.wordpress.com/2015/02/22/getting-hivecontext-to-work-in-cdh/
>
>
>
> Or you may also need to download the datanucleus*.jar files try to add the
> option of “--jars” while starting the spark shell.
>
>
>
> *From:* Anusha Shamanur [mailto:anushasb11@gmail.com]
> *Sent:* Wednesday, March 4, 2015 5:07 AM
> *To:* Cheng, Hao
> *Subject:* Re: Spark SQL Thrift Server start exception :
> java.lang.ClassNotFoundException:
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>
>
>
> Hi,
>
>
>
> I am getting the same error. There is no lib folder in my $SPARK_HOME. But
> I included these jars while calling spark-shell.
>
>
>
> Now, I get this:
>
> Caused by: org.datanucleus.exceptions.ClassNotResolvedException: Class
> "org.datanucleus.store.rdbms.RDBMSStoreManager" was not found in the
> CLASSPATH. Please check your specification and your CLASSPATH.
>
>                at
> org.datanucleus.ClassLoaderResolverImpl.classForName(ClassLoaderResolverImpl.java:218)
>
>
>
> How do I solve this?
>
>
>
> On Mon, Mar 2, 2015 at 11:04 PM, Cheng, Hao <hao.cheng@intel.com> wrote:
>
> Copy those jars into the $SPARK_HOME/lib/
>
> datanucleus-api-jdo-3.2.6.jar
> datanucleus-core-3.2.10.jar
> datanucleus-rdbms-3.2.9.jar
>
> see
> https://github.com/apache/spark/blob/master/bin/compute-classpath.sh#L120
>
>
>
> -----Original Message-----
> From: fanooos [mailto:dev.fanooos@gmail.com]
> Sent: Tuesday, March 3, 2015 2:50 PM
> To: user@spark.apache.org
> Subject: Spark SQL Thrift Server start exception :
> java.lang.ClassNotFoundException:
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>
> I have installed a hadoop cluster (version : 2.6.0), apache spark (version
> :
> 1.2.1 preBuilt for hadoop 2.4 and later), and hive (version 1.0.0).
>
> When I try to start the spark sql thrift server I am getting the following
> exception.
>
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>         at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
>         at
>
> org.apache.spark.sql.hive.HiveContext$$anonfun$4.apply(HiveContext.scala:235)
>         at
>
> org.apache.spark.sql.hive.HiveContext$$anonfun$4.apply(HiveContext.scala:231)
>         at scala.Option.orElse(Option.scala:257)
>         at
> org.apache.spark.sql.hive.HiveContext.x$3$lzycompute(HiveContext.scala:231)
>         at org.apache.spark.sql.hive.HiveContext.x$3(HiveContext.scala:229)
>         at
>
> org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:229)
>         at
> org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:229)
>         at
> org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:292)
>         at
> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
>         at
> org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:248)
>         at
> org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:91)
>         at
> org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:90)
>         at
>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>         at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>         at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:90)
>         at
> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:72)
>         at
>
> org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:51)
>         at
>
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:56)
>         at
>
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>         at
>
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
>         at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
>         at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
>         at
>
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
>         at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
>         ... 26 more
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at
>
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
>         ... 31 more
> Caused by: javax.jdo.JDOFatalUserException: Class
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
> NestedThrowables:
> java.lang.ClassNotFoundException:
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>         at
>
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1175)
>         at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
>         at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
>         at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
>         at
>
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
>         at
>
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
>         at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
>         at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
>         at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>         at
>
> org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
>         at
>
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
>         at
>
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
>         at
>
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
>         at
>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
>         ... 36 more
> Caused by: java.lang.ClassNotFoundException:
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:274)
>         at javax.jdo.JDOHelper$18.run(JDOHelper.java:2018)
>         at javax.jdo.JDOHelper$18.run(JDOHelper.java:2016)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.jdo.JDOHelper.forName(JDOHelper.java:2015)
>         at
>
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1162)
>         ... 55 more
>
> As I understand from the exception, the datanucleus jars is not in the
> ClassPath so I added the following line to my .profile
>
> export $SPARK_OPTS="-Djava.library.path=$SPARK_HOME/lib"
>
> but the same exception still thrown and the server does not run.
>
> How can I solve this issue ?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Thrift-Server-start-exception-java-lang-ClassNotFoundException-org-datanucleus-api-jdo-JDOy-tp21887.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
> commands, e-mail: user-help@spark.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
>
>
> --
>
> Regards,
>
> Anusha
>



-- 
Regards,
Anusha

Mime
View raw message