spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Su She <suhsheka...@gmail.com>
Subject Re: What joda-time dependency does spark submit use/need?
Date Mon, 02 Mar 2015 19:37:34 GMT
Hi Todd,

So I am already specifying joda-time-2.7 (have tried 2.2, 2.3, 2.6, 2.7) in
the --jars option. I tried using the joda-time bundle jar (
http://mvnrepository.com/artifact/org.apache.servicemix.bundles/org.apache.servicemix.bundles.joda-time/2.3_1)
which comes with joda-convert.

I know its pointing to the appropriate jar as I included this in my java
program: System.out.println(DateTimeFormat.forPattern("yyyyMMdd")

    .getClass()
    .getProtectionDomain()
    .getCodeSource()
    .getLocation());


and it tells me that it's pointing to the jar that i'm giving it in the
--jars option. It would be helpful to know if the error I am getting is
because of spark-submit or the java app. Thank you!

Exception in thread "main" java.lang.NoClassDefFoundError:
org/joda/time/format/DateTimeFormat
        at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at java.lang.Class.newInstance(Class.java:374)
        at
com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119)
        at
com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105)
        at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78)
        at
com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307)
        at
com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280)
        at
com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
        at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.setEndpoint(AmazonDynamoDBClient.java:2946)
        at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:351)
        at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:273)
        at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:250)
        at AmazonDynamoDBSample.init(AmazonDynamoDBSample.java:81)
        at AmazonDynamoDBSample.main(AmazonDynamoDBSample.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.joda.time.format.DateTimeFormat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

Best,

Su


On Fri, Feb 27, 2015 at 4:14 PM, Todd Nist <tsindotg@gmail.com> wrote:

> You can specify these jars (joda-time-2.7.jar, joda-convert-1.7.jar)
> either as part of your build and assembly or via the --jars option to
> spark-submit.
>
> HTH.
>
> On Fri, Feb 27, 2015 at 2:48 PM, Su She <suhshekar52@gmail.com> wrote:
>
>> Hello Everyone,
>>
>> I'm having some issues launching (non-spark) applications via the
>> spark-submit commands. The common error I am getting is c/p below. I am
>> able to submit a spark streaming/kafka spark application, but can't start a
>> dynamoDB java app. The common error is related to joda-time.
>>
>> 1) I realized spark-submit was pointing to joda-time-1.6 in the
>> hadoop/lib,so I deleted this and my error changed from NoSuchMethodFound to
>> NoClassDefFoundError.
>>
>> Instead of pointing to the other version of joda-time in the hadoop/lib,
>> it now pointed to the jars I set a path to in my spark-submit command (I
>> tried joda-time versions 2.2, 2.3, 2.6, 2.7), but still got the errors
>>
>> 2) My rudimentary theory is that spark-submit uses < joda-time-2.0, but
>> the applications I'm running need >2.0.
>>
>> Thank you for the help!
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/joda/time/format/DateTimeFormat
>>         at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
>>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>         at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>         at java.lang.Class.newInstance(Class.java:374)
>>         at
>> com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119)
>>         at
>> com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105)
>>         at
>> com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78)
>>         at
>> com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307)
>>         at
>> com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280)
>>         at
>> com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
>>         at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.setEndpoint(AmazonDynamoDBClient.java:2946)
>>         at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:351)
>>         at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:273)
>>         at
>> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:250)
>>         at AmazonDynamoDBSample.init(AmazonDynamoDBSample.java:81)
>>         at AmazonDynamoDBSample.main(AmazonDynamoDBSample.java:87)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: java.lang.ClassNotFoundException:
>> org.joda.time.format.DateTimeFormat
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>>
>

Mime
View raw message