spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Brown <...@mult.ifario.us>
Subject Re: jackson-core-asl jar (1.8.8 vs 1.9.x) conflict with the spark-sql (version 1.x)
Date Sat, 28 Jun 2014 06:22:09 GMT
Hi, Mans --

Both of those versions of Jackson are pretty ancient.  Do you know which of
the Spark dependencies is pulling them in?  It would be good for us (the
Jackson, Woodstox, etc., folks) to see if we can get people to upgrade to
more recent versions of Jackson.

-- Paul

—
prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/


On Fri, Jun 27, 2014 at 12:58 PM, M Singh <mans6singh@yahoo.com> wrote:

>  Hi:
>
> I am using spark to stream data to cassandra and it works fine in local
> mode. But when I execute the application in a standalone clustered env I
> got exception included below (java.lang.NoClassDefFoundError:
> org/codehaus/jackson/annotate/JsonClass).
>
> I think this is due to the jackson-core-asl dependency conflict
> (jackson-core-asl 1.8.8 has the JsonClass but 1.9.x does not).  The 1.9.x
> version is being pulled in by spark-sql project.  I tried adding
> jackson-core-asl 1.8.8 with --jars argument while submitting the
> application for execution but it did not work.  So I created a custom spark
> build excluding sql project.  With this custom spark install I was able to
> resolve the issue at least on a single node cluster (separate master and
> worker).
>
> If there is an alternate way to resolve this conflicting jar issue without
> a custom build (eg: configuration to use the user defined jars in the
> executor class path first), please let me know.
>
> Also, is there a comprehensive list of configuration properties available
> for spark ?
>
> Thanks
>
> Mans
>
> Exception trace
>
>  TaskSetManager: Loss was due to java.lang.NoClassDefFoundError
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
> at
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:229)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:386)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
> at
> org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
> at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602)
>
>

Mime
View raw message