spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Spark streaming + kafka error with json library
Date Wed, 29 Mar 2017 23:13:35 GMT
Try depending on "spark-streaming-kafka-0-10_2.11" (not the assembly)

On Wed, Mar 29, 2017 at 9:59 AM, Srikanth <srikanth.ht@gmail.com> wrote:

> Hello,
>
> I'm trying to use "org.json4s" % "json4s-native" library in a spark
> streaming + kafka direct app.
> When I use the latest version of the lib I get an error similar to this
> <https://github.com/json4s/json4s/issues/316>
> The work around suggest there is to use version 3.2.10. As spark has a
> hard dependency on this version.
>
> I forced this version in SBT with
> dependencyOverrides += "org.json4s" %% "json4s-native" % "3.2.10"
>
> But now it seems to have some conflict with spark-streaming-kafka-0-10-
> assembly
>
> [error] (*:assembly) deduplicate: different file contents found in the
> following:
>
> [error] C:\Users\stati\.ivy2\cache\org.apache.spark\spark-
> streaming-kafka-0-10-assembly_2.11\jars\spark-streaming-
> kafka-0-10-assembly_2.11-2.1.0.jar:scala/util/parsing/combinator/
> ImplicitConversions$$anonfun$flatten2$1.class
>
> [error] C:\Users\stati\.ivy2\cache\org.scala-lang.modules\scala-
> parser-combinators_2.11\bundles\scala-parser-combinators_2.11-1.0.4.jar:
> scala/util/parsing/combinator/ImplicitConversions$$anonfun$
> flatten2$1.class
>
> DependencyTree didn't show spark-streaming-kafka-0-10-assembly pulling
> json4s-native.
> Any idea how to resolve this? I'm using spark version 2.1.0
>
> Thanks,
> Srikanth
>

Mime
View raw message