spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christophe Billiard <christophe.billi...@gmail.com>
Subject Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with akka-http/akka-stream
Date Tue, 06 Jan 2015 11:49:56 GMT
Thanks Pankaj for the assembly plugin tip.

Yes there is a version mismatch of akka actor between Spark 1.1.1 and
akka-http/akka-stream (2.2.3 versus 2.3.x).

After some digging, I see 4 options for this problem (in case others
encounter it):
1) Upgrade to Spark 1.2.0, the same code will work (not possible for me
since I want to use Datastax connector)
2) Make a custom build of Spark 1.1.1
3) Use play instead with akka actor 2.2.3 (play 2.2.3 for instance)
4) Wait for Datastax connector 1.2.0 (released on 31th January 2015)

I currently trying option 3

Thank you all for your help



On Sat, Jan 3, 2015 at 4:11 AM, Pankaj Narang [via Apache Spark User List] <
ml-node+s1001560n20950h72@n3.nabble.com> wrote:

> Like before I get a java.lang.NoClassDefFoundError:
> akka/stream/FlowMaterializer$
>
> This can be solved using assembly plugin. you need to enable assembly
> plugin in global plugins
>
> C:\Users\infoshore\.sbt\0.13\plugins
>  add a line in plugins.sbt  addSbtPlugin("com.eed3si9n" % "sbt-assembly" %
> "0.11.0")
>
>
>
>  and then add the following lines in build.sbt
>
> import AssemblyKeys._ // put this at the top of the file
>
> seq(assemblySettings: _*)
>
> Also in the bottom dont forget to add
>
> assemblySettings
>
> mergeStrategy in assembly := {
>   case m if m.toLowerCase.endsWith("manifest.mf")          =>
> MergeStrategy.discard
>   case m if m.toLowerCase.matches("meta-inf.*\\.sf$")      =>
> MergeStrategy.discard
>   case "log4j.properties"                                  =>
> MergeStrategy.discard
>   case m if m.toLowerCase.startsWith("meta-inf/services/") =>
> MergeStrategy.filterDistinctLines
>   case "reference.conf"                                    =>
> MergeStrategy.concat
>   case _                                                   =>
> MergeStrategy.first
> }
>
>
> Now in your sbt run sbt assembly that will create the jar which can be run
> without --jars options
> as this will be a uber jar containing all jars
>
>
>
> Also nosuchmethod exception is thrown when there is difference in versions
> of complied and runtime versions.
>
> What is the version of spark you are using ? You need to use same version
> in build.sbt
>
>
> Here is your build.sbt
>
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
> //exclude("com.typesafe", "config")
>
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>
> libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core"
> % "2.1.3"
>
> libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector"
> % "1.1.0" withSources() withJavadoc()
>
> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
> "2.0.5"
>
> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>
>
> and your error is Exception in thread "main" java.lang.NoSuchMethodError:
> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
>
>         at
> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
>
>  I think there is version mismatch on the jars you use at runtime
>
>
>  If you need more help add me on skype pankaj.narang
>
>
> ---Pankaj
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20950.html
>  To unsubscribe from NoSuchMethodError:
> com.typesafe.config.Config.getDuration with akka-http/akka-stream, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=20926&code=Y2hyaXN0b3BoZS5iaWxsaWFyZEBnbWFpbC5jb218MjA5MjZ8LTE2ODA2NTAwMDk=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20988.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message