spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Spark dependency library causing problems with conflicting versions at import
Date Tue, 08 Oct 2013 02:49:05 GMT
Hi Mingyu,

The latest version of Spark works with Scala 2.9.3, which is the latest Scala-2.9 version.
There's also a branch called branch-2.10 on GitHub that uses 2.10.3. What specific libraries
are you having trouble with?
> I see other open source projects private-namespacing the dependencies to work around
this problem. For example, Elasticsearch prepends dependent libraries with "org.elasticsearch….".
Would it be possible for Spark to take this path in the future?
This can be tough to do because some libraries use reflection or require only one instance
per JVM (e.g. Log4J). You may be able to package Spark in this way by modifying the Maven
file though.
> We intend to just create a remote connection from our code base, so it's not like we
really need the full Spark implementation in our code base. Is it possible to separate out
"client" project that only allows remote connections via SparkContext such that the "client"
jar only contains a few dependencies?
Yeah good question, this is hard today but might be possible later. I'd recommend writing
a little RPC service on top and accessing that.

Matei


Mime
View raw message