spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Avati <>
Subject Re: Scala 2.11 external dependencies
Date Wed, 06 Aug 2014 06:42:49 GMT
On Mon, Aug 4, 2014 at 1:01 PM, Anand Avati <> wrote:

> On Sun, Aug 3, 2014 at 9:09 PM, Patrick Wendell <>
> wrote:
>> Hey Anand,
>> Thanks for looking into this - it's great to see momentum towards Scala
>> 2.11 and I'd love if this land in Spark 1.2.
>> For the external dependencies, it would be good to create a sub-task of
>> SPARK-1812 to track our efforts encouraging other projects to upgrade. In
>> certain cases (e.g. Kafka) there is fairly late-stage work on this already,
>> so we can e.g. link to those JIRA's as well. A good starting point is to
>> just go to their dev list and ask what the status is, most Scala projects
>> have put at least some thought into this already. Another thing we can do
>> is submit patches ourselves to those projects to help get them upgraded.
>> The twitter libraries, e.g., tend to be pretty small and also open to
>> external contributions.
>> One other thing in the mix here - Prashant Sharma has also spent some
>> time looking at this, so it might be good for you two to connect (probably
>> off list) and sync up. Prashant has contributed to many Scala projects, so
>> he might have cycles to go and help some of our dependencies get upgraded -
>> but I won't commit to that on his behalf :).
>> Regarding Akka - I shaded and published akka as a one-off thing:
>> Over time we've had to publish our own versions of a small number of
>> dependencies. It's somewhat high overhead, but it actually works quite well
>> in terms of avoiding some of the nastier dependency conflicts. At least
>> better than other alternatives I've seen such as using a shader build
>> plug-in.
>> Going forward, I'd actually like to track these in the Spark repo itself.
>> For instance, we have a bash script in the spark repo that can e.g. check
>> out akka, apply a few patches or regular expressions, and then you have a
>> fully shaded dependency that can be published to maven. If you wanted to
>> take a crack at something like that for akka 2.3.4, be my guest. I can help
>> with the actual publishing.
> Will give it a try, thanks!
Patrick, I have a set of scripts at
which generate shaded artifacts for akka 2.3.4 and transitive dependencies
under org.spark-project group for both Scala 2.10 and 2.11 in your local
maven repo ($HOME/.m2/...) Publishing these artifacts is necessary for Once merged, bumping up Scala
version to 2.11 will be smooth w.r.t akka (verified it on my system)

If you think it will be useful, I can submit a PR of those scripts into
spark.git/extras or somewhere.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message