spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Avati <av...@gluster.org>
Subject Re: Scala 2.11 external dependencies
Date Wed, 06 Aug 2014 06:42:49 GMT
On Mon, Aug 4, 2014 at 1:01 PM, Anand Avati <avati@gluster.org> wrote:

>
>
>
> On Sun, Aug 3, 2014 at 9:09 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
>
>> Hey Anand,
>>
>> Thanks for looking into this - it's great to see momentum towards Scala
>> 2.11 and I'd love if this land in Spark 1.2.
>>
>> For the external dependencies, it would be good to create a sub-task of
>> SPARK-1812 to track our efforts encouraging other projects to upgrade. In
>> certain cases (e.g. Kafka) there is fairly late-stage work on this already,
>> so we can e.g. link to those JIRA's as well. A good starting point is to
>> just go to their dev list and ask what the status is, most Scala projects
>> have put at least some thought into this already. Another thing we can do
>> is submit patches ourselves to those projects to help get them upgraded.
>> The twitter libraries, e.g., tend to be pretty small and also open to
>> external contributions.
>>
>> One other thing in the mix here - Prashant Sharma has also spent some
>> time looking at this, so it might be good for you two to connect (probably
>> off list) and sync up. Prashant has contributed to many Scala projects, so
>> he might have cycles to go and help some of our dependencies get upgraded -
>> but I won't commit to that on his behalf :).
>>
>> Regarding Akka - I shaded and published akka as a one-off thing:
>> https://github.com/pwendell/akka/tree/2.2.3-shaded-proto
>>
>> Over time we've had to publish our own versions of a small number of
>> dependencies. It's somewhat high overhead, but it actually works quite well
>> in terms of avoiding some of the nastier dependency conflicts. At least
>> better than other alternatives I've seen such as using a shader build
>> plug-in.
>>
>> Going forward, I'd actually like to track these in the Spark repo itself.
>> For instance, we have a bash script in the spark repo that can e.g. check
>> out akka, apply a few patches or regular expressions, and then you have a
>> fully shaded dependency that can be published to maven. If you wanted to
>> take a crack at something like that for akka 2.3.4, be my guest. I can help
>> with the actual publishing.
>>
>
> Will give it a try, thanks!
>
>
Patrick, I have a set of scripts at https://github.com/avati/spark-shaded
which generate shaded artifacts for akka 2.3.4 and transitive dependencies
under org.spark-project group for both Scala 2.10 and 2.11 in your local
maven repo ($HOME/.m2/...) Publishing these artifacts is necessary for
https://github.com/apache/spark/pull/1685. Once merged, bumping up Scala
version to 2.11 will be smooth w.r.t akka (verified it on my system)

If you think it will be useful, I can submit a PR of those scripts into
spark.git/extras or somewhere.

Thanks

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message