spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: different akka versions and spark
Date Tue, 06 Jan 2015 15:29:15 GMT
if the classes are in the original location than i think its safe to say
that this makes it impossible for us to build one app that can run against
spark 1.0.x, 1.1.x and spark 1.2.x.

thats no big deal, but it does beg the question of what compatibility can
reasonably be expected for spark 1.x series. i have seen a lot of focus on
backwards compatibility of the spark 1.x api, but to me thats kind of a
moot point if i cannot run apps against all 1.x versions anyhow since
dependencies are not compatible.


On Mon, Jan 5, 2015 at 5:08 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> Spark doesn't really shade akka; it pulls a different build (kept
> under the "org.spark-project.akka" group and, I assume, with some
> build-time differences from upstream akka?), but all classes are still
> in the original location.
>
> The upgrade is a little more unfortunate than just changing akka,
> since it also changes some transitive dependencies which also have
> compatibility issues (e.g. the typesafe config library). But I believe
> it's needed to support Scala 2.11...
>
> On Mon, Jan 5, 2015 at 8:27 AM, Koert Kuipers <koert@tresata.com> wrote:
> > since spark shaded akka i wonder if it would work, but i doubt it
> >
> > On Mon, Jan 5, 2015 at 9:56 AM, Cody Koeninger <cody@koeninger.org>
> wrote:
> >>
> >> I haven't tried it with spark specifically, but I've definitely run into
> >> problems trying to depend on multiple versions of akka in one project.
> >>
> >> On Sat, Jan 3, 2015 at 11:22 AM, Koert Kuipers <koert@tresata.com>
> wrote:
> >>>
> >>> hey Ted,
> >>> i am aware of the upgrade efforts for akka. however if spark 1.2 forces
> >>> me to upgrade all our usage of akka to 2.3.x while spark 1.0 and 1.1
> force
> >>> me to use akka 2.2.x then we cannot build one application that runs on
> all
> >>> spark 1.x versions, which i would consider a major incompatibility.
> >>> best, koert
> >>>
> >>>
> >>> On Sat, Jan 3, 2015 at 12:11 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>>
> >>>> Please see http://akka.io/news/2014/05/22/akka-2.3.3-released.html
> which
> >>>> points to
> >>>>
> http://doc.akka.io/docs/akka/2.3.3/project/migration-guide-2.2.x-2.3.x.html?_ga=1.35212129.1385865413.1420220234
> >>>>
> >>>> Cheers
> >>>>
> >>>> On Fri, Jan 2, 2015 at 9:11 AM, Koert Kuipers <koert@tresata.com>
> wrote:
> >>>>>
> >>>>> i noticed spark 1.2.0 bumps the akka version. since spark uses it's
> own
> >>>>> akka version, does this mean it can co-exist with another akka
> version in
> >>>>> the same JVM? has anyone tried this?
> >>>>>
> >>>>> we have some spark apps that also use akka (2.2.3) and spray. if
> >>>>> different akka versions causes conflicts then spark 1.2.0 would
not
> be
> >>>>> backwards compatible for us...
> >>>>>
> >>>>> thanks. koert
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
> --
> Marcelo
>

Mime
View raw message