spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: proposal: replace lift-json with spray-json
Date Wed, 12 Feb 2014 03:42:03 GMT
I am not sure I fully understand this reasoning. I imagine that lift-json
is only one of hundreds of packages that would have to be built if you
wanted to build all of Spark's transitive dependencies from source.

Additionally, to make sure I understand the impact -- this is only intended
to simplify the process of packaging Spark on a new OS distribution that
disallows pulling in binaries?


On Sun, Feb 9, 2014 at 4:50 PM, Will Benton <willb@redhat.com> wrote:

> Matei, sorry if I was unclear:  I'm referring to downstream operating
> system distributions (like Fedora or Debian) that  have policies requiring
> that all packages are built from source (using only tools already packaged
> in the distribution).  So end-users (and distributions with different
> policies) don't have to build Lift to get the lift-json artifact, but it is
> a concern for many open-source communities.
>
> best,
> wb
>
>
> ----- Original Message -----
> > From: "Matei Zaharia" <matei.zaharia@gmail.com>
> > To: dev@spark.incubator.apache.org
> > Sent: Sunday, February 9, 2014 4:29:20 PM
> > Subject: Re: proposal:  replace lift-json with spray-json
> >
> > Will, why are you saying that downstream distributes need to build all of
> > Lift to package lift-json? Spark just downloads it from Maven Central,
> where
> > it's a JAR with no external dependencies. We don't have any dependency on
> > the rest of lift.
> >
> > Matei
> >
> > On Feb 9, 2014, at 11:28 AM, Will Benton <willb@redhat.com> wrote:
> >
> > > lift-json is a nice library, but Lift is a pretty heavyweight
> dependency to
> > > track just for its JSON support.  (lift-json is relatively
> self-contained
> > > as a dependency from an end-user's perspective, but downstream
> > > distributors need to build all of Lift in order to package the JSON
> > > support.)  I understand that this has come up before (cf. SPARK-883)
> and
> > > that the uncertain future of JSON support in the Scala standard
> library is
> > > the motivator for relying on an external library.
> > >
> > > I'm proposing replacing lift-json in Spark with something more
> lightweight.
> > > I've evaluated apparent project liveness and dependency scope for most
> of
> > > the current Scala JSON libraries and believe the best candidate is
> > > spray-json (https://github.com/spray/spray-json), the JSON library
> used by
> > > the Spray HTTP toolkit. spray-json is Apache-licensed, actively
> developed,
> > > and builds and works independently of Spray with only one external
> > > dependency.
> > >
> > > It looks to me like a pretty straightforward change (although
> > > JsonProtocol.scala would be a little more verbose since it couldn't use
> > > the Lift JSON DSL), and I'd like to do it.  I'm writing now to ask for
> > > some community feedback before making the change (and submitting a JIRA
> > > and PR).  If no one has any serious objections (to the effort in
> general
> > > or to to the choice of spark-json in particular), I'll go ahead and do
> it,
> > > but if anyone has concerns, I'd be happy to discuss and address them
> > > before getting started.
> > >
> > >
> > > thanks,
> > > wb
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message