spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Avati <av...@gluster.org>
Subject Re: Spark on Scala 2.11
Date Tue, 13 May 2014 04:31:23 GMT
On Mon, May 12, 2014 at 6:27 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> We can build the REPL separately for each version of Scala, or even give
> that package a different name in Scala 2.11.
>

OK.


> Scala 2.11’s REPL actually added two flags, -Yrepl-class-based and
> -Yrepl-outdir, that encompass the two modifications we made to the REPL
> (using classes instead of objects to wrap each line, and grabbing the files
> from some directory). So it might be possible to run it without
> modifications using just a simple wrapper class around it. That would
> definitely simplify things!
>

Exactly. I have been tracking those changes in 2.11 as well. We would need
a simple wrapper around ILoop, set the Yreplclassbased flag, an HttpServer
to export the repl-outdir, bind sc into repl namespace and maybe little
more initialization (customizing repl init code in 2.11 is not as trivial
as the <=2.10 versions, but still doable.) Still, even with all this, it
should be much simpler than pulling in and refactoring everything like
today. I already have a prototype for this in my working tree, still needs
integration testing.


> BTW did the non-REPL parts run fine on 2.11?
>

Currently fighting to get all the dependencies in 2.11. Quick pointer where
I can get sources for akka-*-X.Y-shared-protobuf? Also, what's the smallest
set of dependencies to build the smallest testable subset of the project?

Thanks!


> Matei
>
> On May 12, 2014, at 2:09 PM, Anand Avati <avati@gluster.org> wrote:
>
> > Matei,
> > Thanks for confirming. I was looking specifically at the REPL part and
> how
> > it can be significantly simplified with 2.11 Scala, without having to
> > inherit a full copy of a refactored repl inside Spark. I am happy to
> > investigate/contribute a simpler 2.11 based REPL if this is were seen as
> a
> > priority (1.1 does not seem "too far" away.) However a 2.10 compatible
> > cross build would still require a separate (existing) REPL code for the
> > 2.10 build, no?
> >
> > Thanks.
> >
> > On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia <matei.zaharia@gmail.com
> >wrote:
> >
> >> We do want to support it eventually, possibly as early as Spark 1.1
> (which
> >> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it
> >> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think
> >> things will mostly work, except for possibly the REPL (which has require
> >> porting over code form the Scala REPL in each version).
> >>
> >> Matei
> >>
> >> On May 8, 2014, at 6:33 PM, Anand Avati <avati@gluster.org> wrote:
> >>
> >>> Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
> >>> Approximate timeline?
> >>>
> >>> Thank
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message