spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: Re : Scala 2.10 Merge
Date Mon, 16 Dec 2013 17:58:50 GMT
Great job everyone!  A big step forward.


On Sat, Dec 14, 2013 at 2:37 AM, andy.petrella@gmail.com <
andy.petrella@gmail.com> wrote:

> That's a very good news!
> Congrats
>
> Envoyé depuis mon HTC
>
> ----- Reply message -----
> De : "Sam Bessalah" <samkiller@gmail.com>
> Pour : "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org>
> Objet : Scala 2.10 Merge
> Date : sam., déc. 14, 2013 11:03
>
>
> Yes. Awesome.
> Great job guys.
>
> Sam Bessalah
>
> > On Dec 14, 2013, at 9:59 AM, Patrick Wendell <pwendell@gmail.com> wrote:
> >
> > Alright I just merged this in - so Spark is officially "Scala 2.10"
> > from here forward.
> >
> > For reference I cut a new branch called scala-2.9 with the commit
> > immediately prior to the merge:
> >
> https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=shortlog;h=refs/heads/scala-2.9
> >
> > - Patrick
> >
> >> On Thu, Dec 12, 2013 at 8:26 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> >> Hey Reymond,
> >>
> >> Let's move this discussion out of this thread and into the associated
> JIRA.
> >> I'll write up our current approach over there.
> >>
> >> https://spark-project.atlassian.net/browse/SPARK-995
> >>
> >> - Patrick
> >>
> >>
> >>> On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <raymond.liu@intel.com>
> wrote:
> >>>
> >>> Hi Patrick
> >>>
> >>>        So what's the plan for support Yarn 2.2 in 0.9? As far as I can
> >>> see, if you want to support both 2.2 and 2.0 , due to protobuf version
> >>> incompatible issue. You need two version of akka anyway.
> >>>
> >>>        Akka 2.3-M1 looks like have a little bit change in API, we
> >>> probably could isolate the code like what we did on yarn part API. I
> >>> remember that it is mentioned that to use reflection for different API
> is
> >>> preferred. So the purpose to use reflection is to use one release bin
> jar to
> >>> support both version of Hadoop/Yarn on runtime, instead of build
> different
> >>> bin jar on compile time?
> >>>
> >>>         Then all code related to hadoop will also be built in separate
> >>> modules for loading on demand? This sounds to me involve a lot of
> works. And
> >>> you still need to have shim layer and separate code for different
> version
> >>> API and depends on different version Akka etc. Sounds like and even
> strict
> >>> demands versus our current approaching on master, and with dynamic
> class
> >>> loader in addition, And the problem we are facing now are still there?
> >>>
> >>> Best Regards,
> >>> Raymond Liu
> >>>
> >>> -----Original Message-----
> >>> From: Patrick Wendell [mailto:pwendell@gmail.com]
> >>> Sent: Thursday, December 12, 2013 5:13 PM
> >>> To: dev@spark.incubator.apache.org
> >>> Subject: Re: Scala 2.10 Merge
> >>>
> >>> Also - the code is still there because of a recent merge that took in
> some
> >>> newer changes... we'll be removing it for the final merge.
> >>>
> >>>
> >>> On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <pwendell@gmail.com>
> >>> wrote:
> >>>
> >>>> Hey Raymond,
> >>>>
> >>>> This won't work because AFAIK akka 2.3-M1 is not binary compatible
> >>>> with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need
> >>>> to still use the older protobuf library, so we'd need to support both.
> >>>>
> >>>> I'd also be concerned about having a reference to a non-released
> >>>> version of akka. Akka is the source of our hardest-to-find bugs and
> >>>> simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting.
> >>>> Of course, if you are building off of master you can maintain a fork
> >>>> that uses this.
> >>>>
> >>>> - Patrick
> >>>>
> >>>>
> >>>> On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond
> >>>> <raymond.liu@intel.com>wrote:
> >>>>
> >>>>> Hi Patrick
> >>>>>
> >>>>>        What does that means for drop YARN 2.2? seems codes are still
> >>>>> there. You mean if build upon 2.2 it will break, and won't and work
> >>>>> right?
> >>>>> Since the home made akka build on scala 2.10 are not there. While,
if
> >>>>> for this case, can we just use akka 2.3-M1 which run on protobuf
2.5
> >>>>> for replacement?
> >>>>>
> >>>>> Best Regards,
> >>>>> Raymond Liu
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Patrick Wendell [mailto:pwendell@gmail.com]
> >>>>> Sent: Thursday, December 12, 2013 4:21 PM
> >>>>> To: dev@spark.incubator.apache.org
> >>>>> Subject: Scala 2.10 Merge
> >>>>>
> >>>>> Hi Developers,
> >>>>>
> >>>>> In the next few days we are planning to merge Scala 2.10 support
into
> >>>>> Spark. For those that haven't been following this, Prashant Sharma
> >>>>> has been maintaining the scala-2.10 branch of Spark for several
> >>>>> months. This branch is current with master and has been reviewed
for
> >>>>> merging:
> >>>>>
> >>>>> https://github.com/apache/incubator-spark/tree/scala-2.10
> >>>>>
> >>>>> Scala 2.10 support is one of the most requested features for Spark
-
> >>>>> it will be great to get this into Spark 0.9! Please note that *Scala
> >>>>> 2.10 is not binary compatible with Scala 2.9*. With that in mind,
I
> >>>>> wanted to give a few heads-up/requests to developers:
> >>>>>
> >>>>> If you are developing applications on top of Spark's master branch,
> >>>>> those will need to migrate to Scala 2.10. You may want to download
> >>>>> and test the current scala-2.10 branch in order to make sure you
will
> >>>>> be okay as Spark developments move forward. Of course, you can always
> >>>>> stick with the current master commit and be fine (I'll cut a tag
when
> >>>>> we do the merge in order to delineate where the version changes).
> >>>>> Please open new threads on the dev list to report and discuss any
> >>>>> issues.
> >>>>>
> >>>>> This merge will temporarily drop support for YARN 2.2 on the master
> >>>>> branch.
> >>>>> This is because the workaround we used was only compiled for Scala
> 2.9.
> >>>>> We are going to come up with a more robust solution to YARN 2.2
> >>>>> support before releasing 0.9.
> >>>>>
> >>>>> Going forward, we will continue to make maintenance releases on
> >>>>> branch-0.8 which will remain compatible with Scala 2.9.
> >>>>>
> >>>>> For those interested, the primary code changes in this merge are
> >>>>> upgrading the akka version, changing the use of Scala 2.9's
> >>>>> ClassManifest construct to Scala 2.10's ClassTag, and updating the
> >>>>> spark shell to work with Scala 2.10's repl.
> >>>>>
> >>>>> - Patrick
> >>
> >>
>



-- 
--
Evan Chan
Staff Engineer
ev@ooyala.com  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message