spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "andy.petrella@gmail.com" <andy.petre...@gmail.com>
Subject Re : Scala 2.10 Merge
Date Sat, 14 Dec 2013 10:37:53 GMT
That's a very good news!
Congrats

Envoyé depuis mon HTC

----- Reply message -----
De : "Sam Bessalah" <samkiller@gmail.com>
Pour : "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org>
Objet : Scala 2.10 Merge
Date : sam., déc. 14, 2013 11:03


Yes. Awesome.
Great job guys.

Sam Bessalah

> On Dec 14, 2013, at 9:59 AM, Patrick Wendell <pwendell@gmail.com> wrote:
> 
> Alright I just merged this in - so Spark is officially "Scala 2.10"
> from here forward.
> 
> For reference I cut a new branch called scala-2.9 with the commit
> immediately prior to the merge:
> https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=shortlog;h=refs/heads/scala-2.9
> 
> - Patrick
> 
>> On Thu, Dec 12, 2013 at 8:26 PM, Patrick Wendell <pwendell@gmail.com> wrote:
>> Hey Reymond,
>> 
>> Let's move this discussion out of this thread and into the associated JIRA.
>> I'll write up our current approach over there.
>> 
>> https://spark-project.atlassian.net/browse/SPARK-995
>> 
>> - Patrick
>> 
>> 
>>> On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <raymond.liu@intel.com> wrote:
>>> 
>>> Hi Patrick
>>> 
>>>        So what's the plan for support Yarn 2.2 in 0.9? As far as I can
>>> see, if you want to support both 2.2 and 2.0 , due to protobuf version
>>> incompatible issue. You need two version of akka anyway.
>>> 
>>>        Akka 2.3-M1 looks like have a little bit change in API, we
>>> probably could isolate the code like what we did on yarn part API. I
>>> remember that it is mentioned that to use reflection for different API is
>>> preferred. So the purpose to use reflection is to use one release bin jar to
>>> support both version of Hadoop/Yarn on runtime, instead of build different
>>> bin jar on compile time?
>>> 
>>>         Then all code related to hadoop will also be built in separate
>>> modules for loading on demand? This sounds to me involve a lot of works. And
>>> you still need to have shim layer and separate code for different version
>>> API and depends on different version Akka etc. Sounds like and even strict
>>> demands versus our current approaching on master, and with dynamic class
>>> loader in addition, And the problem we are facing now are still there?
>>> 
>>> Best Regards,
>>> Raymond Liu
>>> 
>>> -----Original Message-----
>>> From: Patrick Wendell [mailto:pwendell@gmail.com]
>>> Sent: Thursday, December 12, 2013 5:13 PM
>>> To: dev@spark.incubator.apache.org
>>> Subject: Re: Scala 2.10 Merge
>>> 
>>> Also - the code is still there because of a recent merge that took in some
>>> newer changes... we'll be removing it for the final merge.
>>> 
>>> 
>>> On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <pwendell@gmail.com>
>>> wrote:
>>> 
>>>> Hey Raymond,
>>>> 
>>>> This won't work because AFAIK akka 2.3-M1 is not binary compatible
>>>> with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need
>>>> to still use the older protobuf library, so we'd need to support both.
>>>> 
>>>> I'd also be concerned about having a reference to a non-released
>>>> version of akka. Akka is the source of our hardest-to-find bugs and
>>>> simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting.
>>>> Of course, if you are building off of master you can maintain a fork
>>>> that uses this.
>>>> 
>>>> - Patrick
>>>> 
>>>> 
>>>> On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond
>>>> <raymond.liu@intel.com>wrote:
>>>> 
>>>>> Hi Patrick
>>>>> 
>>>>>        What does that means for drop YARN 2.2? seems codes are still
>>>>> there. You mean if build upon 2.2 it will break, and won't and work
>>>>> right?
>>>>> Since the home made akka build on scala 2.10 are not there. While, if
>>>>> for this case, can we just use akka 2.3-M1 which run on protobuf 2.5
>>>>> for replacement?
>>>>> 
>>>>> Best Regards,
>>>>> Raymond Liu
>>>>> 
>>>>> 
>>>>> -----Original Message-----
>>>>> From: Patrick Wendell [mailto:pwendell@gmail.com]
>>>>> Sent: Thursday, December 12, 2013 4:21 PM
>>>>> To: dev@spark.incubator.apache.org
>>>>> Subject: Scala 2.10 Merge
>>>>> 
>>>>> Hi Developers,
>>>>> 
>>>>> In the next few days we are planning to merge Scala 2.10 support into
>>>>> Spark. For those that haven't been following this, Prashant Sharma
>>>>> has been maintaining the scala-2.10 branch of Spark for several
>>>>> months. This branch is current with master and has been reviewed for
>>>>> merging:
>>>>> 
>>>>> https://github.com/apache/incubator-spark/tree/scala-2.10
>>>>> 
>>>>> Scala 2.10 support is one of the most requested features for Spark -
>>>>> it will be great to get this into Spark 0.9! Please note that *Scala
>>>>> 2.10 is not binary compatible with Scala 2.9*. With that in mind, I
>>>>> wanted to give a few heads-up/requests to developers:
>>>>> 
>>>>> If you are developing applications on top of Spark's master branch,
>>>>> those will need to migrate to Scala 2.10. You may want to download
>>>>> and test the current scala-2.10 branch in order to make sure you will
>>>>> be okay as Spark developments move forward. Of course, you can always
>>>>> stick with the current master commit and be fine (I'll cut a tag when
>>>>> we do the merge in order to delineate where the version changes).
>>>>> Please open new threads on the dev list to report and discuss any
>>>>> issues.
>>>>> 
>>>>> This merge will temporarily drop support for YARN 2.2 on the master
>>>>> branch.
>>>>> This is because the workaround we used was only compiled for Scala 2.9.
>>>>> We are going to come up with a more robust solution to YARN 2.2
>>>>> support before releasing 0.9.
>>>>> 
>>>>> Going forward, we will continue to make maintenance releases on
>>>>> branch-0.8 which will remain compatible with Scala 2.9.
>>>>> 
>>>>> For those interested, the primary code changes in this merge are
>>>>> upgrading the akka version, changing the use of Scala 2.9's
>>>>> ClassManifest construct to Scala 2.10's ClassTag, and updating the
>>>>> spark shell to work with Scala 2.10's repl.
>>>>> 
>>>>> - Patrick
>> 
>> 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message