spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Ridley <mrid...@cloudera.com>
Subject Re: should we upgrade to spark 2.1 ?
Date Mon, 24 Apr 2017 16:46:47 GMT
I guess I'm not understanding what we would be supporting on 1.6 for a year since there has
been no release so far. 

Michael Ridley
Senior Solutions Architect
Cloudera

Sent from my mobile.
Pardon any spelling errors.

> On Apr 24, 2017, at 11:03 AM, Barona, Ricardo <ricardo.barona@intel.com> wrote:
> 
> In general, I’ve seen Spark 2.0.0 and 2.1.0 are faster than 1.6.0 because of the “whole-stage
code generation” – as per release notes, (2 – 10X) performance speedups for common operators
in SQL and DataFrames, including joins. The only thing that concerns me is MLlib deprecation
in 2.1.0. 
> 
> Given that, I’d say, we should migrate to 2.0.x, start experimenting with Spark ML
– LDA and give support for 1.6.0, like Nate says, for one year or so.
> 
> On 4/21/17, 6:59 PM, "Austin Leahy" <Austin@digitalminion.com> wrote:
> 
>    Damn Michael beat me to it ;D
>>    On Fri, Apr 21, 2017 at 4:58 PM Michael Ridley <mridley@cloudera.com> wrote:
>> 
>> Given that the project has not had a release, I don't see any reason to
>> stick with 1.6 support. Now seems like a good time to switch to 2 if that's
>> what people want to do. I haven't had time to do a deep dive on Spark 2 yet
>> so I don't have enough information to have a technical opinion, other than
>> that I hear a lot of excitement and preference for Spark 2.
>> 
>> Michael Ridley
>> Senior Solutions Architect
>> Cloudera
>> 
>> Sent from my mobile.
>> Pardon any spelling errors.
>> 
>>> On Apr 21, 2017, at 6:39 PM, Segerlind, Nathan L <
>> nathan.l.segerlind@intel.com> wrote:
>>> 
>>> Hi everybody.
>>> 
>>> There's been some talk about upgrading to Spark 2.1.
>>> 
>>> Do people think this is worthwhile?
>>> 
>>> Would others like to see continued support for 1.6? For how long and it
>> what capacity?
>>> 
>>> Should we maintain two branches?
>>> 
>>> Or perhaps drive the 2.1 branch forward and only send bug fixes to the
>> 1.6 branch for another year or so?
>>> 
>>> 
>> 
> 
> 

Mime
View raw message