spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Grover <m...@apache.org>
Subject Re: YARN Shuffle service and its compatibility
Date Tue, 19 Apr 2016 16:47:33 GMT
On Tue, Apr 19, 2016 at 2:26 AM, Steve Loughran <stevel@hortonworks.com>
wrote:

>
> > On 18 Apr 2016, at 23:05, Marcelo Vanzin <vanzin@cloudera.com> wrote:
> >
> > On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <rxin@databricks.com>
> wrote:
> >> The bigger problem is that it is much easier to maintain backward
> >> compatibility rather than dictating forward compatibility. For example,
> as
> >> Marcin said, if we come up with a slightly different shuffle layout to
> >> improve shuffle performance, we wouldn't be able to do that if we want
> to
> >> allow Spark 1.6 shuffle service to read something generated by Spark
> 2.1.
> >
> > And I think that's really what Mark is proposing. Basically, "don't
> > intentionally break backwards compatibility unless it's really
> > required" (e.g. SPARK-12130). That would allow option B to work.
> >
> > If a new shuffle manager is created, then neither option A nor option
> > B would really work. Moving all the shuffle-related classes to a
> > different package, to support option A, would be really messy. At that
> > point, you're better off maintaining the new shuffle service outside
> > of YARN, which is rather messy too.
> >
>
>
> There's a WiP in YARN to move Aux NM services into their own CP, though
> that doesn't address shared native libs, such as the leveldb support that
> went into 1.6
>
>
> There's already been some fun with Jackson versions and that of Hadoop —
> SPARK-12807; something that per-service classpaths would fix.
>
> would having separate CPs allow multiple spark shuffle JARs to be loaded,
> as long as everything bonded to the right one?

I just checked out https://issues.apache.org/jira/browse/YARN-1593. It's
hard to say if it'd help or not, I wasn't able to find any design doc or
patch attached to that JIRA. If there were a way to specify different JAR
names/locations for starting the separate process, it would work but if the
start happened by pointing to a full class name, that comes back to Option
A, and we'd have to do a good chunk of name/version spacing in order to
isolate.

Mime
View raw message