tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <ss...@apache.org>
Subject Re: api compatibility within a minor release
Date Fri, 11 Dec 2015 04:10:59 GMT
Agree with Andre's point - users do not need to bother about how the data
is stored and retrieved, as long as it's accessible. Users should not need
to know details of ATS - or even about it's existence ideally. The
DAGClient monitors a dag for execution, and provides vertex progress. In
addition, it should provide task progress and counters. Users should not
need to worry about where this information comes from - the AM / ATS / some
other source.

@Bikas, the TezATSClient that you mention - is that meant to be used
internally by the DAGClient, or is that a replacement to the DAGClient or
is it an additional library that users will have to use or something else ?


On Thu, Dec 10, 2015 at 12:59 PM, Bikas Saha <bikas@apache.org> wrote:

> Yes. That is what I was trying to convey with a TezATSClient - it gives a
> Tez facade that provides expected semantics like getFooForVertex() etc. and
> hides the actual ATS details from the user.
>
> -----Original Message-----
> From: Andre Kelpe [mailto:akelpe@concurrentinc.com]
> Sent: Thursday, December 10, 2015 2:09 AM
> To: dev@tez.apache.org
> Subject: Re: api compatibility within a minor release
>
> From our point of view ATS is an implementation detail. As you said in
> Budapest, it changes all the time anyway, so it is not the API we would
> like to talk to. We care about the information stored in the ATS that is
> relevant for the current DAG, but we don't really care about the fact that
> the ATS is used. In MapReduce you also don't know, that the information is
> stored int he JobHistoryServer. There is an API on the RunningJob class to
> get to that information. We thing something along those lines would be
> good. Then you can change the DAGClient and the storage for the information
> all you want, we as consumers never have to think about it.
>
> - André
>
> On Wed, Dec 9, 2015 at 11:39 PM, Bikas Saha <bikas@apache.org> wrote:
>
> > Are we basically looking for detailed data that’s stored in the ATS?
> > Then should we consider creating a TezATSClient that understand Tez
> > semantics and fetches this information (in general for any DAG)
> > instead of augmenting DAGClient (that is responsible for submitting a
> > Tez DAG and monitoring a single DAG)?
> >
> > Bikas
> >
> > -----Original Message-----
> > From: Andre Kelpe [mailto:akelpe@concurrentinc.com]
> > Sent: Wednesday, December 9, 2015 10:32 AM
> > To: Hitesh Shah <hitesh@apache.org>
> > Cc: dev@tez.apache.org
> > Subject: Re: api compatibility within a minor release
> >
> > Hi,
> >
> > (undusting this thread after a while)
> >
> > I have opened https://issues.apache.org/jira/browse/TEZ-2981 to work
> > around the issue. We believe that would be the best way forward.
> >
> > - André
> >
> > On Tue, Sep 15, 2015 at 11:04 PM, Hitesh Shah <hitesh@apache.org> wrote:
> >
> > > Hi Andre,
> > >
> > > I don’t believe I saw a JIRA created for this? Has this issue been
> > > resolved?
> > >
> > > Also, it would be good if you can collate any particular private
> > > APIs that you are using so that we can be aware of potential issues
> > > with such
> > APIs.
> > >
> > > thanks
> > > — Hitesh
> > >
> > > On Aug 20, 2015, at 2:10 AM, Andre Kelpe <akelpe@concurrentinc.com>
> > wrote:
> > >
> > > > Thanks for the answer. We have a work-around for now. I am going
> > > > to make that inventory and submit a patch that changes the
> > > > annotations from @Private to @LimitedPrivate.
> > > >
> > > > From a semantic versioning point of view, I would still expect no
> > > breaking
> > > > changes in a bug fix release, but if the Tez community at large
> > > > can work with that, we have to accept that, I guess.
> > > >
> > > > - André
> > > >
> > > > On Tue, Aug 18, 2015 at 8:34 PM, Hitesh Shah <hitesh@apache.org>
> > wrote:
> > > >
> > > >> Hello Andre,
> > > >>
> > > >> For the most part, @Private is considered internal implementation
> > > >> and subject to change at any point. In this case, even more so as
> > > >> it is an *Impl class.
> > > >>
> > > >> What we can do is try the following:
> > > >>    - Look at all the various @Private classes being used by
> Cascading.
> > > >>    - See which ones should not be used at all and which ones can
> > > >> be considered to be a @LimitedPrivate for Cascading.
> > > >>    - For LimitedPrivate apis, we would then try to be a bit more
> > > >> careful with respect to changing/breaking these APIs. I would
> > > >> probably not say
> > > that
> > > >> they have will have the same guarantees as @Public/@Stable but we
> > > >> can
> > > work
> > > >> with the Cascading community to handle changes to these APIs in a
> > > workable
> > > >> manner on an ongoing basis.
> > > >>
> > > >> As for this API in question, for the short term fix, I guess a
> > > >> simple approach might be to introduce a backward compatible API (
> > > >> I believe one which does not need the timeout param passed in )?
> > > >> Would you mind
> > > filing a
> > > >> jira and hopefully provide a patch too?
> > > >>
> > > >> thanks
> > > >> — Hitesh
> > > >>
> > > >> On Aug 18, 2015, at 8:01 AM, Andre Kelpe
> > > >> <akelpe@concurrentinc.com>
> > > wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> I have found a small API incompatibility in the Tez 0.6.2 release.
> > > >>> The DAGClientTimelineImpl constructor got a new parameter for
> > time-outs.
> > > This
> > > >>> was not present in 0.6.1 and from a semantic versioning point
of
> > > >>> view,
> > > >> that
> > > >>> is problematic. I know that the class is marked with the
> > > >>> @Private annotation, but it would be great if such incompatible
> > > >>> things aren't introduced in a bug-fix release. It would be
> > > >>> easier for downsteam
> > > >> projects,
> > > >>> if you just added a second constructor.
> > > >>>
> > > >>> Is that an oversight or are classes with the @Private annotation
> > > subject
> > > >> to
> > > >>> change and dowstream projects simply have to deal with it?
> > > >>>
> > > >>> Thanks!
> > > >>>
> > > >>> - André
> > > >>>
> > > >>> --
> > > >>> André Kelpe
> > > >>> andre@concurrentinc.com
> > > >>> http://concurrentinc.com
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > André Kelpe
> > > > andre@concurrentinc.com
> > > > http://concurrentinc.com
> > >
> > >
> >
> >
> > --
> > André Kelpe
> > andre@concurrentinc.com
> > http://concurrentinc.com
> >
> >
>
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message