tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <ss...@apache.org>
Subject Re: api compatibility within a minor release
Date Sat, 19 Dec 2015 04:18:35 GMT
I think there's a jira for this already. I'm not sure it needs to be a
separate library in itself - since the DAGClient already provides
monitoring information, counters etc for Vertices.

On Fri, Dec 11, 2015 at 1:16 PM, Bikas Saha <bikas@apache.org> wrote:

> It could be used by DAGClient internally but essentially it’s an
> additional library that can be used to get detailed information about any
> dag - running or completed (vs DAGClient that interacts with the dag is
> submitted). To be clear, I am calling it TezATSClient in provide clarity as
> to what its doing for the context of this thread. It will hide the details
> of ATS from users and provide a Tez-centric view (independent of internal
> details). Hence the actual name of the client will not have ATS in it.
>
> If that makes sense, should we create jiras following that path?
>
> Bikas
>
> -----Original Message-----
> From: Siddharth Seth [mailto:sseth@apache.org]
> Sent: Thursday, December 10, 2015 8:11 PM
> To: dev@tez.apache.org
> Subject: Re: api compatibility within a minor release
>
> Agree with Andre's point - users do not need to bother about how the data
> is stored and retrieved, as long as it's accessible. Users should not need
> to know details of ATS - or even about it's existence ideally. The
> DAGClient monitors a dag for execution, and provides vertex progress. In
> addition, it should provide task progress and counters. Users should not
> need to worry about where this information comes from - the AM / ATS / some
> other source.
>
> @Bikas, the TezATSClient that you mention - is that meant to be used
> internally by the DAGClient, or is that a replacement to the DAGClient or
> is it an additional library that users will have to use or something else ?
>
>
> On Thu, Dec 10, 2015 at 12:59 PM, Bikas Saha <bikas@apache.org> wrote:
>
> > Yes. That is what I was trying to convey with a TezATSClient - it
> > gives a Tez facade that provides expected semantics like
> > getFooForVertex() etc. and hides the actual ATS details from the user.
> >
> > -----Original Message-----
> > From: Andre Kelpe [mailto:akelpe@concurrentinc.com]
> > Sent: Thursday, December 10, 2015 2:09 AM
> > To: dev@tez.apache.org
> > Subject: Re: api compatibility within a minor release
> >
> > From our point of view ATS is an implementation detail. As you said in
> > Budapest, it changes all the time anyway, so it is not the API we
> > would like to talk to. We care about the information stored in the ATS
> > that is relevant for the current DAG, but we don't really care about
> > the fact that the ATS is used. In MapReduce you also don't know, that
> > the information is stored int he JobHistoryServer. There is an API on
> > the RunningJob class to get to that information. We thing something
> > along those lines would be good. Then you can change the DAGClient and
> > the storage for the information all you want, we as consumers never have
> to think about it.
> >
> > - André
> >
> > On Wed, Dec 9, 2015 at 11:39 PM, Bikas Saha <bikas@apache.org> wrote:
> >
> > > Are we basically looking for detailed data that’s stored in the ATS?
> > > Then should we consider creating a TezATSClient that understand Tez
> > > semantics and fetches this information (in general for any DAG)
> > > instead of augmenting DAGClient (that is responsible for submitting
> > > a Tez DAG and monitoring a single DAG)?
> > >
> > > Bikas
> > >
> > > -----Original Message-----
> > > From: Andre Kelpe [mailto:akelpe@concurrentinc.com]
> > > Sent: Wednesday, December 9, 2015 10:32 AM
> > > To: Hitesh Shah <hitesh@apache.org>
> > > Cc: dev@tez.apache.org
> > > Subject: Re: api compatibility within a minor release
> > >
> > > Hi,
> > >
> > > (undusting this thread after a while)
> > >
> > > I have opened https://issues.apache.org/jira/browse/TEZ-2981 to work
> > > around the issue. We believe that would be the best way forward.
> > >
> > > - André
> > >
> > > On Tue, Sep 15, 2015 at 11:04 PM, Hitesh Shah <hitesh@apache.org>
> wrote:
> > >
> > > > Hi Andre,
> > > >
> > > > I don’t believe I saw a JIRA created for this? Has this issue been
> > > > resolved?
> > > >
> > > > Also, it would be good if you can collate any particular private
> > > > APIs that you are using so that we can be aware of potential
> > > > issues with such
> > > APIs.
> > > >
> > > > thanks
> > > > — Hitesh
> > > >
> > > > On Aug 20, 2015, at 2:10 AM, Andre Kelpe
> > > > <akelpe@concurrentinc.com>
> > > wrote:
> > > >
> > > > > Thanks for the answer. We have a work-around for now. I am going
> > > > > to make that inventory and submit a patch that changes the
> > > > > annotations from @Private to @LimitedPrivate.
> > > > >
> > > > > From a semantic versioning point of view, I would still expect
> > > > > no
> > > > breaking
> > > > > changes in a bug fix release, but if the Tez community at large
> > > > > can work with that, we have to accept that, I guess.
> > > > >
> > > > > - André
> > > > >
> > > > > On Tue, Aug 18, 2015 at 8:34 PM, Hitesh Shah <hitesh@apache.org>
> > > wrote:
> > > > >
> > > > >> Hello Andre,
> > > > >>
> > > > >> For the most part, @Private is considered internal
> > > > >> implementation and subject to change at any point. In this
> > > > >> case, even more so as it is an *Impl class.
> > > > >>
> > > > >> What we can do is try the following:
> > > > >>    - Look at all the various @Private classes being used by
> > Cascading.
> > > > >>    - See which ones should not be used at all and which ones
> > > > >> can be considered to be a @LimitedPrivate for Cascading.
> > > > >>    - For LimitedPrivate apis, we would then try to be a bit
> > > > >> more careful with respect to changing/breaking these APIs. I
> > > > >> would probably not say
> > > > that
> > > > >> they have will have the same guarantees as @Public/@Stable but
> > > > >> we can
> > > > work
> > > > >> with the Cascading community to handle changes to these APIs
in
> > > > >> a
> > > > workable
> > > > >> manner on an ongoing basis.
> > > > >>
> > > > >> As for this API in question, for the short term fix, I guess
a
> > > > >> simple approach might be to introduce a backward compatible API
> > > > >> ( I believe one which does not need the timeout param passed
in )?
> > > > >> Would you mind
> > > > filing a
> > > > >> jira and hopefully provide a patch too?
> > > > >>
> > > > >> thanks
> > > > >> — Hitesh
> > > > >>
> > > > >> On Aug 18, 2015, at 8:01 AM, Andre Kelpe
> > > > >> <akelpe@concurrentinc.com>
> > > > wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I have found a small API incompatibility in the Tez 0.6.2
> release.
> > > > >>> The DAGClientTimelineImpl constructor got a new parameter
for
> > > time-outs.
> > > > This
> > > > >>> was not present in 0.6.1 and from a semantic versioning point
> > > > >>> of view,
> > > > >> that
> > > > >>> is problematic. I know that the class is marked with the
> > > > >>> @Private annotation, but it would be great if such
> > > > >>> incompatible things aren't introduced in a bug-fix release.
It
> > > > >>> would be easier for downsteam
> > > > >> projects,
> > > > >>> if you just added a second constructor.
> > > > >>>
> > > > >>> Is that an oversight or are classes with the @Private
> > > > >>> annotation
> > > > subject
> > > > >> to
> > > > >>> change and dowstream projects simply have to deal with it?
> > > > >>>
> > > > >>> Thanks!
> > > > >>>
> > > > >>> - André
> > > > >>>
> > > > >>> --
> > > > >>> André Kelpe
> > > > >>> andre@concurrentinc.com
> > > > >>> http://concurrentinc.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > André Kelpe
> > > > > andre@concurrentinc.com
> > > > > http://concurrentinc.com
> > > >
> > > >
> > >
> > >
> > > --
> > > André Kelpe
> > > andre@concurrentinc.com
> > > http://concurrentinc.com
> > >
> > >
> >
> >
> > --
> > André Kelpe
> > andre@concurrentinc.com
> > http://concurrentinc.com
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message