trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qifan Chen <qifan.c...@esgyn.com>
Subject Re: Proposal to add hive regression tests to check-PR tests
Date Tue, 19 Jul 2016 11:58:12 GMT
Hi Steve and Selva,

I would think the because the check-PR tests are stable now, it is OK to
optimize it. In particular to reduce the redundancy and to help double
check the areas that may be missed by the unit regression test. In this
sense, I envision that the check-PR may be a super set of the unit test. I
know some of it is true already.

I like the idea of simplicity and predicability and think if each test is
of such quality, then the optimized execution of a sub-set of it would be
too.

The other question I was thinking is that if *somehow* we can verify that
the full unit regression test was run after the code change, and the
results are good, can we run even less number of check-PR tests? This is to
address the redundancy concern.

Thanks --Qifan

On Tue, Jul 19, 2016 at 12:27 AM, Selva Govindarajan <
selva.govindarajan@esgyn.com> wrote:

> I totally agree with Steve to use a simple and predictable mechanism to do
> check PR tests, If my memory serves me right, prior to Trafodion becoming
> an
> Apache incubating project, hive tests were part of check-PR. Because of
> unpredictable state of hive regressions then it was decided to suspend
> running hive regressions as part of check-in.   Based on the current state
> of Trafodion, and the fact that the hive regressions have been stabilized
> to
> a greater extent, it is important that this stability is maintained by the
> future contributions. Recently many contributions have come in hive-related
> area of the code.
>
> Adding hive regressions as part of check-PR should not increase the overall
> time to complete the check-PR, but it would require additional resources,.
> Hence, Trafodion Jenkins infrastructure would incur additional cost.
>
> I am expecting the Trafodion Release Manager of R2.1 will help us to
> determine with the community input what is the best option to go with.
>
> Selva
> -----Original Message-----
> From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> Sent: Monday, July 18, 2016 12:27 PM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Proposal to add hive regression tests to check-PR tests
>
> The current test process looks at which files have been modified and puts
> it
> into a bucket, which is used to determine what tests to run.
> However, the only buckets that now exist are DOC and NONDOC.
>
> So if the change consists only of things in the docs/ tree, then it only
> does static check and a docs build.  If there are non-docs changes, it
> assumes it needs to run all the build and tests.
>
> It is pretty conservative, but the more heuristics we put in to customize
> the tests, the more chance that it will miss something.  I can imagine a
> connectivity only change not running the jobs that don't exercise
> connectivity. But figuring out what things might affect hive tests seems
> much harder.
>
> There are many things (installer, executor,...) that might affect any of
> our
> tests.  Seems safer to keep the test heuristics very simple and
> predictable,
> and change the content of the test suites to what ought to be in check
> versus nightly.
>
> --Steve
>
> > -----Original Message-----
> > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > Sent: Monday, July 18, 2016 9:43 AM
> > To: dev <dev@trafodion.incubator.apache.org>
> > Subject: Re: Proposal to add hive regression tests to check-PR tests
> >
> > The author just honestly describes the changes, and the tool picks the
> > right tests.  Thanks --Qifan
> >
> > On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder
> > <sean.broeder@esgyn.com>
> > wrote:
> >
> > > I'd prefer not to leave it up to authors to select which tests are
> > > appropriate.  Sometimes we get it right and others we are horribly
> > > wrong.
> > >
> > > Thanks,
> > > Sean
> > >
> > > -----Original Message-----
> > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > Sent: Monday, July 18, 2016 9:20 AM
> > > To: dev <dev@trafodion.incubator.apache.org>
> > > Subject: Re: Proposal to add hive regression tests to check-PR tests
> > >
> > > I agree with Sandhya and wonder if we can enhance check-PR tests (hive
> > for
> > > example, in question) with the following twist.
> > >
> > >    1. Randomly select several (say 3) tests from regression/hive. The
> > >    rational is that we only need to  sanity check the changes and a
> full
> > > daily
> > >    build with test will follow the merge.
> > >    2. Before the check-in, we always run the full regression test, and
> I
> > > do
> > >    not see the value to run full Hive again in check-PR.
> > >    3. In the future, we could find the most appreciate tests for
> > > check-PR
> > >    (instead of randomly select, or select the full set).  The author
> can
> > > point
> > >    out the nature of the change and the check-in tool does the
> > > selection.
> > > For
> > >    example, a change in DoP for Hbase tables will select some tests
> from
> > >    regress/seabase, but not from regress/hive.
> > >
> > > Thanks
> > >
> > > --Qifan
> > >
> > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan <
> > > sandhya.sundaresan@esgyn.com> wrote:
> > >
> > > > +0 for me.
> > > >  I am not sure of the need to add the whole test suite  to check
> > > > tests.
> > > > The hive regressions do run nightly anyway so failures should be
> clear
> > > > on each nightly run on a daily basis.
> > > > My concern is that long running tests like hive/TEST018   are more to
> > > test
> > > > features like bulkload/unload and since we already have the option to
> > > > run "extra tests" in Jenkins, I'm not sure bringing in entire test
> > > > suites  into check tests is the right approach or trend going forward
> > > > and  adding time and resources to what is supposed to be a sanity
> test
> > > for
> > > > every single  PR.
> > > >
> > > > Sandhya
> > > >
> > > > -----Original Message-----
> > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > Sent: Monday, July 18, 2016 7:22 AM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR tests
> > > >
> > > > Hive regressions takes little less than an hour. As I said before,
> the
> > > > time is not a factor because the regressions are run in parallel in
> > > > different VMs. Seabase regressions which is run as part of check-PR
> > > > takes around 1 hour and 40 mins. Hence hive regressions shouldn't add
> > > > more time for check-PR to complete, but of course it would need
> > > > another
> > > > VM.
> > > >
> > > > Selva
> > > >
> > > > -----Original Message-----
> > > > From: Jin, Jian (Seth) [mailto:Jian.jin@esgyn.cn]
> > > > Sent: Sunday, July 17, 2016 7:31 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR tests
> > > >
> > > > How long will it take for Hive regression?
> > > >
> > > > Br,
> > > >
> > > > Seth
> > > >
> > > > -----Original Message-----
> > > > From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> > > > Sent: 2016年7月16日 9:16
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR tests
> > > >
> > > > +1 to this
> > > >
> > > > -----Original Message-----
> > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > Sent: Saturday, July 16, 2016 9:08 AM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: Proposal to add hive regression tests to check-PR tests
> > > >
> > > > If you have subscribed to Trafodion Daily Build, you would have
> > > > noticed that the daily build has been failing for some days. Most
> > > > often, it is due the failure in hive regression tests run as part of
> > > > the daily build.
> > > > Lately, there has been some conscious effort made successfully to
> > > > ensure that the hive regression tests can be run reliably. To
> maintain
> > > > the Trafodion daily build in that state, I am proposing to include
> > > > hive regressions to check-PR tests.  It shouldn’t add the overall
> time
> > > > taken to regressions tests because tests are run in parallel on
> > > > different VMs, though it would consume more resources.
> > > >
> > > >
> > > >
> > > > -          Selva
> > > >
> > >
> > >
> > >
> > > --
> > > Regards, --Qifan
> > >
> >
> >
> >
> > --
> > Regards, --Qifan
>



-- 
Regards, --Qifan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message