trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Varnau <steve.var...@esgyn.com>
Subject RE: Proposal to add hive regression tests to check-PR tests
Date Wed, 27 Jul 2016 16:23:56 GMT
Jenkins currently runs a suite by passing the suite name to the
core/sql/regress/tools/runallsb script.

How would a sub-set be implemented?

--Steve

> -----Original Message-----
> From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> Sent: Wednesday, July 27, 2016 9:21 AM
> To: dev <dev@trafodion.incubator.apache.org>
> Subject: Re: Proposal to add hive regression tests to check-PR tests
>
> Sure, TEST030 can be included which can finish in 1 minute.
>
> TEST009 tests external tables, and TEST030 has similar tests.  Maybe
> exclude TEST009?
>
> TEST001 should be included since it tests both hive/text and hive/ORC.
>
> --Qifan
>
>
>
> On Wed, Jul 27, 2016 at 11:09 AM, Sandhya Sundaresan <
> sandhya.sundaresan@esgyn.com> wrote:
>
> > Agree TEST018 is definitely a candidate. Even installer changes have an
> > effect on that test.
> > Will let other experts chime in for more suggestions .
> > Sandhya
> >
> > -----Original Message-----
> > From: Roberta Marton [mailto:roberta.marton@esgyn.com]
> > Sent: Wednesday, July 27, 2016 8:52 AM
> > To: dev@trafodion.incubator.apache.org
> > Subject: RE: Proposal to add hive regression tests to check-PR tests
> >
> > Perhaps we should choose tests that fail most frequently instead? Tests
> > 009,
> > 018, and 030 comes to mind.
> >
> >       Roberta
> >
> > -----Original Message-----
> > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > Sent: Wednesday, July 27, 2016 6:36 AM
> > To: dev <dev@trafodion.incubator.apache.org>
> > Subject: Re: Proposal to add hive regression tests to check-PR tests
> >
> > Here is a tally of run-time (in minutes) of one run of the hive
> > regression.
> >
> > TEST1 1
> > TEST2 1
> > TEST3 3
> > TEST5 7
> > TEST6 1
> > TEST9 4
> > TEST15 5
> > TEST17 5
> > TEST18 6
> > TEST21 1
> > TEST30 1
> > TEST31 1
> > TEST33 4
> > TEST34 2
> > TEST35 3
> > TEST36 1
> >
> > The total time used is about 46 minutes.
> >
> > My vote will be to include a subset of from the above list what are fast
> > to
> > run (say all 1 minute ones).  It probably will be a good idea to keep
> > them
> > still in hive tests.
> >
> > Thanks
> >
> > --Qifan
> >
> >
> > On Tue, Jul 26, 2016 at 5:37 PM, Selva Govindarajan <
> > selva.govindarajan@esgyn.com> wrote:
> >
> > > Thanks Steve for resurrecting this discussion.
> > >
> > > Hive tests have been stabilized to a greater extent that we shouldn't
> > > have false failures now. Recently, there has been a quite amount of
> > > contribution coming in the area related to hive in Trafodion.  Hence I
> > > would vote +1 for adding hive tests to check PR.
> > >
> > > Selva
> > >
> > > -----Original Message-----
> > > From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> > > Sent: Tuesday, July 26, 2016 1:12 PM
> > > To: dev@trafodion.incubator.apache.org
> > > Subject: RE: Proposal to add hive regression tests to check-PR tests
> > >
> > > Hi,
> > >
> > > I wanted to revisit this discussion to come to resolution.  There was
> > > a digression into the idea of dynamically choosing tests, but I'd like
> > > to come back to original proposal of adding an extra suite to the
> > > check tests.
> > >
> > > As I read the thread, there were several responses in support of the
> > > proposal, and a couple of reservations. The reservations include
> > > increasing the chance for false failures, which already can be a
> > > headache. Also the concern of adding long running tests that are
> > > included in hive versus maybe adding a few more small tests to core.
> > > Or perhaps using "extra tests" as needed, which is available on
> > > request.
> > >
> > > I'm willing to add another test job if that is what the community
> > > wants, but might it make more sense to more small tests to core or
> > > move some from hive to core?
> > >
> > > --Steve
> > >
> > > > -----Original Message-----
> > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > Sent: Monday, July 18, 2016 10:28 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR tests
> > > >
> > > > I totally agree with Steve to use a simple and predictable mechanism
> > > > to do check PR tests, If my memory serves me right, prior to
> > > > Trafodion becoming an Apache incubating project, hive tests were
> > > > part of check-PR. Because of unpredictable state of hive regressions
> > > > then it was decided to suspend
> > > > running hive regressions as part of check-in.   Based on the current
> > > state
> > > > of Trafodion, and the fact that the hive regressions have been
> > > > stabilized to a greater extent, it is important that this stability
> > > > is maintained by the future contributions. Recently many
> > > > contributions have come in hive-related area of the code.
> > > >
> > > > Adding hive regressions as part of check-PR should not increase the
> > > > overall time to complete the check-PR, but it would require
> > > > additional resources,.
> > > > Hence, Trafodion Jenkins infrastructure would incur additional cost.
> > > >
> > > > I am expecting the Trafodion Release Manager of R2.1 will help us to
> > > > determine with the community input what is the best option to go
> > > > with.
> > > >
> > > > Selva
> > > > -----Original Message-----
> > > > From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> > > > Sent: Monday, July 18, 2016 12:27 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR tests
> > > >
> > > > The current test process looks at which files have been modified and
> > > > puts it into a bucket, which is used to determine what tests to run.
> > > > However, the only buckets that now exist are DOC and NONDOC.
> > > >
> > > > So if the change consists only of things in the docs/ tree, then it
> > > > only does static check and a docs build.  If there are non-docs
> > > > changes, it assumes it needs to run all the build and tests.
> > > >
> > > > It is pretty conservative, but the more heuristics we put in to
> > > > customize the tests, the more chance that it will miss something.  I
> > > > can imagine a connectivity only change not running the jobs that
> > > > don't exercise connectivity. But figuring out what things might
> > > > affect hive tests seems much harder.
> > > >
> > > > There are many things (installer, executor,...) that might affect
> > > > any of our tests.  Seems safer to keep the test heuristics very
> > > > simple and predictable, and change the content of the test suites to
> > > > what ought to be in check versus nightly.
> > > >
> > > > --Steve
> > > >
> > > > > -----Original Message-----
> > > > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > > > Sent: Monday, July 18, 2016 9:43 AM
> > > > > To: dev <dev@trafodion.incubator.apache.org>
> > > > > Subject: Re: Proposal to add hive regression tests to check-PR
> > > > > tests
> > > > >
> > > > > The author just honestly describes the changes, and the tool picks
> > > > > the right tests.  Thanks --Qifan
> > > > >
> > > > > On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder
> > > > > <sean.broeder@esgyn.com>
> > > > > wrote:
> > > > >
> > > > > > I'd prefer not to leave it up to authors to select which tests
> > > > > > are appropriate.  Sometimes we get it right and others we are
> > > > > > horribly wrong.
> > > > > >
> > > > > > Thanks,
> > > > > > Sean
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > > > > Sent: Monday, July 18, 2016 9:20 AM
> > > > > > To: dev <dev@trafodion.incubator.apache.org>
> > > > > > Subject: Re: Proposal to add hive regression tests to check-PR
> > > > > > tests
> > > > > >
> > > > > > I agree with Sandhya and wonder if we can enhance check-PR tests
> > > > > > (hive
> > > > > for
> > > > > > example, in question) with the following twist.
> > > > > >
> > > > > >    1. Randomly select several (say 3) tests from
> > > > > > regression/hive.
> > > > > > The
> > > > > >    rational is that we only need to  sanity check the changes
> > > > > > and a full daily
> > > > > >    build with test will follow the merge.
> > > > > >    2. Before the check-in, we always run the full regression
> > > > > > test, and I do
> > > > > >    not see the value to run full Hive again in check-PR.
> > > > > >    3. In the future, we could find the most appreciate tests
for
> > > > > > check-PR
> > > > > >    (instead of randomly select, or select the full set).  The
> > > > > > author can point
> > > > > >    out the nature of the change and the check-in tool does the
> > > > > > selection.
> > > > > > For
> > > > > >    example, a change in DoP for Hbase tables will select some
> > > > > > tests from
> > > > > >    regress/seabase, but not from regress/hive.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > --Qifan
> > > > > >
> > > > > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan <
> > > > > > sandhya.sundaresan@esgyn.com> wrote:
> > > > > >
> > > > > > > +0 for me.
> > > > > > >  I am not sure of the need to add the whole test suite
 to
> > > > > > > check tests.
> > > > > > > The hive regressions do run nightly anyway so failures
should
> > > > > > > be clear on each nightly run on a daily basis.
> > > > > > > My concern is that long running tests like hive/TEST018
  are
> > more
> > > > > > > to
> > > > > > test
> > > > > > > features like bulkload/unload and since we already have
the
> > > > > > > option to run "extra tests" in Jenkins, I'm not sure bringing
> > > > > > > in entire test suites  into check tests is the right approach
> > > > > > > or trend going forward and  adding time and resources to
what
> > > > > > > is supposed to be a sanity test
> > > > > > for
> > > > > > > every single  PR.
> > > > > > >
> > > > > > > Sandhya
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > > > > Sent: Monday, July 18, 2016 7:22 AM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > > > > tests
> > > > > > >
> > > > > > > Hive regressions takes little less than an hour. As I said
> > > > > > > before, the time is not a factor because the regressions
are
> > > > > > > run in parallel in different VMs. Seabase regressions which
is
> > > > > > > run as part of check-PR takes around 1 hour and 40 mins.
Hence
> > > > > > > hive regressions shouldn't add more time for check-PR to
> > > > > > > complete, but of course it would need another VM.
> > > > > > >
> > > > > > > Selva
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jin, Jian (Seth) [mailto:Jian.jin@esgyn.cn]
> > > > > > > Sent: Sunday, July 17, 2016 7:31 PM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > > > > tests
> > > > > > >
> > > > > > > How long will it take for Hive regression?
> > > > > > >
> > > > > > > Br,
> > > > > > >
> > > > > > > Seth
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> > > > > > > Sent: 2016年7月16日 9:16
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > > > > tests
> > > > > > >
> > > > > > > +1 to this
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > > > > Sent: Saturday, July 16, 2016 9:08 AM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: Proposal to add hive regression tests to check-PR
> > > > > > > tests
> > > > > > >
> > > > > > > If you have subscribed to Trafodion Daily Build, you would
> > > > > > > have noticed that the daily build has been failing for
some
> > > > > > > days.
> > > > > > > Most often, it is due the failure in hive regression tests
run
> > > > > > > as part of the daily build.
> > > > > > > Lately, there has been some conscious effort made successfully
> > > > > > > to ensure that the hive regression tests can be run reliably.
> > > > > > > To maintain the Trafodion daily build in that state, I
am
> > > > > > > proposing to include hive regressions to check-PR tests.
 It
> > > > > > > shouldn’t add the overall time taken to regressions tests
> > > > > > > because tests are run in parallel on different VMs, though
it
> > > > > > > would consume more resources.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -          Selva
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards, --Qifan
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards, --Qifan
> > >
> >
> >
> >
> > --
> > Regards, --Qifan
> >
>
>
>
> --
> Regards, --Qifan

Mime
View raw message