trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandhya Sundaresan <sandhya.sundare...@esgyn.com>
Subject RE: Proposal to add hive regression tests to check-PR tests
Date Wed, 27 Jul 2016 16:27:17 GMT
We need to move them to core or split the hive tests into 2 suites. SO yes
code changes to test harness but too big a deal.
Sandhya

-----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Wednesday, July 27, 2016 9:24 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: Proposal to add hive regression tests to check-PR tests

Jenkins currently runs a suite by passing the suite name to the
core/sql/regress/tools/runallsb script.

How would a sub-set be implemented?

--Steve

> -----Original Message-----
> From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> Sent: Wednesday, July 27, 2016 9:21 AM
> To: dev <dev@trafodion.incubator.apache.org>
> Subject: Re: Proposal to add hive regression tests to check-PR tests
>
> Sure, TEST030 can be included which can finish in 1 minute.
>
> TEST009 tests external tables, and TEST030 has similar tests.  Maybe
> exclude TEST009?
>
> TEST001 should be included since it tests both hive/text and hive/ORC.
>
> --Qifan
>
>
>
> On Wed, Jul 27, 2016 at 11:09 AM, Sandhya Sundaresan <
> sandhya.sundaresan@esgyn.com> wrote:
>
> > Agree TEST018 is definitely a candidate. Even installer changes have
> > an effect on that test.
> > Will let other experts chime in for more suggestions .
> > Sandhya
> >
> > -----Original Message-----
> > From: Roberta Marton [mailto:roberta.marton@esgyn.com]
> > Sent: Wednesday, July 27, 2016 8:52 AM
> > To: dev@trafodion.incubator.apache.org
> > Subject: RE: Proposal to add hive regression tests to check-PR tests
> >
> > Perhaps we should choose tests that fail most frequently instead?
> > Tests 009, 018, and 030 comes to mind.
> >
> >       Roberta
> >
> > -----Original Message-----
> > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > Sent: Wednesday, July 27, 2016 6:36 AM
> > To: dev <dev@trafodion.incubator.apache.org>
> > Subject: Re: Proposal to add hive regression tests to check-PR tests
> >
> > Here is a tally of run-time (in minutes) of one run of the hive
> > regression.
> >
> > TEST1 1
> > TEST2 1
> > TEST3 3
> > TEST5 7
> > TEST6 1
> > TEST9 4
> > TEST15 5
> > TEST17 5
> > TEST18 6
> > TEST21 1
> > TEST30 1
> > TEST31 1
> > TEST33 4
> > TEST34 2
> > TEST35 3
> > TEST36 1
> >
> > The total time used is about 46 minutes.
> >
> > My vote will be to include a subset of from the above list what are
> > fast to run (say all 1 minute ones).  It probably will be a good
> > idea to keep them still in hive tests.
> >
> > Thanks
> >
> > --Qifan
> >
> >
> > On Tue, Jul 26, 2016 at 5:37 PM, Selva Govindarajan <
> > selva.govindarajan@esgyn.com> wrote:
> >
> > > Thanks Steve for resurrecting this discussion.
> > >
> > > Hive tests have been stabilized to a greater extent that we
> > > shouldn't have false failures now. Recently, there has been a
> > > quite amount of contribution coming in the area related to hive in
> > > Trafodion.  Hence I would vote +1 for adding hive tests to check PR.
> > >
> > > Selva
> > >
> > > -----Original Message-----
> > > From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> > > Sent: Tuesday, July 26, 2016 1:12 PM
> > > To: dev@trafodion.incubator.apache.org
> > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > tests
> > >
> > > Hi,
> > >
> > > I wanted to revisit this discussion to come to resolution.  There
> > > was a digression into the idea of dynamically choosing tests, but
> > > I'd like to come back to original proposal of adding an extra
> > > suite to the check tests.
> > >
> > > As I read the thread, there were several responses in support of
> > > the proposal, and a couple of reservations. The reservations
> > > include increasing the chance for false failures, which already
> > > can be a headache. Also the concern of adding long running tests
> > > that are included in hive versus maybe adding a few more small tests
> > > to core.
> > > Or perhaps using "extra tests" as needed, which is available on
> > > request.
> > >
> > > I'm willing to add another test job if that is what the community
> > > wants, but might it make more sense to more small tests to core or
> > > move some from hive to core?
> > >
> > > --Steve
> > >
> > > > -----Original Message-----
> > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > Sent: Monday, July 18, 2016 10:28 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > tests
> > > >
> > > > I totally agree with Steve to use a simple and predictable
> > > > mechanism to do check PR tests, If my memory serves me right,
> > > > prior to Trafodion becoming an Apache incubating project, hive
> > > > tests were part of check-PR. Because of unpredictable state of
> > > > hive regressions then it was decided to suspend
> > > > running hive regressions as part of check-in.   Based on the current
> > > state
> > > > of Trafodion, and the fact that the hive regressions have been
> > > > stabilized to a greater extent, it is important that this
> > > > stability is maintained by the future contributions. Recently
> > > > many contributions have come in hive-related area of the code.
> > > >
> > > > Adding hive regressions as part of check-PR should not increase
> > > > the overall time to complete the check-PR, but it would require
> > > > additional resources,.
> > > > Hence, Trafodion Jenkins infrastructure would incur additional cost.
> > > >
> > > > I am expecting the Trafodion Release Manager of R2.1 will help
> > > > us to determine with the community input what is the best option
> > > > to go with.
> > > >
> > > > Selva
> > > > -----Original Message-----
> > > > From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> > > > Sent: Monday, July 18, 2016 12:27 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > tests
> > > >
> > > > The current test process looks at which files have been modified
> > > > and puts it into a bucket, which is used to determine what tests to
> > > > run.
> > > > However, the only buckets that now exist are DOC and NONDOC.
> > > >
> > > > So if the change consists only of things in the docs/ tree, then
> > > > it only does static check and a docs build.  If there are
> > > > non-docs changes, it assumes it needs to run all the build and
> > > > tests.
> > > >
> > > > It is pretty conservative, but the more heuristics we put in to
> > > > customize the tests, the more chance that it will miss
> > > > something.  I can imagine a connectivity only change not running
> > > > the jobs that don't exercise connectivity. But figuring out what
> > > > things might affect hive tests seems much harder.
> > > >
> > > > There are many things (installer, executor,...) that might
> > > > affect any of our tests.  Seems safer to keep the test
> > > > heuristics very simple and predictable, and change the content
> > > > of the test suites to what ought to be in check versus nightly.
> > > >
> > > > --Steve
> > > >
> > > > > -----Original Message-----
> > > > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > > > Sent: Monday, July 18, 2016 9:43 AM
> > > > > To: dev <dev@trafodion.incubator.apache.org>
> > > > > Subject: Re: Proposal to add hive regression tests to check-PR
> > > > > tests
> > > > >
> > > > > The author just honestly describes the changes, and the tool
> > > > > picks the right tests.  Thanks --Qifan
> > > > >
> > > > > On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder
> > > > > <sean.broeder@esgyn.com>
> > > > > wrote:
> > > > >
> > > > > > I'd prefer not to leave it up to authors to select which
> > > > > > tests are appropriate.  Sometimes we get it right and others
> > > > > > we are horribly wrong.
> > > > > >
> > > > > > Thanks,
> > > > > > Sean
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > > > > Sent: Monday, July 18, 2016 9:20 AM
> > > > > > To: dev <dev@trafodion.incubator.apache.org>
> > > > > > Subject: Re: Proposal to add hive regression tests to
> > > > > > check-PR tests
> > > > > >
> > > > > > I agree with Sandhya and wonder if we can enhance check-PR
> > > > > > tests (hive
> > > > > for
> > > > > > example, in question) with the following twist.
> > > > > >
> > > > > >    1. Randomly select several (say 3) tests from
> > > > > > regression/hive.
> > > > > > The
> > > > > >    rational is that we only need to  sanity check the
> > > > > > changes and a full daily
> > > > > >    build with test will follow the merge.
> > > > > >    2. Before the check-in, we always run the full regression
> > > > > > test, and I do
> > > > > >    not see the value to run full Hive again in check-PR.
> > > > > >    3. In the future, we could find the most appreciate tests
> > > > > > for check-PR
> > > > > >    (instead of randomly select, or select the full set).
> > > > > > The author can point
> > > > > >    out the nature of the change and the check-in tool does
> > > > > > the selection.
> > > > > > For
> > > > > >    example, a change in DoP for Hbase tables will select
> > > > > > some tests from
> > > > > >    regress/seabase, but not from regress/hive.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > --Qifan
> > > > > >
> > > > > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan <
> > > > > > sandhya.sundaresan@esgyn.com> wrote:
> > > > > >
> > > > > > > +0 for me.
> > > > > > >  I am not sure of the need to add the whole test suite
 to
> > > > > > > check tests.
> > > > > > > The hive regressions do run nightly anyway so failures
> > > > > > > should be clear on each nightly run on a daily basis.
> > > > > > > My concern is that long running tests like hive/TEST018
  are
> > more
> > > > > > > to
> > > > > > test
> > > > > > > features like bulkload/unload and since we already have
> > > > > > > the option to run "extra tests" in Jenkins, I'm not sure
> > > > > > > bringing in entire test suites  into check tests is the
> > > > > > > right approach or trend going forward and  adding time
and
> > > > > > > resources to what is supposed to be a sanity test
> > > > > > for
> > > > > > > every single  PR.
> > > > > > >
> > > > > > > Sandhya
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Selva Govindarajan
> > > > > > > [mailto:selva.govindarajan@esgyn.com]
> > > > > > > Sent: Monday, July 18, 2016 7:22 AM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to
> > > > > > > check-PR tests
> > > > > > >
> > > > > > > Hive regressions takes little less than an hour. As I said
> > > > > > > before, the time is not a factor because the regressions
> > > > > > > are run in parallel in different VMs. Seabase regressions
> > > > > > > which is run as part of check-PR takes around 1 hour and
> > > > > > > 40 mins. Hence hive regressions shouldn't add more time
> > > > > > > for check-PR to complete, but of course it would need another
> > > > > > > VM.
> > > > > > >
> > > > > > > Selva
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jin, Jian (Seth) [mailto:Jian.jin@esgyn.cn]
> > > > > > > Sent: Sunday, July 17, 2016 7:31 PM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to
> > > > > > > check-PR tests
> > > > > > >
> > > > > > > How long will it take for Hive regression?
> > > > > > >
> > > > > > > Br,
> > > > > > >
> > > > > > > Seth
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> > > > > > > Sent: 2016年7月16日 9:16
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to
> > > > > > > check-PR tests
> > > > > > >
> > > > > > > +1 to this
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Selva Govindarajan
> > > > > > > [mailto:selva.govindarajan@esgyn.com]
> > > > > > > Sent: Saturday, July 16, 2016 9:08 AM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: Proposal to add hive regression tests to check-PR
> > > > > > > tests
> > > > > > >
> > > > > > > If you have subscribed to Trafodion Daily Build, you would
> > > > > > > have noticed that the daily build has been failing for
> > > > > > > some days.
> > > > > > > Most often, it is due the failure in hive regression tests
> > > > > > > run as part of the daily build.
> > > > > > > Lately, there has been some conscious effort made
> > > > > > > successfully to ensure that the hive regression tests can
be
> > > > > > > run reliably.
> > > > > > > To maintain the Trafodion daily build in that state, I
am
> > > > > > > proposing to include hive regressions to check-PR tests.
> > > > > > > It shouldn’t add the overall time taken to regressions
> > > > > > > tests because tests are run in parallel on different VMs,
> > > > > > > though it would consume more resources.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -          Selva
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards, --Qifan
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards, --Qifan
> > >
> >
> >
> >
> > --
> > Regards, --Qifan
> >
>
>
>
> --
> Regards, --Qifan

Mime
View raw message