trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Selva Govindarajan <selva.govindara...@esgyn.com>
Subject RE: Proposal to add hive regression tests to check-PR tests
Date Wed, 27 Jul 2016 17:19:57 GMT
The decision to split the test suite should be based on the stability and
predictability of the test rather than how long it takes. The long pole for
check-PR tests is seabase and it takes 1hr40mins to complete. But, hive
takes 45mins to complete. Because these tests are run in different Jenkins
VM at the same time, hive tests wouldn't extend the time taken for check-PR
to complete.

I am unable to comprehend the need for split other than how long it would
take to run the tests.

Selva

-----Original Message-----
From: Sandhya Sundaresan [mailto:sandhya.sundaresan@esgyn.com]
Sent: Wednesday, July 27, 2016 9:27 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: Proposal to add hive regression tests to check-PR tests

We need to move them to core or split the hive tests into 2 suites. SO yes
code changes to test harness but too big a deal.
Sandhya

-----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Wednesday, July 27, 2016 9:24 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: Proposal to add hive regression tests to check-PR tests

Jenkins currently runs a suite by passing the suite name to the
core/sql/regress/tools/runallsb script.

How would a sub-set be implemented?

--Steve

> -----Original Message-----
> From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> Sent: Wednesday, July 27, 2016 9:21 AM
> To: dev <dev@trafodion.incubator.apache.org>
> Subject: Re: Proposal to add hive regression tests to check-PR tests
>
> Sure, TEST030 can be included which can finish in 1 minute.
>
> TEST009 tests external tables, and TEST030 has similar tests.  Maybe
> exclude TEST009?
>
> TEST001 should be included since it tests both hive/text and hive/ORC.
>
> --Qifan
>
>
>
> On Wed, Jul 27, 2016 at 11:09 AM, Sandhya Sundaresan <
> sandhya.sundaresan@esgyn.com> wrote:
>
> > Agree TEST018 is definitely a candidate. Even installer changes have
> > an effect on that test.
> > Will let other experts chime in for more suggestions .
> > Sandhya
> >
> > -----Original Message-----
> > From: Roberta Marton [mailto:roberta.marton@esgyn.com]
> > Sent: Wednesday, July 27, 2016 8:52 AM
> > To: dev@trafodion.incubator.apache.org
> > Subject: RE: Proposal to add hive regression tests to check-PR tests
> >
> > Perhaps we should choose tests that fail most frequently instead?
> > Tests 009, 018, and 030 comes to mind.
> >
> >       Roberta
> >
> > -----Original Message-----
> > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > Sent: Wednesday, July 27, 2016 6:36 AM
> > To: dev <dev@trafodion.incubator.apache.org>
> > Subject: Re: Proposal to add hive regression tests to check-PR tests
> >
> > Here is a tally of run-time (in minutes) of one run of the hive
> > regression.
> >
> > TEST1 1
> > TEST2 1
> > TEST3 3
> > TEST5 7
> > TEST6 1
> > TEST9 4
> > TEST15 5
> > TEST17 5
> > TEST18 6
> > TEST21 1
> > TEST30 1
> > TEST31 1
> > TEST33 4
> > TEST34 2
> > TEST35 3
> > TEST36 1
> >
> > The total time used is about 46 minutes.
> >
> > My vote will be to include a subset of from the above list what are
> > fast to run (say all 1 minute ones).  It probably will be a good
> > idea to keep them still in hive tests.
> >
> > Thanks
> >
> > --Qifan
> >
> >
> > On Tue, Jul 26, 2016 at 5:37 PM, Selva Govindarajan <
> > selva.govindarajan@esgyn.com> wrote:
> >
> > > Thanks Steve for resurrecting this discussion.
> > >
> > > Hive tests have been stabilized to a greater extent that we
> > > shouldn't have false failures now. Recently, there has been a
> > > quite amount of contribution coming in the area related to hive in
> > > Trafodion.  Hence I would vote +1 for adding hive tests to check PR.
> > >
> > > Selva
> > >
> > > -----Original Message-----
> > > From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> > > Sent: Tuesday, July 26, 2016 1:12 PM
> > > To: dev@trafodion.incubator.apache.org
> > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > tests
> > >
> > > Hi,
> > >
> > > I wanted to revisit this discussion to come to resolution.  There
> > > was a digression into the idea of dynamically choosing tests, but
> > > I'd like to come back to original proposal of adding an extra
> > > suite to the check tests.
> > >
> > > As I read the thread, there were several responses in support of
> > > the proposal, and a couple of reservations. The reservations
> > > include increasing the chance for false failures, which already
> > > can be a headache. Also the concern of adding long running tests
> > > that are included in hive versus maybe adding a few more small
> > > tests to core.
> > > Or perhaps using "extra tests" as needed, which is available on
> > > request.
> > >
> > > I'm willing to add another test job if that is what the community
> > > wants, but might it make more sense to more small tests to core or
> > > move some from hive to core?
> > >
> > > --Steve
> > >
> > > > -----Original Message-----
> > > > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> > > > Sent: Monday, July 18, 2016 10:28 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > tests
> > > >
> > > > I totally agree with Steve to use a simple and predictable
> > > > mechanism to do check PR tests, If my memory serves me right,
> > > > prior to Trafodion becoming an Apache incubating project, hive
> > > > tests were part of check-PR. Because of unpredictable state of
> > > > hive regressions then it was decided to suspend
> > > > running hive regressions as part of check-in.   Based on the current
> > > state
> > > > of Trafodion, and the fact that the hive regressions have been
> > > > stabilized to a greater extent, it is important that this
> > > > stability is maintained by the future contributions. Recently
> > > > many contributions have come in hive-related area of the code.
> > > >
> > > > Adding hive regressions as part of check-PR should not increase
> > > > the overall time to complete the check-PR, but it would require
> > > > additional resources,.
> > > > Hence, Trafodion Jenkins infrastructure would incur additional cost.
> > > >
> > > > I am expecting the Trafodion Release Manager of R2.1 will help
> > > > us to determine with the community input what is the best option
> > > > to go with.
> > > >
> > > > Selva
> > > > -----Original Message-----
> > > > From: Steve Varnau [mailto:steve.varnau@esgyn.com]
> > > > Sent: Monday, July 18, 2016 12:27 PM
> > > > To: dev@trafodion.incubator.apache.org
> > > > Subject: RE: Proposal to add hive regression tests to check-PR
> > > > tests
> > > >
> > > > The current test process looks at which files have been modified
> > > > and puts it into a bucket, which is used to determine what tests
> > > > to run.
> > > > However, the only buckets that now exist are DOC and NONDOC.
> > > >
> > > > So if the change consists only of things in the docs/ tree, then
> > > > it only does static check and a docs build.  If there are
> > > > non-docs changes, it assumes it needs to run all the build and
> > > > tests.
> > > >
> > > > It is pretty conservative, but the more heuristics we put in to
> > > > customize the tests, the more chance that it will miss
> > > > something.  I can imagine a connectivity only change not running
> > > > the jobs that don't exercise connectivity. But figuring out what
> > > > things might affect hive tests seems much harder.
> > > >
> > > > There are many things (installer, executor,...) that might
> > > > affect any of our tests.  Seems safer to keep the test
> > > > heuristics very simple and predictable, and change the content
> > > > of the test suites to what ought to be in check versus nightly.
> > > >
> > > > --Steve
> > > >
> > > > > -----Original Message-----
> > > > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > > > Sent: Monday, July 18, 2016 9:43 AM
> > > > > To: dev <dev@trafodion.incubator.apache.org>
> > > > > Subject: Re: Proposal to add hive regression tests to check-PR
> > > > > tests
> > > > >
> > > > > The author just honestly describes the changes, and the tool
> > > > > picks the right tests.  Thanks --Qifan
> > > > >
> > > > > On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder
> > > > > <sean.broeder@esgyn.com>
> > > > > wrote:
> > > > >
> > > > > > I'd prefer not to leave it up to authors to select which
> > > > > > tests are appropriate.  Sometimes we get it right and others
> > > > > > we are horribly wrong.
> > > > > >
> > > > > > Thanks,
> > > > > > Sean
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> > > > > > Sent: Monday, July 18, 2016 9:20 AM
> > > > > > To: dev <dev@trafodion.incubator.apache.org>
> > > > > > Subject: Re: Proposal to add hive regression tests to
> > > > > > check-PR tests
> > > > > >
> > > > > > I agree with Sandhya and wonder if we can enhance check-PR
> > > > > > tests (hive
> > > > > for
> > > > > > example, in question) with the following twist.
> > > > > >
> > > > > >    1. Randomly select several (say 3) tests from
> > > > > > regression/hive.
> > > > > > The
> > > > > >    rational is that we only need to  sanity check the
> > > > > > changes and a full daily
> > > > > >    build with test will follow the merge.
> > > > > >    2. Before the check-in, we always run the full regression
> > > > > > test, and I do
> > > > > >    not see the value to run full Hive again in check-PR.
> > > > > >    3. In the future, we could find the most appreciate tests
> > > > > > for check-PR
> > > > > >    (instead of randomly select, or select the full set).
> > > > > > The author can point
> > > > > >    out the nature of the change and the check-in tool does
> > > > > > the selection.
> > > > > > For
> > > > > >    example, a change in DoP for Hbase tables will select
> > > > > > some tests from
> > > > > >    regress/seabase, but not from regress/hive.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > --Qifan
> > > > > >
> > > > > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan <
> > > > > > sandhya.sundaresan@esgyn.com> wrote:
> > > > > >
> > > > > > > +0 for me.
> > > > > > >  I am not sure of the need to add the whole test suite
 to
> > > > > > > check tests.
> > > > > > > The hive regressions do run nightly anyway so failures
> > > > > > > should be clear on each nightly run on a daily basis.
> > > > > > > My concern is that long running tests like hive/TEST018
  are
> > more
> > > > > > > to
> > > > > > test
> > > > > > > features like bulkload/unload and since we already have
> > > > > > > the option to run "extra tests" in Jenkins, I'm not sure
> > > > > > > bringing in entire test suites  into check tests is the
> > > > > > > right approach or trend going forward and  adding time
and
> > > > > > > resources to what is supposed to be a sanity test
> > > > > > for
> > > > > > > every single  PR.
> > > > > > >
> > > > > > > Sandhya
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Selva Govindarajan
> > > > > > > [mailto:selva.govindarajan@esgyn.com]
> > > > > > > Sent: Monday, July 18, 2016 7:22 AM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to
> > > > > > > check-PR tests
> > > > > > >
> > > > > > > Hive regressions takes little less than an hour. As I said
> > > > > > > before, the time is not a factor because the regressions
> > > > > > > are run in parallel in different VMs. Seabase regressions
> > > > > > > which is run as part of check-PR takes around 1 hour and
> > > > > > > 40 mins. Hence hive regressions shouldn't add more time
> > > > > > > for check-PR to complete, but of course it would need
> > > > > > > another VM.
> > > > > > >
> > > > > > > Selva
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jin, Jian (Seth) [mailto:Jian.jin@esgyn.cn]
> > > > > > > Sent: Sunday, July 17, 2016 7:31 PM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to
> > > > > > > check-PR tests
> > > > > > >
> > > > > > > How long will it take for Hive regression?
> > > > > > >
> > > > > > > Br,
> > > > > > >
> > > > > > > Seth
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> > > > > > > Sent: 2016年7月16日 9:16
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: RE: Proposal to add hive regression tests to
> > > > > > > check-PR tests
> > > > > > >
> > > > > > > +1 to this
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Selva Govindarajan
> > > > > > > [mailto:selva.govindarajan@esgyn.com]
> > > > > > > Sent: Saturday, July 16, 2016 9:08 AM
> > > > > > > To: dev@trafodion.incubator.apache.org
> > > > > > > Subject: Proposal to add hive regression tests to check-PR
> > > > > > > tests
> > > > > > >
> > > > > > > If you have subscribed to Trafodion Daily Build, you would
> > > > > > > have noticed that the daily build has been failing for
> > > > > > > some days.
> > > > > > > Most often, it is due the failure in hive regression tests
> > > > > > > run as part of the daily build.
> > > > > > > Lately, there has been some conscious effort made
> > > > > > > successfully to ensure that the hive regression tests can
> > > > > > > be run reliably.
> > > > > > > To maintain the Trafodion daily build in that state, I
am
> > > > > > > proposing to include hive regressions to check-PR tests.
> > > > > > > It shouldn’t add the overall time taken to regressions
> > > > > > > tests because tests are run in parallel on different VMs,
> > > > > > > though it would consume more resources.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -          Selva
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards, --Qifan
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards, --Qifan
> > >
> >
> >
> >
> > --
> > Regards, --Qifan
> >
>
>
>
> --
> Regards, --Qifan

Mime
View raw message