qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rupert Smith" <rupertlssm...@googlemail.com>
Subject Re: Draft Interop Testing Spec - Please Read
Date Wed, 07 Mar 2007 16:05:15 GMT
 - group related tests together.
Coordinator does this (in the sense that it produces a test suite of
all available combinations of clients and test cases).

 - assert expected conditions are met.
Coordinator does this.

 - handle unexpected failures.
If clients fail to produce reports, coordinator will time them out,
they get a fail. I've tried to arrange things deliberately so that
unexpected failures result in timeouts, resulting in a robust
framework that won't lock up and wait forever. Admitedly, you will
lose the failure stack trace when this happens, but can always examine
clients local logs to figure out what happened.

 - collect and report results.
Reports send to coordinator, it compares sender and receiver reports,
decides if the test passed/failed, logs out the result.

On 3/7/07, Rupert Smith <rupertlssmith@googlemail.com> wrote:
> The coordinator will only be implemented in one language, once only.
> There will be many test clients in all languages, just test clients,
> with no coordinator part.
>
> I really don't see what *Unit adds to the test client end. Please note
> Alan, we've switched from my original idea of having each test client
> output its own test results to *Unit XML format, to having the
> coordinator do it, as per Gordons centralized approach. The result of
> this, is that the test clients are really quite simple, reactive
> agents.
>
> Rupert
>
> On 3/7/07, Alan Conway <aconway@redhat.com> wrote:
> > Rupert Smith wrote:
> > > I don't think the test clients will need to use
> > > JUnit/CppUnit/WhateverUnit. Mostly they will just be fairly simple
> > > scripts that react to instructions sent by the coordinator. Although,
> > > having said that, you could use *Unit if you like.
> > The test clients will need a way to:
> >  - group related tests together.
> >  - assert expected conditions are met.
> >  - handle unexpected failures.
> >  - collect and report results.
> >  - provide a natural test environment for each language
> >
> > In other words the tests will need everything that the *Unit frameworks
> > provide. It may not seem like a lot, but would you consider writing
> > JUnit from scratch to test Java code? It won't be any easier to build a
> > distributed test framework from scratch - in fact it'll be harder and
> > the result will be less usable. Trust me, I've been there.
> >
> > > I'm just not sure
> > > it adds anything, because these clients won't do anything when run on
> > > their own. They need the coordinator to push them along.
> > Right but how do you express that on the client side? I would say you
> > express it as a "QpidTestRunner" test runner for *Unit tests. The
> > QpidTestRunner is responsible for talking to the co-ordinator. The
> > co-ordinator tells the test runner "Run BasicTests.getGet" the test
> > runner sets up the test environment executes the corresponding JUnit
> > test, collects the results and sends them back to the co-ordinator. The
> > "test client" *is* a testrunner. That means we can flexibly load as many
> > or as few tests as we want into the runner, run tests selectively and do
> > all the other good stuff that you can do with *Unit tests. It also means
> > the tests themselves are straight *Unit, nothing new to learn for test
> > writers.
> >
> > It may not seem like *Unit is doing a lot in the scenario above, but
> > believe me, speaking as one who has participated in reinventing it
> > several times *it is not worth reinventing*.
> >
> > > The coordinator itself is a different story. I've sketched an outline
> > > for it in Java built on top of JUnit. Trying to use JUnits existing
> > > TestSuite mechanism, and ability to locate tests in *Test classes. A
> > > little bit of adaption perhaps needed to dynamically name the tests,
> > > or to run the same test case many times accross differnt combinations
> > > of clients. A root test case called something like InviteTestCase to
> > > provide common code to do the invite and gather reports, and provide
> > > convenience methods for common tasks etc. Here JUnit adds a lot as a
> > > convenient framework to base this on.
> > All the more reason to use it on both sides: give people a common
> > framwork instead of making them jump from JUnit to some ad-hoc
> > framework. E.g. with JUnit on both ends the client part of a test can
> > sit right beside the co-ordinator part and use some simple naming
> > conventions to make it obvious what goes with what.
> > > I'll aim to get this skeleton code, and skeleton code for a java test
> > > client checked in before fleshing it out. That way, other test clients
> > > can copy the same structure.
> > Looking forward to it  :) I'll try to put some code where my mouth is
> > when the skeleton is in place.
> >
> >
> > Cheers,
> > Alan.
> >
>

Mime
View raw message