qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajith Attapattu <rajit...@gmail.com>
Subject Re: Ideas to rationalise the Java test profiles.
Date Tue, 05 Jul 2011 13:59:14 GMT
Hi Robbie,

I couldn't agree more with your comments !
We definitely need a set of system tests that covers the common
messaging use cases that developers can run before checking in
changes.
We could then rely on a CI instance to cover the various test profiles.

Now that we are on the subject let me take the opportunity to share
some thoughts that I've been meaning to put forth for some time.

1. Coverage
    We currently have close to 1200+  test cases and my impression is
that we still have quite a few gaps. All though it's painful maybe we
should try to get an inventory of our existing test cases. I started
this a few times and gave up :(. This will be a lot less painful if we
share this task among several folks :)

2. Manageability/Maintainability of test suites
   As Robbie pointed out we have a lot of low utility tests. these
tests just bloat our test suite but add very little value.
   We need to get out of the habit of adding a new test case for every
commit. I personally think we have too many tests.
   As Robbie mentioned we need to look at building a suite of
meaningful tests that covers real world messaging use cases.

3. System tests
    We need to make a distinction between client tests and broker
tests and a developer should have the flexibility of running either or
both.

     Broker tests
     -----------------
     I think going forward we should have a common test suite for the
brokers (both C++ and Java) rather than duplicating and spreading
ourselves think

     JMS tests
     --------------
   Quite a few of our tests are written using implementation specific
AMQ* classes. This is a major hurdle in refactoring the client as we
automatically loose a fair percentage of the tests when we write a new
implementation. If we write highly configurable pure JMS based tests
then we can easily reuse them irrespective of the underlying
implementation strategy.

    Qpid API tests
    ------------------
   If and when we come up with the Qpid API equivalent in Java we need
to have tests there as well. But probably we could benefit from a
common test suite across all languages for this. This is easier said
than done, but if we do get it right, it will be a huge win.

Regards,

Rajith.

On Mon, Jul 4, 2011 at 11:45 AM, Robbie Gemmell
<robbie.gemmell@gmail.com> wrote:
> Hi Rajith,
>
> The changes we are proposing to the Java test profiles only relate to
> the Java broker used during test runs, nothing else will change, and
> as always changes should be tested against the C++ broker.
>
> Going forward, the 30-40min test profiles will pretty much be a thing
> of the past as far as humans are concerned, and we will leave those up
> to the Jenkins box 99% of the time, all the profiles we'd expect
> people to run will complete significantly faster.
>
> The obvious way to improve up our testing situation is to stop using
> system tests where unit tests are more appropriate, i.e the vast
> majority of cases. A lot of our current system tests are of low
> utility and would be better implemented as unit tests. We should
> probably also look towards implementing a suite of system tests that
> exercise common messaging use cases, and going forward attempt to
> transition as much of the rest into well thought through and
> specifically targeted unit tests.
>
> Robbie
>
> On 4 July 2011 15:39, Rajith Attapattu <rajith77@gmail.com> wrote:
>> Just to add to it, any changes to the Java client should also be
>> tested with the cpp test profile.
>> Any changes to the client failover mechanism should also be tested
>> with the cpp.cluster profile.
>>
>> I know it's a pain to run all these profiles, especially considering
>> each run takes about 30-40 mins.
>> But atm there is no better alternative. I think long term we need to
>> evaluate our testing strategy to see if we can come up with something
>> more efficient.
>>
>> Regards,
>>
>> Rajith
>>
>> On Mon, Jul 4, 2011 at 7:11 AM, Robert Godfrey <rob.j.godfrey@gmail.com> wrote:
>>> On 4 July 2011 12:44, Keith Wall <keith.wall@gmail.com> wrote:
>>>
>>>> Robbie and I are presently working on QPID-2811 and QPID-2815.  These Jiras
>>>> will rationalise the transport mechanism in Qpid and to allow client/broker
>>>> to be tested together within the same JVM regardless of transport.
>>>>
>>>> It seems to me that now is a good time to review the test-profiles.  I see
>>>> the following problems:
>>>>
>>>> 1) 'default' profile being 0-8 makes little sense as most of the project's
>>>> development focus is on 0-10, and with the removal of the old .NET client,
>>>> we have no client that defaults to 0-8.
>>>> 2) The time taken to run some test profiles is excessive and this is a
>>>> disincentive to developer good practice.  For instance, running java-0.10
>>>> on
>>>> my machine takes 30mins.  This is harming productivity.
>>>>
>>>> Current profiles:
>>>>
>>>> default.testprofile  - tests 0-8 using in-VM, broker/client share JVM,
>>>> Memory Message Store
>>>> java.testprofile - tests 0-9-1, using tcp, separate JVMs, Memory Message
>>>> Store
>>>> java-derby.testprofile - tests 0-9-1, using tcp, separate JVMs, Derby
>>>> Message Store
>>>>
>>>> java.0.10.testprofile   - tests 0-10, using tcp, separate JVMs,, Memory
>>>> Message Store
>>>> java-derby.0.10.testprofile -- tests 0-10, using tcp, separate JVMs, Derby
>>>> Message Store
>>>>
>>>> My proposal:
>>>>
>>>> Four profiles intended for regular developer use.  These profiles will
>>>> exploit the feature for client/broker to share the same VM.  This saves
the
>>>> overhead of each testcase spawning a new broker as a separate process, and
>>>> brings the length of time to run the profile down considerably, assisting
>>>> developer productivity.
>>>>
>>>> (mms=memory message store, dby = Apache Derby)
>>>>
>>>> java-mms.0-10
>>>> java-dby.0-10
>>>>
>>>> java-mms.0-9-1
>>>> java-dby.0-9-1
>>>>
>>>> Running client/broker in separate JVMs remains valuable, but is probably
>>>> not
>>>> something most developers will want to do most of the time (because of the
>>>> time overhead).    The intention would be we'd schedule these to run on
CI
>>>> (using Jenkin's matrix feature) and let Jenkins do the heavy lifting.
>>>>
>>>> java-mms-spawn.0-10
>>>> java-dby-spawn.0-10
>>>>
>>>> java-mms-spawn.0-9-1
>>>> java-dby-spawn.0-9-1
>>>>
>>>> The idea of the default testprofile would disappear and it would become an
>>>> error to run -Dprofile=default.   If the developer omits -Dprofile from
the
>>>> Ant command line, the system will run java-mms.0-10, which I think will be
>>>> useful fallback to most developers.
>>>>
>>>> I think we also need a "Java Test Howto" page on the website, explaining
>>>> the
>>>> purposes behind the profiles, the differences between system and non-system
>>>> tests and giving advice on appropriate use of both.
>>>>
>>>> Any thoughts?
>>>>
>>>> Keith.
>>>>
>>>
>>>
>>> All sounds exceedingly sensible to me.
>>>
>>> -- Rob
>>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


Mime
View raw message