qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gordon Sim <g...@redhat.com>
Subject Re: Draft Interop Testing Spec - Please Read
Date Fri, 23 Feb 2007 08:38:46 GMT
Tomas Restrepo wrote:
> Hi Rupert,
>>  I'm particuarly interested in getting some constructive feedback on
>> this. If you disagree with something, please also suggest an
>> alternative way of doing it, that you feel will be better, more
>> reliable, easier to implement, or whatever. Thanks.
> I think this is a fantastic idea and very needed. Some thoughts:
> 1- Would this be mostly aimed at testing client-client interop,
> client-broker interop or both? It seems to me much of the implementation
> needs you specify seemed to be aimed at client-client, but maybe I'm
> mistaken.

I think it is both. You raise a good point though, in that if the test 
infrastructure assumes a degree of interoperability to begin with, the 
tests it runs are only worthwhile if they test interoperability in a 
more advanced sense. That was the assumption I was making; allowing easy 
creation of tests that give us confidence that systems written using 
different pieces of the product will all work as expected rather than 
just verifying that clients can 'talk to' brokers in a basic way.

> 2- Personally, I'd favor an approach a bit more like Gordon's idea of it
> being more "centrally controlled" by the controller. Start a client-test
> process, launch the controller, and do everything from there. It would be
> simpler to create, run and maintain, I think.
> 3- I think the initial client to broker connection tests might want to be
> handled a bit differently (maybe a set of more automated, but regular,
> integration/unit tests for each client testing different success and failure
> connection conditions).

Yes, I agree. In c++ we have a really simple client test program that 
connects, sends itself a message then stops. Something similar for each 
language would give a simple test for the ability to connect, send and 
receive in basic form. Thats probably the first step before any test 
framework is run. Maybe we should also have a dummy test that doesn't do 
any real work but just allows the operation of the framework to be verified.

> My main reason for saying this is that I think it might be more awkward to
> cram the connection-level tests into the kind of structure proposed, and
> even more if we went with a more central architecture such as the one Gordon
> proposed.

Yes, I completely agree.

> More to the point, one of the basic ideas you propose is already using
> broker communication and queues to connect the test controller with the
> individual test clients, meaning already you assume a fairly high-level of
> interoperability being possible between clients and brokers (and even
> between clients seeing as how the controller would be written in a single
> language and each client in a different one). 
> This is also one of the main reasons I ask whether the tests will mostly
> target client-client scenarios or client-broker scenarios (as for most of
> the infrastructure it seems to assume the latter already works pretty well).
> Then again, maybe I'm just missing something :)

I think you are right. Any framework based on AMQP itself should 
minimise the functionality it relies on. There needs to be some test of 
basic connectivity separate from that to ensure that a given client can 
be relied on to work within the framework. The value of the framework is 
then dependent on what other useful aspects it can test. There is always 
a danger of over-engineering.

However assuming that we want to test more than just single client to 
broker interactions, there are some common aspects that you don't want 
to have to keep re-writing, there are co-ordination issues such as the 
ones Alan raised, verification and reporting of results etc. I also 
think that there is useful interop testing to be done around features 
not required by the framework e.g. acks, txns, more advanced routing 
patterns etc.

View raw message