qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Draft Interop Testing Spec - Please Read
Date Thu, 22 Feb 2007 20:19:12 GMT
On Thu, 2007-02-22 at 14:48 +0000, Rupert Smith wrote:
> I would like to propose an interop testing spec that defines
> exactly how test clients should behave, so that we can write a suite
> of tests that can talk to each other succesfully. 
Good work! We really need this. Comments inline. I need to think
more about the fine detail but I like the overall approach.

>  =====================================
> 
>  Qpid Interop Testing Spec. Draft 1.
> 
>  Draft 1. Rupert Smith. Document started.
> 
> Introduction:
> 
>  The requirements in this specification use a common format, an
> example of which is given below:
> 
>  RE-1. Sample Requirement. A brief descritpion of the requirement.
> 
>  The requirements are numbered sequentially from 1.
> 
> Purpose:
> 
>  * Test sending from and receiving by each of the clients in Qpid over
> both of the broker implementations.
> 
>  * Make tests robust enough to run as part of an automated build. The
> scripts should pass or fail, not hang, wait forever, run out of memory
> or otherwise cause an automated build process to be flaky.
> 
>  * Be capable of running the full test suite on several machines in a
> hands free way. In particular C++ tests need to run on unix and .Net
> on windows.
> 
>  * Run just a few tests to begin with. More can be added later, I'm
> interested in getting the test framework established as quick as
> possible. Just
>    minimal p2p and pub/sub tests to begin with.
> 
> Constraints:
> 
>  IOP-1. Operating System: The test client scripts must run on Unix and
> Windows. If a test client implementation is only available on one of
> these platforms it only needs to run on its supported platform.
> 
>  IOP-2. Scripting Language: Each test client must be startable from a
> Unix shell script. Tests run on Windows will use Cygwin to run these
> scripts. There is no need to support Windows .bat scripts.
> 
> Functional Requirements:
> 
>  Introduction.
> 
>  These requirements describe the behaviour of test clients for interop
> testing between the different client implementations in Qpid. Each
> client is expected to be a program that is capable of sending test
> messages to other clients and receiving and responding to test
> messages received from other test clients. The clients are not to be
> run as seperate programs for the sending and receiving parts for the
> sake of convenience in being able to run the clients as part of an
> automated build. The clients will listen for control messages sent by
> a master client on a topic, to tell them when to begin their tests and
> when to shutdown. The clients will send control messages to each other
> individually on a direct exchange in order to communicate about
> individual test cases being run.
> 
>  Common Requirements.
> 
>  IOP-3. Directory Structure. All scripts to start and stop brokers and
> run test clients will be placed in a directory structure underneath a
> top-level directory called 'interop' that sits at the top level of the
> Qpid project.
> 
>  IOP-4. Test Output Format. Output in junit xml format (because a lot
> of automated build software understands this format). There doesn't
> seem to be a schema or DTD for this format but it is simple enough.
> See Appendix B for an example. Each sending test client will output a
> test suite report for each receiving test client that it runs the test
> cases against.

We can write a CppUnit formatter in junit style, but do we need to agree
on qualified test class names to appear in the report or will
unqualified class name + function name suffice? 

>  IOP-5. Terminate On Timeout. Each client will keep a timeout count.
> Every time it gets a message it will reset this count. If it does not
> hear from the broker at all for 60 seconds then it will assume that
> the broker has died and will terminate with test failures.

Simple, but includes client processing time in the timeout. More
accurate would be to have timeout in effect only when the client has
reason to expect something from the broker. 

>  IOP-6. Default Virtual Host. All test clients will use the virtual
> host '/test' for all tests.
> 
>  IOP-7. Default Control Topic. All test clients will use a control
> topic with the routing key 'iop.control' on the default virtual host
> on the default topic exchange. This control topic is used for sending
> the start messages (IOP-23), declare available messages (IOP-24), test
> complete messages (IOP-25) and terminates messages (IOP-26) only.
> 
>  Use Case 1. Starting a Broker.
> 
>   Run the broker start script.
>   The script starts a broker running and tries to connect to it (or
> otherwise ping it) until it is verified to be running.
>   Once the broker is verified to be running the script terminates with
> no error code.
> 
>   Failure path: The broker fails to start or does not appear to be
> running after a timeout has passed. The script fails with an error
> code.
> 
>  IOP-8. Broker Start Script. The java and c++ brokers will define
> scripts that can start the broker running on the local machine, and
> these scripts will be located at interop/java/broker/start and
> interop/c++/broker/start. The Java and C++ build processes will
> generate these scripts (or copy pre-defined ones to the output
> location) as part of their build processes.
> 
IOP-8a: the *only* prerequisite for the scripts to run is that they
are located in a checkout where cpp/make java/mvn have been run. In
particular they must NOT assume any environment variables are set.

Scripts can calculate the script directory with $0 and deduce any other
files/directories they need to set up environment for their
brokers/clients.

(At some point we will want to run the tests on an install rather than
in a checkout but lets walk before we run.)

>  IOP-9. Broker Start Timeout. If a broker fails to start within 60
> seconds its start script will timeout. Script will terminate with
> error code 1.
> 
>  IOP-10. Broker Start Succesfull. When the broker starts succesfully
> the script will terminate with error code 0.
> 
>  Use Case 2. Stopping a Broker.
> 
>   Run the broker stop script.
>   The script terminates the broker that was started with the start
> script if it is still running.
> 
>   Failure path: The broker won't terminate. The script fails with an error code.
> 
>  IOP-11: Broker Stop Script. The java and c++ brokers will define
> scripts that can stop the broker running on the local machine, and
> these scripts will be located at interop/java/broker/stop and
> interop/c++/broker/stop. The Java and C++ build processes will
> generate these scripts (or copy pre-defined ones to the output
> location) as part of their build processes.

For consistency and to avoid possible headaches with special characters
I'd  suggest cpp rather than c++ for file/directory names.

> 
>  IOP-12. Broker Stop Timeout. If a broker fails to terminate within 60
> seconds its stop script will timeout. Script will terminate with error
> code 1.
> 
>  IOP-13. Broker Stop Succesfull. When the broker stops succesfully the
> script will terminate with error code 0.
> 
>  Use Case 3. Starting a Test Client.
> 
>   Run the client start script. The caller will pass in address of the
> broker to connect to and the number of messages to send per test.
>   The script starts a client running.
>   The client starts running but waits for further instruction before
> running its tests.
>   The start script will terminate but leave the client running as a
> forked process.
> 
>   Failure path: The client will not start, or fails to connect to the
> specified broker. The script will termiante with error code 1.
> 
>  IOP-14. Client Start Scripts. For each client implementation,
> <client>, there will be a start script located at
> interop/<client>/client/start. The build processes for each client
> will generate these scripts and output them to this location as part
> of their build process.
> 
>  IOP-15. Client Start Timeout. If the client fails to start and
> connect to the specified broker within 60 seconds the script will
> terminate with error code 1.
> 
>  IOP-16. Client Start Succesfull. When the client starts successfully
> its script will terminate with error code 0.
> 
>  IOP-17. Client Start Broker Url. The -b <broker_url> option will be
> used to call the start script to connect to the broker specified in
> the url.
> 
>  IOP-18. Client Start Messages Per Test. The -m <num_messages> option
> will be used to tell the client how many messages to send per test.
> 
>  IOP-19. Client Number of Receivers. For topic testing each client
> will simulate the behaviour of many clients listening to the same
> topic. The number of recievers per test client for topic tests will be
> specified by the -r <num_receivers> command line option.
> 
>  Use Case 5. Overall Test Procedure.
> 
>   Start a broker running using its start script as described by Use Case 1.
>   Call the start all clients script on each of the machines where
> there are clients that are to be tested. The caller will pass in
> address of the broker to connect to and the number of messages to send
> per test.

NB: timing issues - e.g. one that plagues the current C++ topic test. If
the publisher finishes publishing before all of the subscribers are
listening (few messages, many subscribers) you get hanging subscribers
that missed the "TERMINATE" message and a hanging publisher waiting for
them. Even if clients all timeout after 60 seconds this still makes for
a very unreliable test.

I suggest such issues should be dealt with by the clients themselves
using queues to synchronize. E.g. a fix for topictest would be a "ready"
queue, the publisher waits till he gets a "ready" message from all
subscribers before starting to publish. It's worth highlighting this
point in the test design guidelines.

>   The start all script will scan for all start scripts located under
> interop/<client>/client/start and call each of them with the broker
> and number of
>   messages arguments. This performs Use Case 3 for each client.
>   Call the master test script. The caller will pass in the address of
> the broker to connect to.
>   The master test script will send out a begin message on the control topic.
>   Each of the clients will receive the begin message.
>   Each client will post a message to the control topic declaring its
> availability to test and locations on which it is expecting test
> messages to be sent. This message will contain the routing key on the
> default direct exchange to which the client recieves control signals,
> the routing key on the default direct exchange to which it expects
> test p2p messages to be sent, and the routing key on the default topic
> exchange to which it expects test pub/sub messages to be sent.
>   Each client will hear on the control topic what other clients are
> available and the addresses on which they expect to be sent messages.
>   Each client will run test cases 1 & 2 against each other client.
>   Each client will post a message to the control topic declaring that
> it has completed its tests.
>   The master test script will listen for each client to send its
> completion message.
>   Once all completion messages have been received the master test
> script will send out a terminate message on the control topic.
>   Each client will receive the terminate message and will shutdown.
>   Terminate the broker using its stop script.
> 
>   IOP-20. Start All Script. There will be a start all clients script,
> located at interop/startall. The startall script finds all client
> starts scripts under interop/<client>/client/start and calls them.
> 
>   IOP-21. Start All Script Options Forwarding. The start all script
> will take the same command line options as the client start scripts
> and will pass these command line options on to them.
> 
>   IOP-22. Master Test Script. There will be a master test script that
> kicks off the testing process once all clients have been started. It
> is to be located at interop/testall. It will start a master test
> client that sends the start and terminate messages.
> 
>   IOP-23. Start Message. The master test script will send a start
> message to begin the test procedure. All test clients will listen for
> this message on the control topic. The start message will be
> identified by the header field, "CONTROL_TYPE", having the value,
> "START".
> 
>   IOP-24. Declare Available Message. Every test client that receives
> the start message will respond by declaring its availability to run
> interop tests with each other client. The client will send a declare
> available message to the control topic. The declare available message
> will be identified by the header field, "CONTROL_TYPE", having the
> value, "DECLARE". The client will also declare the direct exchange and
> topic exchange routing keys on which it expects to be sent test
> messages and per client control messages. The client will also declare
> a unique name by which it can be identified (see IOP-27). The declare
> available message will contain the following header fields with this
> information:
> 
>           "CLIENT_NAME", "<client_name>" (see IOP-27 for rules about
> the client name).
>           "CLIENT_DIRECT_CONTROL_KEY", "<client_name>.control" (see IOP-28)
>           "DIRECT_KEY", "<client_name>.direct" (see IOP-29)
>           "TOPIC_KEY", "<client_name>.topic" (see IOP-30)

Solves the timing problem I mentioned!

> 
>   IOP-25. Test Complete Message. Once a test client has run all of its
> tests it will declare itself ready for termination. It will not
> terminate immediately because it may still need to respond to test
> messagesd from other clients. It will send a test complete message to
> the control topic. This message will be identified by the header
> field, "CONTROL_TYPE", having the value, "COMPLETE". The client will
> also post its name in the "CLIENT_NAME" header field.
> 
>   IOP-26. Terminate Message. The master test script will listen for
> all test clients to complete their tests. Once it has received a test
> complete message from each client that declared itself it will send a
> terminate message to the control topic. The terminate message will be
> identified by the header field, "CONTROL_TOPIC", having the value,
> "TERMINATE". Upon receipt of this message the test clients will
> terminate.
> 
>   IOP-27. Client Name. Each test client will provide a unique name for
> itself that reflects its implementation language and distinguishes it
> from the other clients. Clients should append a timestamp or UUID onto
> this name to cater for the case where the same client is used multiple
> times in an interop test. For example, the same client might be run on
> two different operating systems, in order to check that it works
> correctly on both.
> 
>   IOP-28. Client Control Direct Key. Each test client will listen for
> test control messages directed specifically to it on the default
> direct exchange. The routing key for these messages will consist of
> the client name (see IOP-27) followed by ".control".
> 
>   IOP-29. Client Default P2P Test Direct Key. Each test client will
> listen for test messages on the default direct exchange. The routing
> key for these messages will consist of the client name (see IOP-27)
> followed by ".direct".
> 
>   IOP-30. Client Default Pub/Sub Test Direct Key. Each test client
> will listen for test messages on the default topic exchange. The
> routing key for these messages will consist of the client name (see
> IOP-27) followed by ".topic".
> 
>  Common Requirements for Test Cases.
> 
>   IOP-31. Client Message Receive. Whenever a test client recieves a
> message from another test client it will increment the total count of
> messages received from that client since the last reset. The message
> will contain the name of the sending client in the header field
> "CLIENT_NAME", and the count will be held against that name.
> 
>   IOP-32. Client Reset Message. Whenever a test client wishes to begin
> a new test procedure against another test client it will request that
> its message count be reset. It will send a reset message to the test
> clients individual control queue. The control message will be
> identified by the header field, "CONTROL_TYPE", having the value,
> "RESET", and will contain the name of the sending client in the header
> field "CLIENT_NAME".
> 
>   IOP-33. Client Status Request Message. When a test client has
> completed sending test messages it may request the count of actual
> messages receieved from the test client to which it sent the messages.
> The status request message will be send to the test clients individual
> control queue. This message will be identified by the header field,
> "CONTROL_TYPE", having the value, "STATUS_REQUEST", and will contain
> the name of the sending client in the header field "CLIENT_NAME". The
> message will have its reply-to field set to the routing key of the
> individual control queue of the sending client.
> 
>   IOP-34. Client Status Message. Upon receipt of a status request
> message, a test client will send a status message to the individual
> control queue of the status requesting client, the routing key of
> which can be found in the reply-to of the status request message. The
> status message will be identified by the header field "CONTROL_TYPE",
> having the value, "STATUS". The header field, "MESSAGE_COUNT" will
> contain the count of messages received from the requesting client
> since the last reset as an unsigned 32-bit integer. The client sending
> the status message will also place its identifying name in the header
> field "CLIENT_NAME".
> 
>  Test Case 1. Basic P2P Test.
> 
>   The sending client sends a reset message to the receiving clients
> control queue.
>   The receiving client will reset its message count to zero.
>   The sending client will send the required number of test messages to
> the receiving clients test direct routing key.
>   The sending client will send a status request message to the
> receiving clients control queue.
>   The receiving client will send a status message to the control queue
> of the sending client (given as the reply to location of the status
>   message for convenience). This status message will contain the
> number of messages received since the last reset.
>   The sending client will compare the messages received to the
> messages sent and pass or fail the test accordingly.

I'm not clear on how the sending/receiving clients identify each other 
in the first place. Hard coded topic names?

>  Test Case 2. Basic Pub/Sub Test.
> 
>   The sending client sends a reset message to the receiving clients
> control queue.
>   The receiving client will reset its message count to zero.
>   The sending client will send the required number of test messages to
> the receiving clients test topic routing key.
>   The sending client will send a status request message to the
> receiving clients control queue.
>   The receiving client will send a status message to the control queue
> of the sending client (given as the reply to location of the status
>   message for convenience). This status message will contain the
> number of messages received since the last reset. This number will be
> the number
>   of messages sent multiplied by the number of topic receivers being
> simulated by the receiving client.
>   The sending client will compare the messages received to the
> messages sent and pass or fail the test accordingly.
> 
> Waiting Room:
> 
>  Contains ideas for future additions to this spec.
> 
>  How I anticipate this being run as part of a fully automated build:
> Will try to get a free licence for Anthill Pro 3 as they offer free
> licences for open source projects. Viewtier Parabuild is another
> possibility. Anthill Pro runs a central build server that does all its
> work through build agents that can run on many boxes. It also lets you
> define build workflows. I imagine running a Unix agent to build the
> c++, java and python stuff, and a Windows agent for the .net stuff.
> Will define a workflow that starts a broker on the unix box, then
> starts all clients built on the unix and windows boxes in parallel,
> then runs the entire test procedure across all clients, then
> terminates the broker on the unix box. The agents send back the test
> results to the central server.
> 
>  Full testing of field tables. Make sure that every possible data type
> is tested and confirmed to encode and decode correctly between all
> client implementations.
> 
>  Testing more of the protocol. Add tests to more fully exercise the
> complete AMQP protocol.
> 
> Appendix A, General Notes:
> 
>  Brokers that need to be interop tested: C++ and Java
> 
>  Clients that need to be interop tested: C++, Java, Java 1.4
> retrotranslation, C++, .Net 2.0, .Net 1.1, (Mono?), Python, Ruby.
> 
> Appendix B, Example of XML Format for Test Ouput:
> 
>  I don't think there is a DTD or schema for this but the XML output
> from JUnit looks like the example below. This is a convenient choice
> for the output format from these test results even if the code does
> not actually use JUnit (or cppunit or nunit) iternally, because
> automated build servers generally understand and are able to produce
> test reports from it.
> 
>  Example:
> 
> <?xml version="1.0" encoding="UTF-8" ?>
> <testsuite errors="0" skipped="0" tests="18" time="0.02" failures="0"
> name="org.apache.qpid.framing.BasicContentHeaderPropertiesTest">
>   <properties>
>     <property value="Java(TM) 2 Runtime Environment, Standard Edition"
> name="java.runtime.name"/>
>     ... (there were lots of properties).
>   </properties>
>   <testcase time="0.02" name="testRejectedExecution"/>
>   ... (there were lots of test cases).
> </testsuite>


Mime
View raw message