beehive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eddie O'Neil" <ekon...@bea.com>
Subject Running NetUI Tests -- was Re: build fine, tests passed, coreWeb gives "internal error"
Date Wed, 28 Jul 2004 17:10:08 GMT

  Seems like a good time to describe how the NetUI test infrastructure 
works, specifically "coreWeb" webapp tests.

  Note, one of the things that we're waiting for before doing more 
documentation of the tests, infrastructure, and code base is for the 
wiki to come on-line.  Once that's in place, it's easier to distribute 
information in a forum that's easier to digest and reference.

  The text below talks about how NetUI tests are structured, how to 
diagnose problems, and how to start / stop / build the coreWeb test webapp.

  Hope this helps.

Running NetUI Tests
===============

The NetUI tests consist of two parts -- standalone JUnit tests and Test 
Recorder (TR) tests.  Both types of test are based on JUnit.  The 
standalone JUnit tests are just that; TestCase subclasses that test the 
NetUI APIs outside of a running server.  The Test Recorder tests are 
tests that run against a running Tomcat instance in Beehive.

Both the standalone JUnit and Test Recorder tests have two broad 
categorizations -- DRTs and BVTs. 

The DRT (Developer Regression Test) suite *must* pass at 100% before any 
checkins can be submitted into the $BEEHIVE_HOME/netui tree. 

The BVT (Build Verification Tests) suite consists of all of tests to 
run.  Most of us tend to run the BVTs before we checkin, though they 
don't always pass at 100%; for example, today we've got 27 TR tests that 
fail out of nearly 400.  Generally, additional BVTs shouldn't break on 
checkins either.  <g>

Currently, all test suites produce JUnit test results in the usual 
format using the <junitreport> Ant task.  These results are produced in 
a subdirectory of $BEEHIVE_HOME/netui/build.  The specific locations for 
the JUnit and TR tests are below.

NetUI Server Tests
=============
In addition to the API-level JUnit tests, NetUI uses a piece of 
home-grown software called the Test Recorder (TR) to run tests against a 
live server -- Tomcat is the server in the context of Beehive. 

The DRT suite currently runs from this directory

    $BEEHIVE_HOME/netui/test/webapps/drt/

against a webapp in a subdirectory of the above called "coreWeb" using 
the Test Recorder configuration / tests stored subdirectories 
"testRecorder/config" and "testRecorder/tests" respectively.  There is 
also a build.xml file in the above directory that controls the Tomcat 
server, coreWeb build, and running test suites.  This directory is 
assumed to be the working directory for the rest of the Ant commands in 
this section.

In order to run a set of TR tests, the following steps occur:

- build the coreWeb webapp
- start Tomcat
- deploy the coreWeb webapp
- run a test suite (drt, bvt, data binding, etc), which produces an HTML 
test result report
- stop Tomcat

Currently in Beehive, the instance of Tomcat that is used lives in:

  $BEEHIVE_HOME/installed/jakarta-tomcat-5.0.25/...

When the DRTs run as part of an "ant clean deploy drt" from either 
$BEEHIVE_HOME/ or $BEEHIVE_HOME/netui, this process is done 
automatically meaning that the build takes care of building the webapp, 
starting / stopping Tomcat, and running the tests.  The result of a DRT 
run can be found here:

    
$BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html

In addition to an automatic DRT run, the same steps can be run manually 
from the $BEEHIVE_HOME/netui/test/webapps/drt directory.  This is very 
useful when developing tests or just learning about NetUI as a normal 
web browser can be used to walk the tests manually.  To start the Tomcat 
server, you'll need two shells both with a Beehive development 
environment (see $BEEHIVE_HOME/BUILDING.txt) in the directory 
$BEEHIVE_HOME/netui/test/webapps/drt.

In the first shell, start Tomcat:

    ant start

In the second shell, build the coreWeb web application:

    ant build

Now, the "coreWeb" web application is ready to be installed on the 
running Tomcat server.  The steps needed to deploy to Tomcat can vary 
depending on the state of Tomcat.  There are three relevant targets:

deploy -- deploys the webapp for the first time.  This will fail if the 
webapp is already running.
undeploy -- undeploys a running webapp from Tomcat.  This will fail if 
the webapp is not running or did not start successfully
redeploy -- redeploy a successfully running webapp.  This will fail if 
the webapp did not start successfully

These targets can be run in combination to make sure that a webapp 
deploys correctly to Tomcat; for example, if the "coreWeb" webapp is 
already deployed, it needs to be redeployed or undeployed and then 
deployed.  Feedback for each step is available on the console of shell 
two.  Also, it is possible for a webapp to fail deployment, so just be 
sure to read the console output messages to make sure that deployment 
was in fact successful.  To deploy "coreWeb" initially, run in shell two:

    ant deploy

To redeploy this webapp, run "ant redeploy" or "ant undeploy deploy".  
The latter is usually a safe command as it will remove the webapp if it 
doesn't exist and then deploy it. 

Note, it's acceptable for some of these targets to fail -- if "coreWeb" 
isn't running on Tomcat and "ant undeploy" is run, it will fail like this:
:::::
Buildfile: build.xml
undeploy:
undeploy:
[tomcatundeploy] FAIL - No context exists for path /coreWeb
BUILD FAILED
C:\dev\apache\beehive\netui\test\webapps\drt\build.xml:147: Following 
error occured while executing this line
C:\dev\apache\beehive\test\ant\runTomcat.xml:31: FAIL - No context 
exists for path /coreWeb
:::::

To deploy it, just run "ant deploy" which succeeds with output like this:
:::::
Buildfile: build.xml
deploy:
deploy:
     [echo] deploy webapp from 
file://c:\dev\apache\beehive/netui/test/webapps/drt/coreWeb with context 
path coreWeb
[tomcatdeploy] OK - Deployed application at context path /coreWeb
BUILD SUCCESSFUL
Total time: 13 seconds
:::::

Once the "coreWeb" webapp is deployed successfully, it is accessible 
using the URL

    http://localhost:8080/coreWeb

which should bring up a test suite index page with the Beehive logo and 
a list of tests and test categories.  When running a particular test, 
the yellow bar at the bottom of each page is the Test Recorder.

To run the DRT test suite against a running server, run this command in 
shell two:

    ant drt.running

which puts the test results here:

    
$BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html

To run the BVT test suite from the same directory, run this command in 
shell two:

    ant bvt.running

which puts the test results here:

    
$BEEHIVE_HOME/netui/build/testRecorder/junit/bvt/html/junit-noframes.html

For more information, take a look through the 
$BEEHIVE_HOME/netui/test/webapps/drt/build.xml file and look at the 
targets and how they call into the webapp build and start / stop Tomcat 
targets in build files that live in:

    $BEEHIVE_HOME/test/ant

To stop Tomcat, run "ant stop" in shell two.

When diagnosing a test failure or problem running the webapp, there are 
several places to look to see if errors have occurred:

- the Tomcat consoleb
- the shell two console for errors reported when test recorder tests fail
- the Tomcat log files, which are stored by date in 
$BEEHIVE_HOME/external/jakarta-tomcat-5.0.25/logs
- in addition, NetUI can log error messages into the Tomcat console, and 
soon into a netui.log file

When reporting errors into the beehive-dev mailing list, having any 
failures / errors reported in the above helps when diagnosing a problem. 

Also, because we have error / failure test cases, it is often acceptable 
for NetUI tests to throw exceptions or errors in the course of regular 
execution.  While seeing a stack trace can be alarming, lots of tests 
should do this.  <g>  So, just be sure to check the test result output 
as the source of record for the success of a particular test or suite.

Overview of the Test Recorder
=====================
The Test Recorder tests comprise the bulk of the NetUI tests suite.  The 
TR is a piece of software that NetUI develops which consists of client 
and server pieces.  It simulates the interaction of a web browser with a 
running server.  The TR runs in two modes -- "record" and "playback". 

Developers use the TR in "record" mode to record a set of user 
interactions through a normal web browser (like FireFox) against a 
running, Test Recorder enabled web application.  The result is a 
"recorded" test which is stored in an XML file; examples of these can be 
found in $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/tests. 

When the DRT or BVT suites run, the client half of the TR uses these 
"record" XML files to "playback" a set of user interactions with the 
server.  This is done by reading request parameters out of the "record" 
file and building requests to send to the server; then, the response 
stream is captured which results in a "playback" file.  The "record" and 
"playback" results are compared; if there are no differences between the 
two, the tests are considered a pass.  Otherwise, the test is a 
failure.  Currently, this diff is done line-by-line and in the future 
could be done as an XML or HTML diff.

The TR is a very strict and accurate way for changes in the NetUI 
runtime to be verified.  Because the response stream of a rendered JSP 
is differenced against an expected result, even simple changes can break 
tests, which can be good because it helps NetUI developers understand 
the impact of changes to the code base.  Thus, we these tests help to 
maintain NetUI stability.

The TR is configured on a per-webapp basis using files like those used 
to configure the DRT's coreWeb webapp:

    $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/config

The TR uses its own Ant file to run a test suite against a server:

    $BEEHIVE_HOME/netui/test/ant/testRecorder.xml

Certainly, more documentation will follow on the TR and how to record / 
update test results.

JUnit API Tests
===========
The source for these tests lives in 
$BEEHIVE_HOME/netui/test/src/junitTests/...  The TestCases which run as 
part of the DRTs are specified in 
$BEEHIVE_HOME/netui/test/ant/junitDrt.properties.  The JUnit DRTs can be 
run like this:

    cd $BEEHIVE_HOME/netui/test/ant
    ant drt

This will build all of the test code and run the JUnit tests specified 
in the above file.  We use a simple <netui-junit> Ant task that takes 
the .properties file above and filters all of the classes in the 
junitTests.jar file to produce the set of TestCases to run as DRTs.  The 
Ant to drive the JUnit DRTs lives in 
$BEEHIVE_HOME/netui/test/ant/junit.xml.  The JUnit test results can be 
found here:

    $BEEHIVE_HOME/netui/build/drt.testResults/html/junit-noframes.html

Mime
View raw message