beehive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Gozin <>
Subject Re: Running NetUI Tests -- was Re: build fine, tests passed, coreWeb gives "internal error"
Date Sun, 01 Aug 2004 11:08:19 GMT
Finally got back to try the beehive tests. Most of it went smooth except 
the following two tests.


After hitting /coreWeb I noticed both of them have a diff-link set. Here 
is what they both say.

Test Diffs: FormatTags

Uri( /coreWeb/tags/formatTags/Controller.jpf )
Test Number( 1 )
Line (62) does not match.
R: <span>1/17/02 1:30 PM</span>
P: <span>17/01/02 13:30</span>
Line (64) does not match.
R: <span>1/17/02 1:30 PM</span>
P: <span>17/01/02 13:30</span>

Test Diffs: ModuleConfigLocator

Uri( /coreWeb/miniTests/moduleConfigLocator/index.jsp )
Test Number( 1 )
Line (9) does not match.
R: <html lang="en">
P: <html lang="nl">

Both are locale issues where parts of the webapp are compared to a 
snapshot from a US-locale based webapp. Perhaps it should be possible to 
enforce a locale globally in /WEB-INF/netui-config.xml if at all 
possible. Or is it acceptable to set <netui:html useLocale="something"> 
on a per file basis just to get tests to pass? Same thing for date 
formatting as I think the webapp is using <netui:formatString> to do this.
Since they are tests compared to a snapshot I don't think it's 
unreasonable to enforce a certain locale as long as it can be set 
globally. It would also be nice if one could make their own snapshot (in 
a different locale) to test against.


Eddie O'Neil wrote:

>  Seems like a good time to describe how the NetUI test infrastructure 
> works, specifically "coreWeb" webapp tests.
>  Note, one of the things that we're waiting for before doing more 
> documentation of the tests, infrastructure, and code base is for the 
> wiki to come on-line.  Once that's in place, it's easier to distribute 
> information in a forum that's easier to digest and reference.
>  The text below talks about how NetUI tests are structured, how to 
> diagnose problems, and how to start / stop / build the coreWeb test 
> webapp.
>  Hope this helps.
> Running NetUI Tests
> ===============
> The NetUI tests consist of two parts -- standalone JUnit tests and 
> Test Recorder (TR) tests.  Both types of test are based on JUnit.  The 
> standalone JUnit tests are just that; TestCase subclasses that test 
> the NetUI APIs outside of a running server.  The Test Recorder tests 
> are tests that run against a running Tomcat instance in Beehive.
> Both the standalone JUnit and Test Recorder tests have two broad 
> categorizations -- DRTs and BVTs.
> The DRT (Developer Regression Test) suite *must* pass at 100% before 
> any checkins can be submitted into the $BEEHIVE_HOME/netui tree.
> The BVT (Build Verification Tests) suite consists of all of tests to 
> run.  Most of us tend to run the BVTs before we checkin, though they 
> don't always pass at 100%; for example, today we've got 27 TR tests 
> that fail out of nearly 400.  Generally, additional BVTs shouldn't 
> break on checkins either.  <g>
> Currently, all test suites produce JUnit test results in the usual 
> format using the <junitreport> Ant task.  These results are produced 
> in a subdirectory of $BEEHIVE_HOME/netui/build.  The specific 
> locations for the JUnit and TR tests are below.
> NetUI Server Tests
> =============
> In addition to the API-level JUnit tests, NetUI uses a piece of 
> home-grown software called the Test Recorder (TR) to run tests against 
> a live server -- Tomcat is the server in the context of Beehive.
> The DRT suite currently runs from this directory
>    $BEEHIVE_HOME/netui/test/webapps/drt/
> against a webapp in a subdirectory of the above called "coreWeb" using 
> the Test Recorder configuration / tests stored subdirectories 
> "testRecorder/config" and "testRecorder/tests" respectively.  There is 
> also a build.xml file in the above directory that controls the Tomcat 
> server, coreWeb build, and running test suites.  This directory is 
> assumed to be the working directory for the rest of the Ant commands 
> in this section.
> In order to run a set of TR tests, the following steps occur:
> - build the coreWeb webapp
> - start Tomcat
> - deploy the coreWeb webapp
> - run a test suite (drt, bvt, data binding, etc), which produces an 
> HTML test result report
> - stop Tomcat
> Currently in Beehive, the instance of Tomcat that is used lives in:
>  $BEEHIVE_HOME/installed/jakarta-tomcat-5.0.25/...
> When the DRTs run as part of an "ant clean deploy drt" from either 
> $BEEHIVE_HOME/ or $BEEHIVE_HOME/netui, this process is done 
> automatically meaning that the build takes care of building the 
> webapp, starting / stopping Tomcat, and running the tests.  The result 
> of a DRT run can be found here:
> $BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html
> In addition to an automatic DRT run, the same steps can be run 
> manually from the $BEEHIVE_HOME/netui/test/webapps/drt directory.  
> This is very useful when developing tests or just learning about NetUI 
> as a normal web browser can be used to walk the tests manually.  To 
> start the Tomcat server, you'll need two shells both with a Beehive 
> development environment (see $BEEHIVE_HOME/BUILDING.txt) in the 
> directory $BEEHIVE_HOME/netui/test/webapps/drt.
> In the first shell, start Tomcat:
>    ant start
> In the second shell, build the coreWeb web application:
>    ant build
> Now, the "coreWeb" web application is ready to be installed on the 
> running Tomcat server.  The steps needed to deploy to Tomcat can vary 
> depending on the state of Tomcat.  There are three relevant targets:
> deploy -- deploys the webapp for the first time.  This will fail if 
> the webapp is already running.
> undeploy -- undeploys a running webapp from Tomcat.  This will fail if 
> the webapp is not running or did not start successfully
> redeploy -- redeploy a successfully running webapp.  This will fail if 
> the webapp did not start successfully
> These targets can be run in combination to make sure that a webapp 
> deploys correctly to Tomcat; for example, if the "coreWeb" webapp is 
> already deployed, it needs to be redeployed or undeployed and then 
> deployed.  Feedback for each step is available on the console of shell 
> two.  Also, it is possible for a webapp to fail deployment, so just be 
> sure to read the console output messages to make sure that deployment 
> was in fact successful.  To deploy "coreWeb" initially, run in shell two:
>    ant deploy
> To redeploy this webapp, run "ant redeploy" or "ant undeploy deploy".  
> The latter is usually a safe command as it will remove the webapp if 
> it doesn't exist and then deploy it.
> Note, it's acceptable for some of these targets to fail -- if 
> "coreWeb" isn't running on Tomcat and "ant undeploy" is run, it will 
> fail like this:
> :::::
> Buildfile: build.xml
> undeploy:
> undeploy:
> [tomcatundeploy] FAIL - No context exists for path /coreWeb
> C:\dev\apache\beehive\netui\test\webapps\drt\build.xml:147: Following 
> error occured while executing this line
> C:\dev\apache\beehive\test\ant\runTomcat.xml:31: FAIL - No context 
> exists for path /coreWeb
> :::::
> To deploy it, just run "ant deploy" which succeeds with output like this:
> :::::
> Buildfile: build.xml
> deploy:
> deploy:
>     [echo] deploy webapp from 
> file://c:\dev\apache\beehive/netui/test/webapps/drt/coreWeb with 
> context path coreWeb
> [tomcatdeploy] OK - Deployed application at context path /coreWeb
> Total time: 13 seconds
> :::::
> Once the "coreWeb" webapp is deployed successfully, it is accessible 
> using the URL
>    http://localhost:8080/coreWeb
> which should bring up a test suite index page with the Beehive logo 
> and a list of tests and test categories.  When running a particular 
> test, the yellow bar at the bottom of each page is the Test Recorder.
> To run the DRT test suite against a running server, run this command 
> in shell two:
>    ant drt.running
> which puts the test results here:
> $BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html
> To run the BVT test suite from the same directory, run this command in 
> shell two:
>    ant bvt.running
> which puts the test results here:
> $BEEHIVE_HOME/netui/build/testRecorder/junit/bvt/html/junit-noframes.html
> For more information, take a look through the 
> $BEEHIVE_HOME/netui/test/webapps/drt/build.xml file and look at the 
> targets and how they call into the webapp build and start / stop 
> Tomcat targets in build files that live in:
>    $BEEHIVE_HOME/test/ant
> To stop Tomcat, run "ant stop" in shell two.
> When diagnosing a test failure or problem running the webapp, there 
> are several places to look to see if errors have occurred:
> - the Tomcat consoleb
> - the shell two console for errors reported when test recorder tests fail
> - the Tomcat log files, which are stored by date in 
> $BEEHIVE_HOME/external/jakarta-tomcat-5.0.25/logs
> - in addition, NetUI can log error messages into the Tomcat console, 
> and soon into a netui.log file
> When reporting errors into the beehive-dev mailing list, having any 
> failures / errors reported in the above helps when diagnosing a problem.
> Also, because we have error / failure test cases, it is often 
> acceptable for NetUI tests to throw exceptions or errors in the course 
> of regular execution.  While seeing a stack trace can be alarming, 
> lots of tests should do this.  <g>  So, just be sure to check the test 
> result output as the source of record for the success of a particular 
> test or suite.
> Overview of the Test Recorder
> =====================
> The Test Recorder tests comprise the bulk of the NetUI tests suite.  
> The TR is a piece of software that NetUI develops which consists of 
> client and server pieces.  It simulates the interaction of a web 
> browser with a running server.  The TR runs in two modes -- "record" 
> and "playback".
> Developers use the TR in "record" mode to record a set of user 
> interactions through a normal web browser (like FireFox) against a 
> running, Test Recorder enabled web application.  The result is a 
> "recorded" test which is stored in an XML file; examples of these can 
> be found in $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/tests.
> When the DRT or BVT suites run, the client half of the TR uses these 
> "record" XML files to "playback" a set of user interactions with the 
> server.  This is done by reading request parameters out of the 
> "record" file and building requests to send to the server; then, the 
> response stream is captured which results in a "playback" file.  The 
> "record" and "playback" results are compared; if there are no 
> differences between the two, the tests are considered a pass.  
> Otherwise, the test is a failure.  Currently, this diff is done 
> line-by-line and in the future could be done as an XML or HTML diff.
> The TR is a very strict and accurate way for changes in the NetUI 
> runtime to be verified.  Because the response stream of a rendered JSP 
> is differenced against an expected result, even simple changes can 
> break tests, which can be good because it helps NetUI developers 
> understand the impact of changes to the code base.  Thus, we these 
> tests help to maintain NetUI stability.
> The TR is configured on a per-webapp basis using files like those used 
> to configure the DRT's coreWeb webapp:
>    $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/config
> The TR uses its own Ant file to run a test suite against a server:
>    $BEEHIVE_HOME/netui/test/ant/testRecorder.xml
> Certainly, more documentation will follow on the TR and how to record 
> / update test results.
> JUnit API Tests
> ===========
> The source for these tests lives in 
> $BEEHIVE_HOME/netui/test/src/junitTests/...  The TestCases which run 
> as part of the DRTs are specified in 
> $BEEHIVE_HOME/netui/test/ant/  The JUnit DRTs can 
> be run like this:
>    cd $BEEHIVE_HOME/netui/test/ant
>    ant drt
> This will build all of the test code and run the JUnit tests specified 
> in the above file.  We use a simple <netui-junit> Ant task that takes 
> the .properties file above and filters all of the classes in the 
> junitTests.jar file to produce the set of TestCases to run as DRTs.  
> The Ant to drive the JUnit DRTs lives in 
> $BEEHIVE_HOME/netui/test/ant/junit.xml.  The JUnit test results can be 
> found here:
>    $BEEHIVE_HOME/netui/build/drt.testResults/html/junit-noframes.html

View raw message