beehive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eddie O'Neil" <ekon...@bea.com>
Subject Re: Running NetUI Tests -- was Re: build fine, tests passed, coreWeb gives "internal error"
Date Sun, 01 Aug 2004 15:28:51 GMT
Sebastian--

  Terrific; thanks for taking another pass through these.  I agree with 
your assessment -- the Locale just needs to be set at some level, either 
in the specific test or at the webapp level.  The <netui:formatDate> 
tag, for example, has country / language attributes that just don't seem 
to be set in one of those cases.

  That being said, one thing we do need is i18n testing, which could 
take place in the same or in a different webapp.

  Thanks again -- we're almost there.  <g>

Eddie


 

Sebastian Gozin wrote:

> Finally got back to try the beehive tests. Most of it went smooth 
> except the following two tests.
>
> FormatTags
> ModuleConfigLocator
>
> After hitting /coreWeb I noticed both of them have a diff-link set. 
> Here is what they both say.
>
> Test Diffs: FormatTags
>
> Uri( /coreWeb/tags/formatTags/Controller.jpf )
> Test Number( 1 )
> Line (62) does not match.
> R: <span>1/17/02 1:30 PM</span>
> P: <span>17/01/02 13:30</span>
> Line (64) does not match.
> R: <span>1/17/02 1:30 PM</span>
> P: <span>17/01/02 13:30</span>
>
>
> Test Diffs: ModuleConfigLocator
>
> Uri( /coreWeb/miniTests/moduleConfigLocator/index.jsp )
> Test Number( 1 )
> Line (9) does not match.
> R: <html lang="en">
> P: <html lang="nl">
>
>
> Both are locale issues where parts of the webapp are compared to a 
> snapshot from a US-locale based webapp. Perhaps it should be possible 
> to enforce a locale globally in /WEB-INF/netui-config.xml if at all 
> possible. Or is it acceptable to set <netui:html 
> useLocale="something"> on a per file basis just to get tests to pass? 
> Same thing for date formatting as I think the webapp is using 
> <netui:formatString> to do this.
> Since they are tests compared to a snapshot I don't think it's 
> unreasonable to enforce a certain locale as long as it can be set 
> globally. It would also be nice if one could make their own snapshot 
> (in a different locale) to test against.
>
> Sebastian
>
>
> Eddie O'Neil wrote:
>
>>
>>  Seems like a good time to describe how the NetUI test infrastructure 
>> works, specifically "coreWeb" webapp tests.
>>
>>  Note, one of the things that we're waiting for before doing more 
>> documentation of the tests, infrastructure, and code base is for the 
>> wiki to come on-line.  Once that's in place, it's easier to 
>> distribute information in a forum that's easier to digest and reference.
>>
>>  The text below talks about how NetUI tests are structured, how to 
>> diagnose problems, and how to start / stop / build the coreWeb test 
>> webapp.
>>
>>  Hope this helps.
>>
>> Running NetUI Tests
>> ===============
>>
>> The NetUI tests consist of two parts -- standalone JUnit tests and 
>> Test Recorder (TR) tests.  Both types of test are based on JUnit.  
>> The standalone JUnit tests are just that; TestCase subclasses that 
>> test the NetUI APIs outside of a running server.  The Test Recorder 
>> tests are tests that run against a running Tomcat instance in Beehive.
>>
>> Both the standalone JUnit and Test Recorder tests have two broad 
>> categorizations -- DRTs and BVTs.
>> The DRT (Developer Regression Test) suite *must* pass at 100% before 
>> any checkins can be submitted into the $BEEHIVE_HOME/netui tree.
>> The BVT (Build Verification Tests) suite consists of all of tests to 
>> run.  Most of us tend to run the BVTs before we checkin, though they 
>> don't always pass at 100%; for example, today we've got 27 TR tests 
>> that fail out of nearly 400.  Generally, additional BVTs shouldn't 
>> break on checkins either.  <g>
>>
>> Currently, all test suites produce JUnit test results in the usual 
>> format using the <junitreport> Ant task.  These results are produced 
>> in a subdirectory of $BEEHIVE_HOME/netui/build.  The specific 
>> locations for the JUnit and TR tests are below.
>>
>> NetUI Server Tests
>> =============
>> In addition to the API-level JUnit tests, NetUI uses a piece of 
>> home-grown software called the Test Recorder (TR) to run tests 
>> against a live server -- Tomcat is the server in the context of Beehive.
>> The DRT suite currently runs from this directory
>>
>>    $BEEHIVE_HOME/netui/test/webapps/drt/
>>
>> against a webapp in a subdirectory of the above called "coreWeb" 
>> using the Test Recorder configuration / tests stored subdirectories 
>> "testRecorder/config" and "testRecorder/tests" respectively.  There 
>> is also a build.xml file in the above directory that controls the 
>> Tomcat server, coreWeb build, and running test suites.  This 
>> directory is assumed to be the working directory for the rest of the 
>> Ant commands in this section.
>>
>> In order to run a set of TR tests, the following steps occur:
>>
>> - build the coreWeb webapp
>> - start Tomcat
>> - deploy the coreWeb webapp
>> - run a test suite (drt, bvt, data binding, etc), which produces an 
>> HTML test result report
>> - stop Tomcat
>>
>> Currently in Beehive, the instance of Tomcat that is used lives in:
>>
>>  $BEEHIVE_HOME/installed/jakarta-tomcat-5.0.25/...
>>
>> When the DRTs run as part of an "ant clean deploy drt" from either 
>> $BEEHIVE_HOME/ or $BEEHIVE_HOME/netui, this process is done 
>> automatically meaning that the build takes care of building the 
>> webapp, starting / stopping Tomcat, and running the tests.  The 
>> result of a DRT run can be found here:
>>
>>    
>> $BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html 
>>
>>
>> In addition to an automatic DRT run, the same steps can be run 
>> manually from the $BEEHIVE_HOME/netui/test/webapps/drt directory.  
>> This is very useful when developing tests or just learning about 
>> NetUI as a normal web browser can be used to walk the tests 
>> manually.  To start the Tomcat server, you'll need two shells both 
>> with a Beehive development environment (see 
>> $BEEHIVE_HOME/BUILDING.txt) in the directory 
>> $BEEHIVE_HOME/netui/test/webapps/drt.
>>
>> In the first shell, start Tomcat:
>>
>>    ant start
>>
>> In the second shell, build the coreWeb web application:
>>
>>    ant build
>>
>> Now, the "coreWeb" web application is ready to be installed on the 
>> running Tomcat server.  The steps needed to deploy to Tomcat can vary 
>> depending on the state of Tomcat.  There are three relevant targets:
>>
>> deploy -- deploys the webapp for the first time.  This will fail if 
>> the webapp is already running.
>> undeploy -- undeploys a running webapp from Tomcat.  This will fail 
>> if the webapp is not running or did not start successfully
>> redeploy -- redeploy a successfully running webapp.  This will fail 
>> if the webapp did not start successfully
>>
>> These targets can be run in combination to make sure that a webapp 
>> deploys correctly to Tomcat; for example, if the "coreWeb" webapp is 
>> already deployed, it needs to be redeployed or undeployed and then 
>> deployed.  Feedback for each step is available on the console of 
>> shell two.  Also, it is possible for a webapp to fail deployment, so 
>> just be sure to read the console output messages to make sure that 
>> deployment was in fact successful.  To deploy "coreWeb" initially, 
>> run in shell two:
>>
>>    ant deploy
>>
>> To redeploy this webapp, run "ant redeploy" or "ant undeploy 
>> deploy".  The latter is usually a safe command as it will remove the 
>> webapp if it doesn't exist and then deploy it.
>> Note, it's acceptable for some of these targets to fail -- if 
>> "coreWeb" isn't running on Tomcat and "ant undeploy" is run, it will 
>> fail like this:
>> :::::
>> Buildfile: build.xml
>> undeploy:
>> undeploy:
>> [tomcatundeploy] FAIL - No context exists for path /coreWeb
>> BUILD FAILED
>> C:\dev\apache\beehive\netui\test\webapps\drt\build.xml:147: Following 
>> error occured while executing this line
>> C:\dev\apache\beehive\test\ant\runTomcat.xml:31: FAIL - No context 
>> exists for path /coreWeb
>> :::::
>>
>> To deploy it, just run "ant deploy" which succeeds with output like 
>> this:
>> :::::
>> Buildfile: build.xml
>> deploy:
>> deploy:
>>     [echo] deploy webapp from 
>> file://c:\dev\apache\beehive/netui/test/webapps/drt/coreWeb with 
>> context path coreWeb
>> [tomcatdeploy] OK - Deployed application at context path /coreWeb
>> BUILD SUCCESSFUL
>> Total time: 13 seconds
>> :::::
>>
>> Once the "coreWeb" webapp is deployed successfully, it is accessible 
>> using the URL
>>
>>    http://localhost:8080/coreWeb
>>
>> which should bring up a test suite index page with the Beehive logo 
>> and a list of tests and test categories.  When running a particular 
>> test, the yellow bar at the bottom of each page is the Test Recorder.
>>
>> To run the DRT test suite against a running server, run this command 
>> in shell two:
>>
>>    ant drt.running
>>
>> which puts the test results here:
>>
>>    
>> $BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html 
>>
>>
>> To run the BVT test suite from the same directory, run this command 
>> in shell two:
>>
>>    ant bvt.running
>>
>> which puts the test results here:
>>
>>    
>> $BEEHIVE_HOME/netui/build/testRecorder/junit/bvt/html/junit-noframes.html 
>>
>>
>> For more information, take a look through the 
>> $BEEHIVE_HOME/netui/test/webapps/drt/build.xml file and look at the 
>> targets and how they call into the webapp build and start / stop 
>> Tomcat targets in build files that live in:
>>
>>    $BEEHIVE_HOME/test/ant
>>
>> To stop Tomcat, run "ant stop" in shell two.
>>
>> When diagnosing a test failure or problem running the webapp, there 
>> are several places to look to see if errors have occurred:
>>
>> - the Tomcat consoleb
>> - the shell two console for errors reported when test recorder tests 
>> fail
>> - the Tomcat log files, which are stored by date in 
>> $BEEHIVE_HOME/external/jakarta-tomcat-5.0.25/logs
>> - in addition, NetUI can log error messages into the Tomcat console, 
>> and soon into a netui.log file
>>
>> When reporting errors into the beehive-dev mailing list, having any 
>> failures / errors reported in the above helps when diagnosing a problem.
>> Also, because we have error / failure test cases, it is often 
>> acceptable for NetUI tests to throw exceptions or errors in the 
>> course of regular execution.  While seeing a stack trace can be 
>> alarming, lots of tests should do this.  <g>  So, just be sure to 
>> check the test result output as the source of record for the success 
>> of a particular test or suite.
>>
>> Overview of the Test Recorder
>> =====================
>> The Test Recorder tests comprise the bulk of the NetUI tests suite.  
>> The TR is a piece of software that NetUI develops which consists of 
>> client and server pieces.  It simulates the interaction of a web 
>> browser with a running server.  The TR runs in two modes -- "record" 
>> and "playback".
>> Developers use the TR in "record" mode to record a set of user 
>> interactions through a normal web browser (like FireFox) against a 
>> running, Test Recorder enabled web application.  The result is a 
>> "recorded" test which is stored in an XML file; examples of these can 
>> be found in $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/tests.
>> When the DRT or BVT suites run, the client half of the TR uses these 
>> "record" XML files to "playback" a set of user interactions with the 
>> server.  This is done by reading request parameters out of the 
>> "record" file and building requests to send to the server; then, the 
>> response stream is captured which results in a "playback" file.  The 
>> "record" and "playback" results are compared; if there are no 
>> differences between the two, the tests are considered a pass.  
>> Otherwise, the test is a failure.  Currently, this diff is done 
>> line-by-line and in the future could be done as an XML or HTML diff.
>>
>> The TR is a very strict and accurate way for changes in the NetUI 
>> runtime to be verified.  Because the response stream of a rendered 
>> JSP is differenced against an expected result, even simple changes 
>> can break tests, which can be good because it helps NetUI developers 
>> understand the impact of changes to the code base.  Thus, we these 
>> tests help to maintain NetUI stability.
>>
>> The TR is configured on a per-webapp basis using files like those 
>> used to configure the DRT's coreWeb webapp:
>>
>>    $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/config
>>
>> The TR uses its own Ant file to run a test suite against a server:
>>
>>    $BEEHIVE_HOME/netui/test/ant/testRecorder.xml
>>
>> Certainly, more documentation will follow on the TR and how to record 
>> / update test results.
>>
>> JUnit API Tests
>> ===========
>> The source for these tests lives in 
>> $BEEHIVE_HOME/netui/test/src/junitTests/...  The TestCases which run 
>> as part of the DRTs are specified in 
>> $BEEHIVE_HOME/netui/test/ant/junitDrt.properties.  The JUnit DRTs can 
>> be run like this:
>>
>>    cd $BEEHIVE_HOME/netui/test/ant
>>    ant drt
>>
>> This will build all of the test code and run the JUnit tests 
>> specified in the above file.  We use a simple <netui-junit> Ant task 
>> that takes the .properties file above and filters all of the classes 
>> in the junitTests.jar file to produce the set of TestCases to run as 
>> DRTs.  The Ant to drive the JUnit DRTs lives in 
>> $BEEHIVE_HOME/netui/test/ant/junit.xml.  The JUnit test results can 
>> be found here:
>>
>>    $BEEHIVE_HOME/netui/build/drt.testResults/html/junit-noframes.html
>>
>
>

Mime
View raw message