jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe Mouawad <philippe.moua...@gmail.com>
Subject JMeter Performance evolution : My 2 cents
Date Mon, 21 May 2012 21:12:42 GMT
Hello,
I read recently this little comparison :
https://github.com/excilys/gatling/wiki/Benchmarks<https://github.com/excilys/gatling/wiki/Benchmark>

I reviewed the test plan that was used to make the test.
It seems to me the test is little biased:

   - View Results Tree is in test plan (as it uses a lot of memory, it's a
   big issue)
   - View Results in Table (same thing)
   - 3 Non Standard JMeter listener, so It's not pure JMeter:
      - jp@gc - Transactions per Second
      - jp@gc - Response Times Over Time
      - jp@gc - Active Threads Over Time
   - Default XML output seems to have been used , it's against best
   practices


Besides the following conclusions that seem to me non scientific and purely
subjective:

   - The testers, new to both Gatling and JMeter found that JMeter was
   harder to learn and use than Gatling to create the simulations, despite the
   use of a proxy.


I will only look at other ideas expressed:

   - JMeter creates one thread per user simulated. If there is not enough
   memory allocated to the JVM, it can crash trying to create these threads
      - This need to be detailed, cause either it fails with OOM and it's
      not during thread creation, either it fails with "unable to create new
      native thread"
   - For instance, JMeter could not run 1500 users with 512 MB (what was
   used for Gatling even with 2000 users); OutOfMemoryErrors are recorded in
   the table as *OOM*
      - => I made a Test with up to 2000 Threads with 512 m without any
      crash, it depends on Test and on application
   - Another problem occurred with the 2000 users simulations; it seems
   that JMeter can not simulate more than 1514 users independently from the
   memory that was allocated to the JVM
      - => I made a Test with up to 2000 Threads with 512 m without any
      crash, so assertion is false, it depends on Test and on application


As it's difficult to install the application used for test (last version
does not seem to work as expected) , if they provide a working WAR against
a local Postgres DB I will be happy to test with it.
But in current state, application seems to be packaged for cloud or H2
local DB, I didn't want to spend too much time setting up application as I
don't know its real status.

I just tried to run Test Plan against a blank tomcat to verify what they
say about Thread Creation, I didn't find any issue on this.

So I decided to make a very simple scenario test on Tomcat Examples (It
goes to Session Example, adds attribute, go back to index, go back to
Session Example, test contains Response assertion for each Request).
It is not at all representative but it is a way for me to check potential
issues in JMeter and performance changes accross versions.

I ran the test with 1500 VU using JMeter 2.5.1,  2.7 (current trunk) with
-Xmx512m, 10 minutes run and CSV output against a local Tomcat (I restard
tomcat between tests and control its health):

   - I noticed that current trunk version behaves much better in terms of
   memory than 2.5.1 or 2.6:
      - In 2.5.1 :
      - GC activity is much higher with around 5 GC CPU peaks every 2
         minutes,  and 20 FULL GC of 700 to 800 ms each
         - Throughput: 97,71%
         - Pauses : 13,69s
         - Mém : 391M/min
         - Full GC tend to be much more frequent at end of test
         - 2.7:
         -  no GC CPU peak, 1 FULL GC
         - Throughput:98.54
         - Pauses : 8.9s
         - 1108m /min
         - Summary:
      - 25.1:
         - 164676 samples in 605,1s
         - 272,2/s
         - Avg:    97
         - 2.7:
         - 165367 in 605.0s
         - 273.3/s
         - Avg:   228
         - I also noticed results have a much better look:
      - in 2.5.1, Transactions/s are around 300 / sec during 4 minutes,
      then drop to 200/s, go up to 400/s , then down to 260/s and
finally 200/sec
      - in 2.7, Transactions stay around 300/sec



Actions:

   - I think it would be useful to have some reference application on which
   we could test JMeter behaviour
   - What would the best place to put these comparisons ? wiki ? which
   indicators should we put ?
   - Further testing should be done against a richer application that could
   be deployed locally


Regards
Philippe

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message