jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milamber <milam...@apache.org>
Subject Re: JMeter Performance evolution : My 2 cents
Date Mon, 21 May 2012 21:51:58 GMT


Le 21/05/2012 22:12, Philippe Mouawad a ecrit :
> Hello,
> I read recently this little comparison :
> https://github.com/excilys/gatling/wiki/Benchmarks<https://github.com/excilys/gatling/wiki/Benchmark>
>
> I reviewed the test plan that was used to make the test.
> It seems to me the test is little biased:
>
>    - View Results Tree is in test plan (as it uses a lot of memory, it's a
>    big issue)
>    - View Results in Table (same thing)
>    - 3 Non Standard JMeter listener, so It's not pure JMeter:
>       - jp@gc - Transactions per Second
>       - jp@gc - Response Times Over Time
>       - jp@gc - Active Threads Over Time
>    - Default XML output seems to have been used , it's against best
>    practices
>
>
> Besides the following conclusions that seem to me non scientific and purely
> subjective:
>
>    - The testers, new to both Gatling and JMeter found that JMeter was
>    harder to learn and use than Gatling to create the simulations, despite the
>    use of a proxy.
>
>
> I will only look at other ideas expressed:
>
>    - JMeter creates one thread per user simulated. If there is not enough
>    memory allocated to the JVM, it can crash trying to create these threads
>       - This need to be detailed, cause either it fails with OOM and it's
>       not during thread creation, either it fails with "unable to create new
>       native thread"
>    - For instance, JMeter could not run 1500 users with 512 MB (what was
>    used for Gatling even with 2000 users); OutOfMemoryErrors are recorded in
>    the table as *OOM*
>       - => I made a Test with up to 2000 Threads with 512 m without any
>       crash, it depends on Test and on application
>    - Another problem occurred with the 2000 users simulations; it seems
>    that JMeter can not simulate more than 1514 users independently from the
>    memory that was allocated to the JVM
>       - => I made a Test with up to 2000 Threads with 512 m without any
>       crash, so assertion is false, it depends on Test and on application
>
>
> As it's difficult to install the application used for test (last version
> does not seem to work as expected) , if they provide a working WAR against
> a local Postgres DB I will be happy to test with it.
> But in current state, application seems to be packaged for cloud or H2
> local DB, I didn't want to spend too much time setting up application as I
> don't know its real status.
>
> I just tried to run Test Plan against a blank tomcat to verify what they
> say about Thread Creation, I didn't find any issue on this.
>
> So I decided to make a very simple scenario test on Tomcat Examples (It
> goes to Session Example, adds attribute, go back to index, go back to
> Session Example, test contains Response assertion for each Request).
>   

Please, indicate the Tomcat version used? it's the same machine? config
of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM editor?

JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)?

Can you post the jmx file?

Milamber


> It is not at all representative but it is a way for me to check potential
> issues in JMeter and performance changes accross versions.
>
> I ran the test with 1500 VU using JMeter 2.5.1,  2.7 (current trunk) with
> -Xmx512m, 10 minutes run and CSV output against a local Tomcat (I restard
> tomcat between tests and control its health):
>
>    - I noticed that current trunk version behaves much better in terms of
>    memory than 2.5.1 or 2.6:
>       - In 2.5.1 :
>       - GC activity is much higher with around 5 GC CPU peaks every 2
>          minutes,  and 20 FULL GC of 700 to 800 ms each
>          - Throughput: 97,71%
>          - Pauses : 13,69s
>          - Mém : 391M/min
>          - Full GC tend to be much more frequent at end of test
>          - 2.7:
>          -  no GC CPU peak, 1 FULL GC
>          - Throughput:98.54
>          - Pauses : 8.9s
>          - 1108m /min
>          - Summary:
>       - 25.1:
>          - 164676 samples in 605,1s
>          - 272,2/s
>          - Avg:    97
>          - 2.7:
>          - 165367 in 605.0s
>          - 273.3/s
>          - Avg:   228
>          - I also noticed results have a much better look:
>       - in 2.5.1, Transactions/s are around 300 / sec during 4 minutes,
>       then drop to 200/s, go up to 400/s , then down to 260/s and
> finally 200/sec
>       - in 2.7, Transactions stay around 300/sec
>
>
>
> Actions:
>
>    - I think it would be useful to have some reference application on which
>    we could test JMeter behaviour
>    - What would the best place to put these comparisons ? wiki ? which
>    indicators should we put ?
>    - Further testing should be done against a richer application that could
>    be deployed locally
>
>
> Regards
> Philippe
>
>   


Mime
View raw message