jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe Mouawad <philippe.moua...@gmail.com>
Subject Re: JMeter Performance evolution : My 2 cents
Date Thu, 24 May 2012 21:15:49 GMT
Hello,
How can I insert image ? button is disabled in GUI mode.
To insert Test Plan, is it an attachment ?

Thanks
Regards
Philippe

On Wed, May 23, 2012 at 12:22 AM, sebb <sebbaz@gmail.com> wrote:

> On 22 May 2012 20:18, Philippe Mouawad <philippe.mouawad@gmail.com> wrote:
> > Hello Milamber,
> > My responses below.
> > Regards
> > Philippe
> >
> > On Mon, May 21, 2012 at 11:51 PM, Milamber <milamber@apache.org> wrote:
> >
> >>
> >>
> >> Le 21/05/2012 22:12, Philippe Mouawad a ecrit :
> >> > Hello,
> >> > I read recently this little comparison :
> >> > https://github.com/excilys/gatling/wiki/Benchmarks<
> >> https://github.com/excilys/gatling/wiki/Benchmark>
> >> >
> >> > I reviewed the test plan that was used to make the test.
> >> > It seems to me the test is little biased:
> >> >
> >> >    - View Results Tree is in test plan (as it uses a lot of memory,
> it's
> >> a
> >> >    big issue)
> >> >    - View Results in Table (same thing)
> >> >    - 3 Non Standard JMeter listener, so It's not pure JMeter:
> >> >       - jp@gc - Transactions per Second
> >> >       - jp@gc - Response Times Over Time
> >> >       - jp@gc - Active Threads Over Time
> >> >    - Default XML output seems to have been used , it's against best
> >> >    practices
> >> >
> >> >
> >> > Besides the following conclusions that seem to me non scientific and
> >> purely
> >> > subjective:
> >> >
> >> >    - The testers, new to both Gatling and JMeter found that JMeter was
> >> >    harder to learn and use than Gatling to create the simulations,
> >> despite the
> >> >    use of a proxy.
> >> >
> >> >
> >> > I will only look at other ideas expressed:
> >> >
> >> >    - JMeter creates one thread per user simulated. If there is not
> enough
> >> >    memory allocated to the JVM, it can crash trying to create these
> >> threads
> >> >       - This need to be detailed, cause either it fails with OOM and
> it's
> >> >       not during thread creation, either it fails with "unable to
> create
> >> new
> >> >       native thread"
> >> >    - For instance, JMeter could not run 1500 users with 512 MB (what
> was
> >> >    used for Gatling even with 2000 users); OutOfMemoryErrors are
> >> recorded in
> >> >    the table as *OOM*
> >> >       - => I made a Test with up to 2000 Threads with 512 m without
> any
> >> >       crash, it depends on Test and on application
> >> >    - Another problem occurred with the 2000 users simulations; it
> seems
> >> >    that JMeter can not simulate more than 1514 users independently
> from
> >> the
> >> >    memory that was allocated to the JVM
> >> >       - => I made a Test with up to 2000 Threads with 512 m without
> any
> >> >       crash, so assertion is false, it depends on Test and on
> application
> >> >
> >> >
> >> > As it's difficult to install the application used for test (last
> version
> >> > does not seem to work as expected) , if they provide a working WAR
> >> against
> >> > a local Postgres DB I will be happy to test with it.
> >> > But in current state, application seems to be packaged for cloud or H2
> >> > local DB, I didn't want to spend too much time setting up application
> as
> >> I
> >> > don't know its real status.
> >> >
> >> > I just tried to run Test Plan against a blank tomcat to verify what
> they
> >> > say about Thread Creation, I didn't find any issue on this.
> >> >
> >> > So I decided to make a very simple scenario test on Tomcat Examples
> (It
> >> > goes to Session Example, adds attribute, go back to index, go back to
> >> > Session Example, test contains Response assertion for each Request).
> >> >
> >>
> >> Please, indicate the Tomcat version used? it's the same machine? config
> >> of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM editor?
> >>
> >> Tomcat apache-tomcat-6.0.24
> > -Xms256m -Xmx1024m
> >    <Connector port="8080" protocol="HTTP/1.1"
> >
> >
> compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json"
> >                compression="off"
> >                socketBuffer="8"
> >                maxThreads="400"
> >               connectionTimeout="20000"
> >               redirectPort="8443" />
> > Set session timeout in web.xml  to 1min
> > java version "1.6.0_29"
> > Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
> > Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> >
> > Mac OS 10.6.8
> > JMeter  and Tomcat on same machine
> > No particular OS Tuning
> > 8GO RAM
> > No swap, nothing runnning except these 2
> > Tomcat CPU around 5%
> > Memory around 50 mo
> >
> > JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)?
> >>
> >> Same config
> > No TCP tuning
> >
> >> Can you post the jmx file?
> >>
> >> Will it be accepted on this list ?
>
> Best to create a Wiki page and attach the test file there.
>
> >
> >> Milamber
> >>
> >>
> >> > It is not at all representative but it is a way for me to check
> potential
> >> > issues in JMeter and performance changes accross versions.
> >> >
> >> > I ran the test with 1500 VU using JMeter 2.5.1,  2.7 (current trunk)
> with
> >> > -Xmx512m, 10 minutes run and CSV output against a local Tomcat (I
> restard
> >> > tomcat between tests and control its health):
> >> >
> >> >    - I noticed that current trunk version behaves much better in
> terms of
> >> >    memory than 2.5.1 or 2.6:
> >> >       - In 2.5.1 :
> >> >       - GC activity is much higher with around 5 GC CPU peaks every 2
> >> >          minutes,  and 20 FULL GC of 700 to 800 ms each
> >> >          - Throughput: 97,71%
> >> >          - Pauses : 13,69s
> >> >          - Mém : 391M/min
> >> >          - Full GC tend to be much more frequent at end of test
> >> >          - 2.7:
> >> >          -  no GC CPU peak, 1 FULL GC
> >> >          - Throughput:98.54
> >> >          - Pauses : 8.9s
> >> >          - 1108m /min
> >> >          - Summary:
> >> >       - 25.1:
> >> >          - 164676 samples in 605,1s
> >> >          - 272,2/s
> >> >          - Avg:    97
> >> >          - 2.7:
> >> >          - 165367 in 605.0s
> >> >          - 273.3/s
> >> >          - Avg:   228
> >> >          - I also noticed results have a much better look:
> >> >       - in 2.5.1, Transactions/s are around 300 / sec during 4
> minutes,
> >> >       then drop to 200/s, go up to 400/s , then down to 260/s and
> >> > finally 200/sec
> >> >       - in 2.7, Transactions stay around 300/sec
> >> >
> >> >
> >> >
> >> > Actions:
> >> >
> >> >    - I think it would be useful to have some reference application on
> >> which
> >> >    we could test JMeter behaviour
> >> >    - What would the best place to put these comparisons ? wiki ? which
> >> >    indicators should we put ?
> >> >    - Further testing should be done against a richer application that
> >> could
> >> >    be deployed locally
> >> >
> >> >
> >> > Regards
> >> > Philippe
> >> >
> >> >
> >>
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>



-- 
Cordialement.
Philippe Mouawad.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message