jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: JMeter Performance evolution : My 2 cents
Date Sun, 27 May 2012 22:02:29 GMT
On Sun, 2012-05-27 at 23:08 +0200, Philippe Mouawad wrote:
> Hello Oleg,milamber,
> Thanks for your investigations.
> 
> Oleg, Regarding connectionmanager, do you think it's better to use one
> connmgr for all vus ?
> Today we use one per VU but we reuse httpclient instance when downloading
> embedded résources in parallèl mode.
> What would be the compromise ?
> 
> Thanks
> Regards
> Philippe
> 

Hi Philippe

It really depends on what you define as 'better' and what your
objectives are. If your goal is to emulate load generated by 20 browser
instances as realistic as possible you should be using a separate
connection manager per virtual user. (In this case, though, you should
be using a pooling connection configured to allow up to 2 simultaneous
connections to the same host instead of a basic connection manager with
a single connection). If your goal is to generate as much load as
possible using 20 virtual users you should be using a pooling connection
manager configured to allow up to 20 connections to the same host shared
by all virtual users.

You should also take into account that Java standard HttpUrlConnection
is not well suited for the former scenario as it always uses one global
connection pool for all its instances. This is likely to be the reason
why the HttpUrlConnection based sampler tends to yield the highest
request per second in the performance tests.

Hope this helps

Oleg 

> 
> 
> On Sunday, May 27, 2012, Oleg Kalnichevski wrote:
> 
> > On Sun, 2012-05-27 at 16:47 +0100, Milamber wrote:
> > >
> > >
> > > Le 27/05/2012 14:57, Oleg Kalnichevski a ecrit :
> > > > On Sat, 2012-05-26 at 23:14 +0100, Milamber wrote:
> > > >
> > > > > Le 26/05/2012 21:25, Milamber a ecrit :
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Thanks for the wiki page.
> > > > > >
> > > > > > I've got the test plan, and I run some tests.
> > > > > >
> > > > > > Target:
> > > > > > 1 VM on KVM host | 2 Gb RAM | 4 vCPU
> > > > > > Debian 6.0.5 (64 bits)
> > > > > > Tomcat 6.0.35 (from Apache site)
> > > > > > Sun Java 1.6u32 (64bits)
> > > > > > Restart Tomcat before each test
> > > > > > Tomcat with default params.
> > > > > >
> > > > > > JMeter machine (another machine) :
> > > > > > 8 Gb RAM | 2 core 2.53GHz
> > > > > > Debian 6.0.5  (64 bits)
> > > > > > Sun Java 1.6u32 (64bits)
> > > > > > JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt
> > > > > >
> > > > > > With the GC log, I've make some stats.
> > > > > >
> > > > > > With the original test plan: HttpClient 4:
> > > > > > (==>4.1)
> > > > > > JMeter_251    Nb GC: 202    Full GC: 17    Time GC: 13.9376210
> >  Full:
> > > > > > 11.3386420    Total: 25.2762630
> > > > > > JMeter_26    Nb GC: 216    Full GC: 18    Time GC: 14.6083630
> >  Full:
> > > > > > 12.0604050    Total: 26.6687680
> > > > > > JMeter_27    Nb GC: 199    Full GC: 15    Time GC: 11.7238800
> >  Full:
> > > > > > 9.5478390    Total: 21.2717190
> > > > > >
> > > > > > Stats results HC4: (==>4.1)
> > > > > > JMeter 2.5.1: Requests 163917 in 604.8s =  271.0/s Avg:   148
> > > > > > JMeter 2.6: Requests 163208 in 605.0s =  269.7/s Avg:   155
> > > > > > JMeter 2.7RC3: 164035 in 605.1s =  271.1/s Avg:   159
> > > > > >
> > > > > >
> > > > > With JMeter trunk and HttpClient 4.2 (core, client, mime):
> > > > >
> > > > > JMeter_trunk_hc42    Nb GC: 131    Full GC: 1    Time GC: 5.7903490
> > > > > Full: .0285440    Total: 5.8188930
> > > > >
> > > > > JMeter trunk with HC 4.2 (not commit): Nb Requests 164762 in 605.2s
=
> > > > > 272.3/s Avg:   250
> > > > >
> > > > >
> > > > > Results (GC counts and avg req/s) with the HC 4.2 are same with HC3.1
> > > > > and Java.
> > > > >
> > > > > => HC 4.2 seems fixed a performance issue with JVM memory?
> > > > >
> > > > > => Release 2.7.1 to do with HC4.2 ? (or update HC and make a RC4?)
> > > > >
> > > > > Milamber
> > > > >
> > > > >
> > > > >
> > > > > > Changes HttpClient to 3.1:
> > > > > > JMeter_251    Nb GC: 136    Full GC: 1    Time GC: 8.6780490
> >  Full:
> > > > > > .0254120    Total: 8.7034610
> > > > > > JMeter_26    Nb GC: 146    Full GC: 1    Time GC: 9.0567930
> >  Full:
> > > > > > .0305490    Total: 9.0873420
> > > > > > JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.9212530
> >  Full:
> > > > > > .0282870    Total: 6.9495400
> > > > > >
> > > > > > Stats results HC3.1:
> > > > > > JMeter 2.5.1: Requests 165103 in 605.1s =  272.9/s Avg:   233
> > > > > > JMeter 2.6: Requests 165063 in 605.1s =  272.8/s Avg:   214
> > > > > > JMeter 2.7RC3: Requests 164860 in 604.5s =  272.7/s Avg:   180
> > > > > >
> > > > > > Changes to Java http client:
> > > > > > JMeter_251    Nb GC: 138    Full GC: 1    Time GC: 8.1128620
> >  Full:
> > > > > > .0253190    Total: 8.1381810
> > > > > > JMeter_26    Nb GC: 149    Full GC: 2    Time GC: 8.4811800
> >  Full:
> > > > > > .6600440    Total: 9.1412240
> > > > > > JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.4912220
> >  Full:
> > > > > > .0283680    To>  Biggest Top-Level Dominator Classes
> > >
> > >     Label
> > >   Number of
> > >    Objects
> > > Used Heap Size
> > > Retained Heap
> > >      Size
> > > Retained Heap,
> > >       %
> > > org.apache.jmeter.protocol.http.control.HeaderManager
> > >          6,000
> > >        384,000
> > >     40,142,160
> > >         18.26%
> > > org.apache.http.impl.conn.SingleClientConnManager
> > >          1,500
> > >        120,000
> > >     33,289,328
> > >         15.14%
> > > org.apache.jmeter.threads.JMeterThread
> > >          1,500
> > >        216,000
> > >     31,718,400
> > >         14.43%
> > > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy
> > >          6,004
> > >        384,256
> > >     30,097,176
> > >         13.69%
> > > java.lang.Thread
> > >          1,510
> > >        265,760
> > >     18,165,056
> > >          8.26%
> > > org.apache.jmeter.assertions.ResponseAssertion
> > >          6,000
> > >        336,000
> > >      9,504,000
> > >          4.32%
> > > org.apache.jmeter.threads.JMeterContext
> > >          1,500
> > >        120,000
> > >      8,230,528
> > >          3.74%
> > > org.apache.jmeter.sampler.TestAction
> > >          6,000
> > >        336,000
> > >      7,680,000
> > >          3.49%
> > > org.apache.jmeter.threads.ThreadGroup
> > >          1,501
> > >         96,064
> > >      5,379,528
> > >          2.45%
> > > org.apache.jmeter.config.ConfigTestElement
> > >          1,500
> > >         84,000
> > >      4,648,080
> > >          2.11%
> > > org.apache.jmeter.protocol.http.control.CookieManager
> > >          1,501
> > >        108,072
> > >      4,296,984
> > >          1.95%
> > > java.lang.Class
> > >          1,759
> > >         27,840
> > >      4,177,528
> > >          1.90%
> > > org.apache.http.conn.scheme.SchemeRegistry
> > >          1,500
> > >         36,000
> > >      3,725,520
> > >          1.69%
> > > org.apache.jmeter.config.Arguments
> > >          1,500
> > >         84,000
> > >      3,180,000
> > >          1.45%
> > > org.apache.jmeter.control.TransactionController
> > >          1,500
> > >        204,000
> > >      2,832,000
> > >          1.29%
> > >
> > >       * Total:
> > >         15
> > >         entries
> > >         40,775
> > >      2,801,992
> > >    207,066,288
> > >
> > >
> > >
> > >
> > >
> > > With JMeter Trunk + HttpClient 4.2 :
> > >
> > >
> > >  Biggest Top-Level Dominator Classes
> > >     Label
> > >   Number of
> > >    Objects
> > > Used Heap Size
> > > Retained Heap
> > >      Size
> > > Retained Heap,
> > >       %
> > > org.apache.jmeter.protocol.http.control.HeaderManager
> > >          6,000
> > >        384,000
> > >     41,283,936
> > >         22.75%
> > > org.apache.jmeter.threads.JMeterThread
> > >          1,500
> > >        216,000
> > >     31,925,472
> > >         17.59%
> > > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy
> > >          6,004
> > >        384,256
> > >     28,151,656
> > >         15.51%
> > > java.lang.Thread
> > >          1,504
> > >        264,704
> > >     15,899,168
> > >          8.76%
> > > org.apache.jmeter.assertions.ResponseAssertion
> > >          6,000
> > >        336,000
> > >      9,441,672
> > >          5.20%
> > > org.apache.jmeter.sampler.TestAction
> > >          6,000
> > >        336,000
> > >      7,601,096
> > >          4.19%
> > > org.apache.jmeter.threads.JMeterContext
> > >          1,332
> > >        106,560
> > >      7,156,624
> > >          3.94%
> > > org.apache.jmeter.threads.ThreadGroup
> > >          1,501
> > >         96,064
> > >      5,353,896
> > >          2.95%
> > > org.apache.jmeter.config.ConfigTestElement
> > >          1,500
> > >         84,000
> > >      5,141,184
> > >          2.83%
> > > org.apache.jmeter.protocol.http.control.CookieManager
> > >          1,501
> > >        108,072
> > >      4,582,536
> > >          2.52%
> > > org.apache.http.impl.conn.BasicClientConnectionManager
> > >          1,332
> > >         85,248
> > >      4,256,488
> > >          2.35%
> > > java.lang.Class
> > >          1,228
> > >         22,200
> > >      4,006,160
> > >          2.21%
> > > org.apache.http.conn.scheme.SchemeRegistry
> > >          1,332
> > >         31,968
> > >      3,314,016
> > >          1.83%
> > > org.apache.jmeter.config.Arguments
> > >          1,500
> > >         84,000
> > >      3,161,184
> > >          1.74%
> > > org.apache.jmeter.control.TransactionController
> > >          1,500
> > >        204,000
> > >      2,737,920
> > >          1.51%
> > > org.apache.jmeter.control.LoopController
> > >          1,500
> > >        144,000
> > >      2,150,592
> > >          1.18%
> > >
> > >       * Total:
> > >         16
> > >         entries
> > >         41,234
> > >      2,887,072
> > >    176,163,600
> > >
> > >
> > >
> > >
> > >
> > >
> > > When I observed in the HC4.1/JMeter, with outgoing reference for
> > > SingleClientConnManager :
> > > connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl
> > >
> > > Class Name
> > > | Shallow Heap | Retained Heap
> > >
> > ------------------------------------------------------------------------------------------------------------------------------------------
> > > class org.apache.http.impl.conn.SingleClientConnManager @ 0xd8bd20a0
> > > |            8 |           336
> > > |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> > > 0xe6be9450                                  |           80 |
> > > 27,008
> > > |  |- connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl
> > > $4 @ 0xe6be9320                       |          168 |         9,536
> > > |  |  |- value java.util.HashMap$Entry @ 0xe6be9258
> > > |           48 |         9,784
> > > |  |  |  '- [10] java.util.HashMap$Entry[16] @ 0xe6be9208
> > > |          152 |         9,936
> > > |  |  |     '- table java.util.HashMap @ 0xe6be91d8
> > > |           64 |        10,000
> > > |  |  |        '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @
> > > 0xe6be91b8                            |           56 |        10,056
> > > |  |  |           '- [4] java.lang.ThreadLocal$ThreadLocalMap
> > > $Entry[16] @ 0xe3fed728                       |          152 |
> > > 11,064
> > > |  |  |              '- table java.lang.ThreadLocal$ThreadLocalMap @
> > > 0xe3fed710                            |           32 |        11,096
> > > |  |  |                 '- threadLocals java.lang.Thread @ 0xe3fe4b00
> > > Thread Group 1-1483 Thread          |          176 |        11,592
> > > |  |  |                    |- [1486] java.lang.Thread[2048] @
> > > 0xe3218f40                                   |       16,408 |
> > > 16,408
> > > |  |  |                    |  '- threads java.lang.ThreadGroup @
> > > 0xe0000030  main                          |           72 |
> > > 16,552
> > > |  |  |                    |- <Java Local>, me java.lang.Thread @
> > > 0xe3fe4b00  Thread Group 1-1483 Thread   |          176 |
> > > 11,592
> > > |  |  |                    |  |- [1486] java.lang.Thread[2048] @
> > > 0xe3218f40                                |       16,408 |
> > > 16,408
> > > |  |  |                    |  |- <Java Local>, me java.lang.Thread @
> > > 0xe3fe4b00  Thread Group 1-1483 Thread|          176 |        11,592
> > > |  |  |                    |  |- value
> > > java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0       |
> > > 48 |            48
> > > |  |  |                    |  '- Total: 3 entries
> > > |     The new connection managers are definitely better in terms of code
> > quality. However, I did not expect them to be any faster, though, as
> > they pretty much shared the same design with the old ones. At the same
> > time it does appear that BasicClientConnectionManager tends to have a
> > much smaller footprint judging by the heap dumps, which is a pleasant
> > surprise.
> >
> > What is not quite clear to me is why JMeter creates so many connection
> > managers at all. Does it create a separate connection manager per
> > virtual user or per request? In any case this explains why HttpJava
> > tends to be faster in your tests. HttpJava uses one global connection
> > pool for all HttpUrlConnection instances. Therefore it uses fewer
> > physical connections to execute the same number of requests which
> > usually is always faster. This would be equivalent to HttpClient using
> > one pooling connection manager for all virtual users and requests.
> >
> > Hope this helps interpret the numbers.
> >
> > Oleg
> >
> > > The TestPlan from the wiki:
> > >
> > http://wiki.apache.org/jmeter/JMeterPerformance?action=AttachFile&do=view&target=TestPlan.jmx
> > >
> > > Run from JMeter 2.7RC3:
> > > http://people.apache.org/~milamber/jmeter-2.7RC3/dist/
> > >
> > > Run from JMeter trung with HC4.2 (temporary)
> > > http://people.apache.org/~milamber/temp/
> > >
> > > Milamber
> > >
> > >
> > >
> > > [1]
> > > http://www.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES.txt
> > > [2]
> > > http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES.txt
> > >
> > >
> > >
> > >
> > >
> > > > Oleg
> > > >
> > > >
> > > >
> > > > > > Le 26/05/2012 02:00, Philippe Mouawad a ecrit :
> > > > > >
> > > > > >
> > > > > > > Hello,
> > > > > > > I finally managed to insert images and Script:
> > > > > > > http://wiki.apache.org/jmeter/JMeterPerformance
> > > > > > >
> > > > > > > Regards
> > > > > > > Philippe
> > > > > > > On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad <
> > > > > > > philippe.mouawad@gmail.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Hello,
> > > > > > > > And are you able to insert images ? or Test Plan ?
> > > > > > > >
> > > > > > > > Sebb, I think I tried with Attachment to insert Plan
or Image,
> > it proposes
> > > > > > > > a link type with a protocol, but I don't see how to
upload the
> > image or
> > > > > > > > Test Plan.
> > > > > > > >
> > > > > > > > There is a button Insert/Edit image but it is disabled.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Philippe
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 25, 2012 at 12:06 PM, Milamber <
> > milamber@apache.org> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > On
> 
> 
> 



Mime
View raw message