jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milamber <milam...@apache.org>
Subject Re: JMeter Performance evolution : My 2 cents
Date Sun, 27 May 2012 15:47:04 GMT


Le 27/05/2012 14:57, Oleg Kalnichevski a ecrit :
> On Sat, 2012-05-26 at 23:14 +0100, Milamber wrote:
>   
>> Le 26/05/2012 21:25, Milamber a ecrit :
>>     
>>> Hello,
>>>
>>> Thanks for the wiki page.
>>>
>>> I've got the test plan, and I run some tests.
>>>
>>> Target:
>>> 1 VM on KVM host | 2 Gb RAM | 4 vCPU
>>> Debian 6.0.5 (64 bits)
>>> Tomcat 6.0.35 (from Apache site)
>>> Sun Java 1.6u32 (64bits)
>>> Restart Tomcat before each test
>>> Tomcat with default params.
>>>
>>> JMeter machine (another machine) :
>>> 8 Gb RAM | 2 core 2.53GHz
>>> Debian 6.0.5  (64 bits)
>>> Sun Java 1.6u32 (64bits)
>>> JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt
>>>
>>> With the GC log, I've make some stats.
>>>
>>> With the original test plan: HttpClient 4:
>>> (==>4.1)
>>> JMeter_251    Nb GC: 202    Full GC: 17    Time GC: 13.9376210    Full:
>>> 11.3386420    Total: 25.2762630
>>> JMeter_26    Nb GC: 216    Full GC: 18    Time GC: 14.6083630    Full:
>>> 12.0604050    Total: 26.6687680
>>> JMeter_27    Nb GC: 199    Full GC: 15    Time GC: 11.7238800    Full:
>>> 9.5478390    Total: 21.2717190
>>>
>>> Stats results HC4: (==>4.1)
>>> JMeter 2.5.1: Requests 163917 in 604.8s =  271.0/s Avg:   148
>>> JMeter 2.6: Requests 163208 in 605.0s =  269.7/s Avg:   155
>>> JMeter 2.7RC3: 164035 in 605.1s =  271.1/s Avg:   159
>>>   
>>>       
>> With JMeter trunk and HttpClient 4.2 (core, client, mime):
>>
>> JMeter_trunk_hc42    Nb GC: 131    Full GC: 1    Time GC: 5.7903490   
>> Full: .0285440    Total: 5.8188930
>>
>> JMeter trunk with HC 4.2 (not commit): Nb Requests 164762 in 605.2s = 
>> 272.3/s Avg:   250
>>
>>
>> Results (GC counts and avg req/s) with the HC 4.2 are same with HC3.1
>> and Java.
>>
>> => HC 4.2 seems fixed a performance issue with JVM memory?
>>
>> => Release 2.7.1 to do with HC4.2 ? (or update HC and make a RC4?)
>>
>> Milamber
>>
>>
>>     
>>> Changes HttpClient to 3.1:
>>> JMeter_251    Nb GC: 136    Full GC: 1    Time GC: 8.6780490    Full:
>>> .0254120    Total: 8.7034610
>>> JMeter_26    Nb GC: 146    Full GC: 1    Time GC: 9.0567930    Full:
>>> .0305490    Total: 9.0873420
>>> JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.9212530    Full:
>>> .0282870    Total: 6.9495400
>>>
>>> Stats results HC3.1:
>>> JMeter 2.5.1: Requests 165103 in 605.1s =  272.9/s Avg:   233
>>> JMeter 2.6: Requests 165063 in 605.1s =  272.8/s Avg:   214
>>> JMeter 2.7RC3: Requests 164860 in 604.5s =  272.7/s Avg:   180
>>>
>>> Changes to Java http client:
>>> JMeter_251    Nb GC: 138    Full GC: 1    Time GC: 8.1128620    Full:
>>> .0253190    Total: 8.1381810
>>> JMeter_26    Nb GC: 149    Full GC: 2    Time GC: 8.4811800    Full:
>>> .6600440    Total: 9.1412240
>>> JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.4912220    Full:
>>> .0283680    Total: 6.5195900
>>>
>>> Stats results HJ:
>>> JMeter 2.5.1: Requests 164966 in 605.0s =  272.7/s Avg:   164
>>> JMeter 2.6: Requests 165069 in 605.2s =  272.8/s Avg:   198
>>> JMeter 2.7RC3: Requests 165233 in 605.0s =  273.1/s Avg:   213
>>>
>>>
>>> * JMeter 2.7(RC3) and HttpJava requests seems the best with the GC
>>> (HC3.1 too)
>>> * HttpJava and HC 3.1 seems the best with GC and req/s
>>> * HC 4 seems have the best results with the time average
>>> * HC 4 seems bring some (big) differences with the java virtual machine.
>>>
>>> An investigation on HC4 can be done? perhaps send an email on
>>> HttpComponents team.
>>>
>>> Milamber
>>>
>>>
>>>       
> I cannot think of any changes in the 4.2 branch that could make a big
> difference in terms of performance or memory footprint compared to 4.1.
> It also seems somewhat strange that the 4.1 based sampler appears to be
> generating massively more intermediate garbage while still managing to
> perform at the same level as other samplers (unless I am misreading the
> results somehow). 
>
> Do you know what kind of objects cause excessive garbage collection when
> running the 4.1 sampler? 
>   

Hello Oleg,

I've make some heap dump during tests (with oracle jvisualvm)
I uses Eclipse Memory Analyzer to analyze the heap.
I've found perhaps a "org.apache.http.impl.conn.SingleClientConnManager"
issue.

With JMeter 2.7RC3 (HttpClient 4.1) :


          Biggest Top-Level Dominator Classes


Label 	Number of Objects 	Used Heap Size 	Retained Heap Size 	Retained
Heap, %
org.apache.jmeter.protocol.http.control.HeaderManager
<mat://object/0xd87ebd20> 	6,000 	384,000 	40,142,160 	18.26%
org.apache.http.impl.conn.SingleClientConnManager
<mat://object/0xd8bd20a0> 	1,500 	120,000 	33,289,328 	15.14%
org.apache.jmeter.threads.JMeterThread <mat://object/0xd89df388>
1,500 	216,000 	31,718,400 	14.43%
org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy
<mat://object/0xd87d8d48> 	6,004 	384,256 	30,097,176 	13.69%
java.lang.Thread <mat://object/0xd8049cf0> 	1,510 	265,760 	18,165,056
8.26%
org.apache.jmeter.assertions.ResponseAssertion
<mat://object/0xd87f7138> 	6,000 	336,000 	9,504,000 	4.32%
org.apache.jmeter.threads.JMeterContext <mat://object/0xd89a2280>
1,500 	120,000 	8,230,528 	3.74%
org.apache.jmeter.sampler.TestAction <mat://object/0xd8801e78> 	6,000
336,000 	7,680,000 	3.49%
org.apache.jmeter.threads.ThreadGroup <mat://object/0xd87c21f8> 	1,501
96,064 	5,379,528 	2.45%
org.apache.jmeter.config.ConfigTestElement <mat://object/0xd87bb898>
1,500 	84,000 	4,648,080 	2.11%
org.apache.jmeter.protocol.http.control.CookieManager
<mat://object/0xd87d0978> 	1,501 	108,072 	4,296,984 	1.95%
java.lang.Class <mat://object/0xd8022528> 	1,759 	27,840 	4,177,528 	1.90%
org.apache.http.conn.scheme.SchemeRegistry <mat://object/0xd8a7d940>
1,500 	36,000 	3,725,520 	1.69%
org.apache.jmeter.config.Arguments <mat://object/0xd87bc000> 	1,500
84,000 	3,180,000 	1.45%
org.apache.jmeter.control.TransactionController
<mat://object/0xd87d5930> 	1,500 	204,000 	2,832,000 	1.29%

    * Total: 15 entries

	40,775 	2,801,992 	207,066,288 	




With JMeter Trunk + HttpClient 4.2 :


          Biggest Top-Level Dominator Classes

Label 	Number of Objects 	Used Heap Size 	Retained Heap Size 	Retained
Heap, %
org.apache.jmeter.protocol.http.control.HeaderManager
<mat://object/0xd87ebb08> 	6,000 	384,000 	41,283,936 	22.75%
org.apache.jmeter.threads.JMeterThread <mat://object/0xd89e0d98>
1,500 	216,000 	31,925,472 	17.59%
org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy
<mat://object/0xd87d8878> 	6,004 	384,256 	28,151,656 	15.51%
java.lang.Thread <mat://object/0xd8049cf0> 	1,504 	264,704 	15,899,168
8.76%
org.apache.jmeter.assertions.ResponseAssertion
<mat://object/0xd87f6f70> 	6,000 	336,000 	9,441,672 	5.20%
org.apache.jmeter.sampler.TestAction <mat://object/0xd8801d10> 	6,000
336,000 	7,601,096 	4.19%
org.apache.jmeter.threads.JMeterContext <mat://object/0xd89a39e8>
1,332 	106,560 	7,156,624 	3.94%
org.apache.jmeter.threads.ThreadGroup <mat://object/0xd87c19b0> 	1,501
96,064 	5,353,896 	2.95%
org.apache.jmeter.config.ConfigTestElement <mat://object/0xd87baff8>
1,500 	84,000 	5,141,184 	2.83%
org.apache.jmeter.protocol.http.control.CookieManager
<mat://object/0xd87d02c8> 	1,501 	108,072 	4,582,536 	2.52%
org.apache.http.impl.conn.BasicClientConnectionManager
<mat://object/0xd8be05c8> 	1,332 	85,248 	4,256,488 	2.35%
java.lang.Class <mat://object/0xd8022528> 	1,228 	22,200 	4,006,160 	2.21%
org.apache.http.conn.scheme.SchemeRegistry <mat://object/0xd8a8a118>
1,332 	31,968 	3,314,016 	1.83%
org.apache.jmeter.config.Arguments <mat://object/0xd87bb788> 	1,500
84,000 	3,161,184 	1.74%
org.apache.jmeter.control.TransactionController
<mat://object/0xd87d5310> 	1,500 	204,000 	2,737,920 	1.51%
org.apache.jmeter.control.LoopController <mat://object/0xd87c8010>
1,500 	144,000 	2,150,592 	1.18%

    * Total: 16 entries

	41,234 	2,887,072 	176,163,600 	





When I observed in the HC4.1/JMeter, with outgoing reference for
SingleClientConnManager :
*connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl*

Class
Name                                                                                                
| Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------
class org.apache.http.impl.conn.SingleClientConnManager @
0xd8bd20a0                                       |            8
|           336
|- <class> org.apache.http.impl.conn.SingleClientConnManager @
0xe6be9450                                  |           80 |        27,008
|  |- *connManager
org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4* @
0xe6be9320                       |          168 |         9,536
|  |  |- value java.util.HashMap$Entry @
0xe6be9258                                                       
|           48 |         9,784
|  |  |  '- [10] java.util.HashMap$Entry[16] @
0xe6be9208                                                  |         
152 |         9,936
|  |  |     '- table java.util.HashMap @
0xe6be91d8                                                       
|           64 |        10,000
|  |  |        '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @
0xe6be91b8                            |           56 |        10,056
|  |  |           '- [4] java.lang.ThreadLocal$ThreadLocalMap$Entry[16]
@ 0xe3fed728                       |          152 |        11,064
|  |  |              '- table java.lang.ThreadLocal$ThreadLocalMap @
0xe3fed710                            |           32 |        11,096
|  |  |                 '- threadLocals java.lang.Thread @ 0xe3fe4b00 
Thread Group 1-1483 Thread          |          176 |        11,592
|  |  |                    |- [1486] java.lang.Thread[2048] @
0xe3218f40                                   |       16,408 |        16,408
|  |  |                    |  '- threads java.lang.ThreadGroup @
0xe0000030  main                          |           72 |        16,552
|  |  |                    |- <Java Local>, me java.lang.Thread @
0xe3fe4b00  Thread Group 1-1483 Thread   |          176 |        11,592
|  |  |                    |  |- [1486] java.lang.Thread[2048] @
0xe3218f40                                |       16,408 |        16,408
|  |  |                    |  |- <Java Local>, me java.lang.Thread @
0xe3fe4b00  Thread Group 1-1483 Thread|          176 |        11,592
|  |  |                    |  |- value
java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0      
|           48 |            48
|  |  |                    |  '- Total: 3
entries                                                         
|              |             
|  |  |                    |- value
java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0         
|           48 |            48
|  |  |                    '- Total: 3
entries                                                            
|              |             
|  |  |- this$1 org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4$1
@ 0xe6bee448                       |           32 |            32
|  |  '- Total: 2
entries                                                                                 
|              |             
|  |- referent java.lang.ref.Finalizer @
0xe6bf2eb0                                                       
|           64 |       113,152
|  |- this$0 org.apache.http.impl.conn.SingleClientConnManager$PoolEntry
@ 0xe844eee0                      |           64 |        26,896
|  '- Total: 3
entries                                                                                    
|              |             
|- <class> org.apache.http.impl.conn.SingleClientConnManager @
0xe6be6d88                                  |           80 |        27,008
|- <class> org.apache.http.impl.conn.SingleClientConnManager @
0xe6be40c8                                  |           80 |        27,008
|- <class> org.apache.http.impl.conn.SingleClientConnManager @
0xe6be19c8                                  |           80 |            80
|- <class> org.apache.http.impl.conn.SingleClientConnManager @
0xe6bde940                                  |           80 |        27,008
|- <class> org.apache.http.impl.conn.SingleClientConnManager @
0xe6bdc268                                  |           80 |        26,984
------------------------------------------------------------------------------------------------------------------------------------------
Class Name                | Shallow Heap | Retained Heap
---------------------------------------------------------
Total: 25 of 1,501 entries|              |             
---------------------------------------------------------

Not observed with JMeter trunk with HC4.2.

Perhaps, the new implementation [1] [2] of ConnectionManager with HC42
is better ?


The TestPlan from the wiki:
http://wiki.apache.org/jmeter/JMeterPerformance?action=AttachFile&do=view&target=TestPlan.jmx

Run from JMeter 2.7RC3:
http://people.apache.org/~milamber/jmeter-2.7RC3/dist/

Run from JMeter trung with HC4.2 (temporary)
http://people.apache.org/~milamber/temp/

Milamber



[1] http://www.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES.txt
[2] http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES.txt





> Oleg
>
>
>   
>>> Le 26/05/2012 02:00, Philippe Mouawad a ecrit :
>>>   
>>>       
>>>> Hello,
>>>> I finally managed to insert images and Script:
>>>> http://wiki.apache.org/jmeter/JMeterPerformance
>>>>
>>>> Regards
>>>> Philippe
>>>> On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad <
>>>> philippe.mouawad@gmail.com> wrote:
>>>>
>>>>   
>>>>     
>>>>         
>>>>> Hello,
>>>>> And are you able to insert images ? or Test Plan ?
>>>>>
>>>>> Sebb, I think I tried with Attachment to insert Plan or Image, it proposes
>>>>> a link type with a protocol, but I don't see how to upload the image or
>>>>> Test Plan.
>>>>>
>>>>> There is a button Insert/Edit image but it is disabled.
>>>>>
>>>>> Regards
>>>>> Philippe
>>>>>
>>>>>
>>>>> On Fri, May 25, 2012 at 12:06 PM, Milamber <milamber@apache.org> wrote:
>>>>>
>>>>>     
>>>>>       
>>>>>           
>>>>>> On Fri, May 25, 2012 at 10:35 AM, sebb <sebbaz@gmail.com> wrote:
>>>>>>
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> On 25 May 2012 10:31, Milamber <milamber@apache.org> wrote:
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Actuallly, the page
>>>>>>>>           
>>>>>>>>             
>>>>>>>>                 
>>>>>> http://wiki.apache.org/jmeter/JMeterPerformancereturns
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>> a http 500 error (others pages works)
>>>>>>>>           
>>>>>>>>             
>>>>>>>>                 
>>>>>>> Works for me at present.
>>>>>>>
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>> Ok for me too.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>> Milamber
>>>>>>>>
>>>>>>>> ===
>>>>>>>> Internal Server Error
>>>>>>>>
>>>>>>>> The server encountered an internal error or misconfiguration and was
>>>>>>>>           
>>>>>>>>             
>>>>>>>>                 
>>>>>>> unable
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>> to complete your request.
>>>>>>>>
>>>>>>>> Please contact the server administrator at infrastructure@apache.orgto
>>>>>>>> inform them of the time this error occurred, and the actions you
>>>>>>>>           
>>>>>>>>             
>>>>>>>>                 
>>>>>>> performed
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>> just before this error.
>>>>>>>>
>>>>>>>> More information about this error may be available in the server error
>>>>>>>>           
>>>>>>>>             
>>>>>>>>                 
>>>>>>> log.
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>> ------------------------------
>>>>>>>> Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org Port 80
>>>>>>>>
>>>>>>>> ===
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, May 25, 2012 at 1:27 AM, sebb <sebbaz@gmail.com> wrote:
>>>>>>>>
>>>>>>>>           
>>>>>>>>             
>>>>>>>>                 
>>>>>>>>> On 24 May 2012 22:15, Philippe Mouawad <philippe.mouawad@gmail.com>
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>> wrote:
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>> Hello,
>>>>>>>>>> How can I insert image ? button is disabled in GUI mode.
>>>>>>>>>> To insert Test Plan, is it an attachment ?
>>>>>>>>>>               
>>>>>>>>>>                 
>>>>>>>>>>                     
>>>>>>>>> Try clicking "Attachments" before starting the edit.
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>> Thanks
>>>>>>>>>> Regards
>>>>>>>>>> Philippe
>>>>>>>>>>
>>>>>>>>>> On Wed, May 23, 2012 at 12:22 AM, sebb <sebbaz@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>               
>>>>>>>>>>                 
>>>>>>>>>>                     
>>>>>>>>>>> On 22 May 2012 20:18, Philippe Mouawad <
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>> philippe.mouawad@gmail.com>
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> wrote:
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>> Hello Milamber,
>>>>>>>>>>>> My responses below.
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Philippe
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, May 21, 2012 at 11:51 PM, Milamber <milamber@apache.org
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>> wrote:
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>>>>>>> Le 21/05/2012 22:12, Philippe Mouawad a ecrit :
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> I read recently this little comparison :
>>>>>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmarks<
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmark>
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>> I reviewed the test plan that was used to make the test.
>>>>>>>>>>>>>> It seems to me the test is little biased:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - View Results Tree is in test plan (as it uses a lot of
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> memory,
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> it's
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>> a
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>    big issue)
>>>>>>>>>>>>>>    - View Results in Table (same thing)
>>>>>>>>>>>>>>    - 3 Non Standard JMeter listener, so It's not pure JMeter:
>>>>>>>>>>>>>>       - jp@gc - Transactions per Second
>>>>>>>>>>>>>>       - jp@gc - Response Times Over Time
>>>>>>>>>>>>>>       - jp@gc - Active Threads Over Time
>>>>>>>>>>>>>>    - Default XML output seems to have been used , it's
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> against
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> best
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>>>>>    practices
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Besides the following conclusions that seem to me non
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> scientific
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> and
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>> purely
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>> subjective:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - The testers, new to both Gatling and JMeter found that
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> JMeter
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>> was
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>>>    harder to learn and use than Gatling to create the
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> simulations,
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>>>> despite the
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>    use of a proxy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will only look at other ideas expressed:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - JMeter creates one thread per user simulated. If there
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> is
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> not
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> enough
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>    memory allocated to the JVM, it can crash trying to create
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> these
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>>>> threads
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>       - This need to be detailed, cause either it fails with
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> OOM
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> and
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>> it's
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>       not during thread creation, either it fails with
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> "unable
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> to
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> create
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>> new
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>       native thread"
>>>>>>>>>>>>>>    - For instance, JMeter could not run 1500 users with 512
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> MB
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> (what
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>> was
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>    used for Gatling even with 2000 users); OutOfMemoryErrors
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> are
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>>>> recorded in
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>    the table as *OOM*
>>>>>>>>>>>>>>       - => I made a Test with up to 2000 Threads with 512 m
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> without
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> any
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>       crash, it depends on Test and on application
>>>>>>>>>>>>>>    - Another problem occurred with the 2000 users
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> simulations;
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> it
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> seems
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>    that JMeter can not simulate more than 1514 users
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> independently
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> from
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>> the
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>    memory that was allocated to the JVM
>>>>>>>>>>>>>>       - => I made a Test with up to 2000 Threads with 512 m
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> without
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> any
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>       crash, so assertion is false, it depends on Test and on
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>>>> application
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> As it's difficult to install the application used for test
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> (last
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>> version
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> does not seem to work as expected) , if they provide a
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> working
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> WAR
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>>>> against
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>> a local Postgres DB I will be happy to test with it.
>>>>>>>>>>>>>> But in current state, application seems to be packaged for
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> cloud
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> or H2
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>>> local DB, I didn't want to spend too much time setting up
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>> application
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>> as
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>> I
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>> don't know its real status.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just tried to run Test Plan against a blank tomcat to
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> verify
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> what
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> they
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> say about Thread Creation, I didn't find any issue on this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I decided to make a very simple scenario test on Tomcat
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> Examples
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> (It
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> goes to Session Example, adds attribute, go back to index, go
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> back
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>> to
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>>> Session Example, test contains Response assertion for each
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>> Request).
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>>>>>> Please, indicate the Tomcat version used? it's the same
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>> machine?
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> config
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>> of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>> editor?
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>> Tomcat apache-tomcat-6.0.24
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>> -Xms256m -Xmx1024m
>>>>>>>>>>>>    <Connector port="8080" protocol="HTTP/1.1"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>> compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json"
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>>>                compression="off"
>>>>>>>>>>>>                socketBuffer="8"
>>>>>>>>>>>>                maxThreads="400"
>>>>>>>>>>>>               connectionTimeout="20000"
>>>>>>>>>>>>               redirectPort="8443" />
>>>>>>>>>>>> Set session timeout in web.xml  to 1min
>>>>>>>>>>>> java version "1.6.0_29"
>>>>>>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
>>>>>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>> mode)
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>>> Mac OS 10.6.8
>>>>>>>>>>>> JMeter  and Tomcat on same machine
>>>>>>>>>>>> No particular OS Tuning
>>>>>>>>>>>> 8GO RAM
>>>>>>>>>>>> No swap, nothing runnning except these 2
>>>>>>>>>>>> Tomcat CPU around 5%
>>>>>>>>>>>> Memory around 50 mo
>>>>>>>>>>>>
>>>>>>>>>>>> JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)?
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>>>>>>> Same config
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>> No TCP tuning
>>>>>>>>>>>>
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>>>>>>> Can you post the jmx file?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Will it be accepted on this list ?
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>> Best to create a Wiki page and attach the test file there.
>>>>>>>>>>>
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>>>>>>> Milamber
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>> It is not at all representative but it is a way for me to
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> check
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>> potential
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> issues in JMeter and performance changes accross versions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I ran the test with 1500 VU using JMeter 2.5.1,  2.7 (current
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>> trunk)
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>> with
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> -Xmx512m, 10 minutes run and CSV output against a local
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> Tomcat
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>> (I
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>> restard
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>> tomcat between tests and control its health):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - I noticed that current trunk version behaves much
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> better in
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>> terms of
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>    memory than 2.5.1 or 2.6:
>>>>>>>>>>>>>>       - In 2.5.1 :
>>>>>>>>>>>>>>       - GC activity is much higher with around 5 GC CPU peaks
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>> every 2
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>>>          minutes,  and 20 FULL GC of 700 to 800 ms each
>>>>>>>>>>>>>>          - Throughput: 97,71%
>>>>>>>>>>>>>>          - Pauses : 13,69s
>>>>>>>>>>>>>>          - Mém : 391M/min
>>>>>>>>>>>>>>          - Full GC tend to be much more frequent at end of
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> test
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>>>>>          - 2.7:
>>>>>>>>>>>>>>          -  no GC CPU peak, 1 FULL GC
>>>>>>>>>>>>>>          - Throughput:98.54
>>>>>>>>>>>>>>          - Pauses : 8.9s
>>>>>>>>>>>>>>          - 1108m /min
>>>>>>>>>>>>>>          - Summary:
>>>>>>>>>>>>>>       - 25.1:
>>>>>>>>>>>>>>          - 164676 samples in 605,1s
>>>>>>>>>>>>>>          - 272,2/s
>>>>>>>>>>>>>>          - Avg:    97
>>>>>>>>>>>>>>          - 2.7:
>>>>>>>>>>>>>>          - 165367 in 605.0s
>>>>>>>>>>>>>>          - 273.3/s
>>>>>>>>>>>>>>          - Avg:   228
>>>>>>>>>>>>>>          - I also noticed results have a much better look:
>>>>>>>>>>>>>>       - in 2.5.1, Transactions/s are around 300 / sec during
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> 4
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>>>> minutes,
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>>>>>>       then drop to 200/s, go up to 400/s , then down to 260/s
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> and
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>>>>>>> finally 200/sec
>>>>>>>>>>>>>>       - in 2.7, Transactions stay around 300/sec
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Actions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - I think it would be useful to have some reference
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>> application
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>>>> on
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>> which
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>    we could test JMeter behaviour
>>>>>>>>>>>>>>    - What would the best place to put these comparisons ?
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> wiki ?
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> which
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>>>    indicators should we put ?
>>>>>>>>>>>>>>    - Further testing should be done against a richer
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>> application
>>>>>>       
>>>>>>         
>>>>>>             
>>>>>>>>> that
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>>>>>>> could
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>    be deployed locally
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Philippe
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>                             
>>>>>>>>>>>>>                     
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>                           
>>>>>>>>>>>> --
>>>>>>>>>>>> Cordialement.
>>>>>>>>>>>> Philippe Mouawad.
>>>>>>>>>>>>                   
>>>>>>>>>>>>                     
>>>>>>>>>>>>                         
>>>>>>>>>>>                 
>>>>>>>>>>>                   
>>>>>>>>>>>                       
>>>>>>>>>> --
>>>>>>>>>> Cordialement.
>>>>>>>>>> Philippe Mouawad.
>>>>>>>>>>               
>>>>>>>>>>                 
>>>>>>>>>>                     
>>>>>>>>>             
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>         
>>>>>>>           
>>>>>>>               
>>>>>>       
>>>>>>         
>>>>>>             
>>>>> --
>>>>> Cordialement.
>>>>> Philippe Mouawad.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>       
>>>>>           
>>>>   
>>>>     
>>>>         
>>>   
>>>       
>>     
>
>
>   


Mime
View raw message