jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: JMeter Performance evolution : My 2 cents
Date Sun, 27 May 2012 18:11:59 GMT
On Sun, 2012-05-27 at 16:47 +0100, Milamber wrote:
> 
> 
> Le 27/05/2012 14:57, Oleg Kalnichevski a ecrit : 
> > On Sat, 2012-05-26 at 23:14 +0100, Milamber wrote:
> >   
> > > Le 26/05/2012 21:25, Milamber a ecrit :
> > >     
> > > > Hello,
> > > > 
> > > > Thanks for the wiki page.
> > > > 
> > > > I've got the test plan, and I run some tests.
> > > > 
> > > > Target:
> > > > 1 VM on KVM host | 2 Gb RAM | 4 vCPU
> > > > Debian 6.0.5 (64 bits)
> > > > Tomcat 6.0.35 (from Apache site)
> > > > Sun Java 1.6u32 (64bits)
> > > > Restart Tomcat before each test
> > > > Tomcat with default params.
> > > > 
> > > > JMeter machine (another machine) :
> > > > 8 Gb RAM | 2 core 2.53GHz
> > > > Debian 6.0.5  (64 bits)
> > > > Sun Java 1.6u32 (64bits)
> > > > JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt
> > > > 
> > > > With the GC log, I've make some stats.
> > > > 
> > > > With the original test plan: HttpClient 4:
> > > > (==>4.1)
> > > > JMeter_251    Nb GC: 202    Full GC: 17    Time GC: 13.9376210    Full:
> > > > 11.3386420    Total: 25.2762630
> > > > JMeter_26    Nb GC: 216    Full GC: 18    Time GC: 14.6083630    Full:
> > > > 12.0604050    Total: 26.6687680
> > > > JMeter_27    Nb GC: 199    Full GC: 15    Time GC: 11.7238800    Full:
> > > > 9.5478390    Total: 21.2717190
> > > > 
> > > > Stats results HC4: (==>4.1)
> > > > JMeter 2.5.1: Requests 163917 in 604.8s =  271.0/s Avg:   148
> > > > JMeter 2.6: Requests 163208 in 605.0s =  269.7/s Avg:   155
> > > > JMeter 2.7RC3: 164035 in 605.1s =  271.1/s Avg:   159
> > > >   
> > > >       
> > > With JMeter trunk and HttpClient 4.2 (core, client, mime):
> > > 
> > > JMeter_trunk_hc42    Nb GC: 131    Full GC: 1    Time GC: 5.7903490   
> > > Full: .0285440    Total: 5.8188930
> > > 
> > > JMeter trunk with HC 4.2 (not commit): Nb Requests 164762 in 605.2s = 
> > > 272.3/s Avg:   250
> > > 
> > > 
> > > Results (GC counts and avg req/s) with the HC 4.2 are same with HC3.1
> > > and Java.
> > > 
> > > => HC 4.2 seems fixed a performance issue with JVM memory?
> > > 
> > > => Release 2.7.1 to do with HC4.2 ? (or update HC and make a RC4?)
> > > 
> > > Milamber
> > > 
> > > 
> > >     
> > > > Changes HttpClient to 3.1:
> > > > JMeter_251    Nb GC: 136    Full GC: 1    Time GC: 8.6780490    Full:
> > > > .0254120    Total: 8.7034610
> > > > JMeter_26    Nb GC: 146    Full GC: 1    Time GC: 9.0567930    Full:
> > > > .0305490    Total: 9.0873420
> > > > JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.9212530    Full:
> > > > .0282870    Total: 6.9495400
> > > > 
> > > > Stats results HC3.1:
> > > > JMeter 2.5.1: Requests 165103 in 605.1s =  272.9/s Avg:   233
> > > > JMeter 2.6: Requests 165063 in 605.1s =  272.8/s Avg:   214
> > > > JMeter 2.7RC3: Requests 164860 in 604.5s =  272.7/s Avg:   180
> > > > 
> > > > Changes to Java http client:
> > > > JMeter_251    Nb GC: 138    Full GC: 1    Time GC: 8.1128620    Full:
> > > > .0253190    Total: 8.1381810
> > > > JMeter_26    Nb GC: 149    Full GC: 2    Time GC: 8.4811800    Full:
> > > > .6600440    Total: 9.1412240
> > > > JMeter_27    Nb GC: 137    Full GC: 1    Time GC: 6.4912220    Full:
> > > > .0283680    Total: 6.5195900
> > > > 
> > > > Stats results HJ:
> > > > JMeter 2.5.1: Requests 164966 in 605.0s =  272.7/s Avg:   164
> > > > JMeter 2.6: Requests 165069 in 605.2s =  272.8/s Avg:   198
> > > > JMeter 2.7RC3: Requests 165233 in 605.0s =  273.1/s Avg:   213
> > > > 
> > > > 
> > > > * JMeter 2.7(RC3) and HttpJava requests seems the best with the GC
> > > > (HC3.1 too)
> > > > * HttpJava and HC 3.1 seems the best with GC and req/s
> > > > * HC 4 seems have the best results with the time average
> > > > * HC 4 seems bring some (big) differences with the java virtual machine.
> > > > 
> > > > An investigation on HC4 can be done? perhaps send an email on
> > > > HttpComponents team.
> > > > 
> > > > Milamber
> > > > 
> > > > 
> > > >       
> > I cannot think of any changes in the 4.2 branch that could make a big
> > difference in terms of performance or memory footprint compared to 4.1.
> > It also seems somewhat strange that the 4.1 based sampler appears to be
> > generating massively more intermediate garbage while still managing to
> > perform at the same level as other samplers (unless I am misreading the
> > results somehow). 
> > 
> > Do you know what kind of objects cause excessive garbage collection when
> > running the 4.1 sampler? 
> >   
> 
> Hello Oleg,
> 
> I've make some heap dump during tests (with oracle jvisualvm)
> I uses Eclipse Memory Analyzer to analyze the heap.
> I've found perhaps a
> "org.apache.http.impl.conn.SingleClientConnManager" issue.
> 
> With JMeter 2.7RC3 (HttpClient 4.1) :
> 
>  Biggest Top-Level Dominator Classes
> 
>     Label
>   Number of
>    Objects
> Used Heap Size
> Retained Heap
>      Size
> Retained Heap,
>       %
> org.apache.jmeter.protocol.http.control.HeaderManager
>          6,000
>        384,000
>     40,142,160
>         18.26%
> org.apache.http.impl.conn.SingleClientConnManager
>          1,500
>        120,000
>     33,289,328
>         15.14%
> org.apache.jmeter.threads.JMeterThread
>          1,500
>        216,000
>     31,718,400
>         14.43%
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy
>          6,004
>        384,256
>     30,097,176
>         13.69%
> java.lang.Thread
>          1,510
>        265,760
>     18,165,056
>          8.26%
> org.apache.jmeter.assertions.ResponseAssertion
>          6,000
>        336,000
>      9,504,000
>          4.32%
> org.apache.jmeter.threads.JMeterContext
>          1,500
>        120,000
>      8,230,528
>          3.74%
> org.apache.jmeter.sampler.TestAction
>          6,000
>        336,000
>      7,680,000
>          3.49%
> org.apache.jmeter.threads.ThreadGroup
>          1,501
>         96,064
>      5,379,528
>          2.45%
> org.apache.jmeter.config.ConfigTestElement
>          1,500
>         84,000
>      4,648,080
>          2.11%
> org.apache.jmeter.protocol.http.control.CookieManager
>          1,501
>        108,072
>      4,296,984
>          1.95%
> java.lang.Class
>          1,759
>         27,840
>      4,177,528
>          1.90%
> org.apache.http.conn.scheme.SchemeRegistry
>          1,500
>         36,000
>      3,725,520
>          1.69%
> org.apache.jmeter.config.Arguments
>          1,500
>         84,000
>      3,180,000
>          1.45%
> org.apache.jmeter.control.TransactionController
>          1,500
>        204,000
>      2,832,000
>          1.29%
> 
>       * Total:
>         15
>         entries
>         40,775
>      2,801,992
>    207,066,288
> 
> 
> 
> 
> 
> With JMeter Trunk + HttpClient 4.2 :
> 
> 
>  Biggest Top-Level Dominator Classes
>     Label
>   Number of
>    Objects
> Used Heap Size
> Retained Heap
>      Size
> Retained Heap,
>       %
> org.apache.jmeter.protocol.http.control.HeaderManager
>          6,000
>        384,000
>     41,283,936
>         22.75%
> org.apache.jmeter.threads.JMeterThread
>          1,500
>        216,000
>     31,925,472
>         17.59%
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy
>          6,004
>        384,256
>     28,151,656
>         15.51%
> java.lang.Thread
>          1,504
>        264,704
>     15,899,168
>          8.76%
> org.apache.jmeter.assertions.ResponseAssertion
>          6,000
>        336,000
>      9,441,672
>          5.20%
> org.apache.jmeter.sampler.TestAction
>          6,000
>        336,000
>      7,601,096
>          4.19%
> org.apache.jmeter.threads.JMeterContext
>          1,332
>        106,560
>      7,156,624
>          3.94%
> org.apache.jmeter.threads.ThreadGroup
>          1,501
>         96,064
>      5,353,896
>          2.95%
> org.apache.jmeter.config.ConfigTestElement
>          1,500
>         84,000
>      5,141,184
>          2.83%
> org.apache.jmeter.protocol.http.control.CookieManager
>          1,501
>        108,072
>      4,582,536
>          2.52%
> org.apache.http.impl.conn.BasicClientConnectionManager
>          1,332
>         85,248
>      4,256,488
>          2.35%
> java.lang.Class
>          1,228
>         22,200
>      4,006,160
>          2.21%
> org.apache.http.conn.scheme.SchemeRegistry
>          1,332
>         31,968
>      3,314,016
>          1.83%
> org.apache.jmeter.config.Arguments
>          1,500
>         84,000
>      3,161,184
>          1.74%
> org.apache.jmeter.control.TransactionController
>          1,500
>        204,000
>      2,737,920
>          1.51%
> org.apache.jmeter.control.LoopController
>          1,500
>        144,000
>      2,150,592
>          1.18%
> 
>       * Total:
>         16
>         entries
>         41,234
>      2,887,072
>    176,163,600
> 
> 
> 
> 
> 
> 
> When I observed in the HC4.1/JMeter, with outgoing reference for
> SingleClientConnManager :
> connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl
> 
> Class Name
> | Shallow Heap | Retained Heap
> ------------------------------------------------------------------------------------------------------------------------------------------
> class org.apache.http.impl.conn.SingleClientConnManager @ 0xd8bd20a0
> |            8 |           336
> |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> 0xe6be9450                                  |           80 |
> 27,008
> |  |- connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl
> $4 @ 0xe6be9320                       |          168 |         9,536
> |  |  |- value java.util.HashMap$Entry @ 0xe6be9258
> |           48 |         9,784
> |  |  |  '- [10] java.util.HashMap$Entry[16] @ 0xe6be9208
> |          152 |         9,936
> |  |  |     '- table java.util.HashMap @ 0xe6be91d8
> |           64 |        10,000
> |  |  |        '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @
> 0xe6be91b8                            |           56 |        10,056
> |  |  |           '- [4] java.lang.ThreadLocal$ThreadLocalMap
> $Entry[16] @ 0xe3fed728                       |          152 |
> 11,064
> |  |  |              '- table java.lang.ThreadLocal$ThreadLocalMap @
> 0xe3fed710                            |           32 |        11,096
> |  |  |                 '- threadLocals java.lang.Thread @ 0xe3fe4b00
> Thread Group 1-1483 Thread          |          176 |        11,592
> |  |  |                    |- [1486] java.lang.Thread[2048] @
> 0xe3218f40                                   |       16,408 |
> 16,408
> |  |  |                    |  '- threads java.lang.ThreadGroup @
> 0xe0000030  main                          |           72 |
> 16,552
> |  |  |                    |- <Java Local>, me java.lang.Thread @
> 0xe3fe4b00  Thread Group 1-1483 Thread   |          176 |
> 11,592
> |  |  |                    |  |- [1486] java.lang.Thread[2048] @
> 0xe3218f40                                |       16,408 |
> 16,408
> |  |  |                    |  |- <Java Local>, me java.lang.Thread @
> 0xe3fe4b00  Thread Group 1-1483 Thread|          176 |        11,592
> |  |  |                    |  |- value
> java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0       |
> 48 |            48
> |  |  |                    |  '- Total: 3 entries
> |              |              
> |  |  |                    |- value
> java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0
> |           48 |            48
> |  |  |                    '- Total: 3 entries
> |              |              
> |  |  |- this$1 org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4
> $1 @ 0xe6bee448                       |           32 |            32
> |  |  '- Total: 2 entries
> |              |              
> |  |- referent java.lang.ref.Finalizer @ 0xe6bf2eb0
> |           64 |       113,152
> |  |- this$0 org.apache.http.impl.conn.SingleClientConnManager
> $PoolEntry @ 0xe844eee0                      |           64 |
> 26,896
> |  '- Total: 3 entries
> |              |              
> |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> 0xe6be6d88                                  |           80 |
> 27,008
> |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> 0xe6be40c8                                  |           80 |
> 27,008
> |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> 0xe6be19c8                                  |           80 |
> 80
> |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> 0xe6bde940                                  |           80 |
> 27,008
> |- <class> org.apache.http.impl.conn.SingleClientConnManager @
> 0xe6bdc268                                  |           80 |
> 26,984
> ------------------------------------------------------------------------------------------------------------------------------------------
> Class Name                | Shallow Heap | Retained Heap
> ---------------------------------------------------------
> Total: 25 of 1,501 entries|              |              
> ---------------------------------------------------------
> 
> Not observed with JMeter trunk with HC4.2.
> 
> Perhaps, the new implementation [1] [2] of ConnectionManager with HC42
> is better ?
> 
> 

The new connection managers are definitely better in terms of code
quality. However, I did not expect them to be any faster, though, as
they pretty much shared the same design with the old ones. At the same
time it does appear that BasicClientConnectionManager tends to have a
much smaller footprint judging by the heap dumps, which is a pleasant
surprise.

What is not quite clear to me is why JMeter creates so many connection
managers at all. Does it create a separate connection manager per
virtual user or per request? In any case this explains why HttpJava
tends to be faster in your tests. HttpJava uses one global connection
pool for all HttpUrlConnection instances. Therefore it uses fewer
physical connections to execute the same number of requests which
usually is always faster. This would be equivalent to HttpClient using
one pooling connection manager for all virtual users and requests.

Hope this helps interpret the numbers. 

Oleg 

> The TestPlan from the wiki:
> http://wiki.apache.org/jmeter/JMeterPerformance?action=AttachFile&do=view&target=TestPlan.jmx
> 
> Run from JMeter 2.7RC3:
> http://people.apache.org/~milamber/jmeter-2.7RC3/dist/
> 
> Run from JMeter trung with HC4.2 (temporary)
> http://people.apache.org/~milamber/temp/ 
> 
> Milamber
> 
> 
> 
> [1]
> http://www.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES.txt
> [2]
> http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES.txt
> 
> 
> 
> 
> 
> > Oleg
> > 
> > 
> >   
> > > > Le 26/05/2012 02:00, Philippe Mouawad a ecrit :
> > > >   
> > > >       
> > > > > Hello,
> > > > > I finally managed to insert images and Script:
> > > > > http://wiki.apache.org/jmeter/JMeterPerformance
> > > > > 
> > > > > Regards
> > > > > Philippe
> > > > > On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad <
> > > > > philippe.mouawad@gmail.com> wrote:
> > > > > 
> > > > >   
> > > > >     
> > > > >         
> > > > > > Hello,
> > > > > > And are you able to insert images ? or Test Plan ?
> > > > > > 
> > > > > > Sebb, I think I tried with Attachment to insert Plan or Image, it proposes
> > > > > > a link type with a protocol, but I don't see how to upload the image or
> > > > > > Test Plan.
> > > > > > 
> > > > > > There is a button Insert/Edit image but it is disabled.
> > > > > > 
> > > > > > Regards
> > > > > > Philippe
> > > > > > 
> > > > > > 
> > > > > > On Fri, May 25, 2012 at 12:06 PM, Milamber <milamber@apache.org> wrote:
> > > > > > 
> > > > > >     
> > > > > >       
> > > > > >           
> > > > > > > On Fri, May 25, 2012 at 10:35 AM, sebb <sebbaz@gmail.com> wrote:
> > > > > > > 
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > On 25 May 2012 10:31, Milamber <milamber@apache.org> wrote:
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > Hello,
> > > > > > > > > 
> > > > > > > > > Actuallly, the page
> > > > > > > > >           
> > > > > > > > >             
> > > > > > > > >                 
> > > > > > > http://wiki.apache.org/jmeter/JMeterPerformancereturns
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > a http 500 error (others pages works)
> > > > > > > > >           
> > > > > > > > >             
> > > > > > > > >                 
> > > > > > > > Works for me at present.
> > > > > > > > 
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > Ok for me too.
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > 
> > > > > > > >               
> > > > > > > > > Milamber
> > > > > > > > > 
> > > > > > > > > ===
> > > > > > > > > Internal Server Error
> > > > > > > > > 
> > > > > > > > > The server encountered an internal error or misconfiguration and was
> > > > > > > > >           
> > > > > > > > >             
> > > > > > > > >                 
> > > > > > > > unable
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > to complete your request.
> > > > > > > > > 
> > > > > > > > > Please contact the server administrator at infrastructure@apache.orgto
> > > > > > > > > inform them of the time this error occurred, and the actions you
> > > > > > > > >           
> > > > > > > > >             
> > > > > > > > >                 
> > > > > > > > performed
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > just before this error.
> > > > > > > > > 
> > > > > > > > > More information about this error may be available in the server error
> > > > > > > > >           
> > > > > > > > >             
> > > > > > > > >                 
> > > > > > > > log.
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > ------------------------------
> > > > > > > > > Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org Port 80
> > > > > > > > > 
> > > > > > > > > ===
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > On Fri, May 25, 2012 at 1:27 AM, sebb <sebbaz@gmail.com> wrote:
> > > > > > > > > 
> > > > > > > > >           
> > > > > > > > >             
> > > > > > > > >                 
> > > > > > > > > > On 24 May 2012 22:15, Philippe Mouawad <philippe.mouawad@gmail.com>
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > wrote:
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > Hello,
> > > > > > > > > > > How can I insert image ? button is disabled in GUI mode.
> > > > > > > > > > > To insert Test Plan, is it an attachment ?
> > > > > > > > > > >               
> > > > > > > > > > >                 
> > > > > > > > > > >                     
> > > > > > > > > > Try clicking "Attachments" before starting the edit.
> > > > > > > > > > 
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > Thanks
> > > > > > > > > > > Regards
> > > > > > > > > > > Philippe
> > > > > > > > > > > 
> > > > > > > > > > > On Wed, May 23, 2012 at 12:22 AM, sebb <sebbaz@gmail.com> wrote:
> > > > > > > > > > > 
> > > > > > > > > > >               
> > > > > > > > > > >                 
> > > > > > > > > > >                     
> > > > > > > > > > > > On 22 May 2012 20:18, Philippe Mouawad <
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > philippe.mouawad@gmail.com>
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > wrote:
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > Hello Milamber,
> > > > > > > > > > > > > My responses below.
> > > > > > > > > > > > > Regards
> > > > > > > > > > > > > Philippe
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Mon, May 21, 2012 at 11:51 PM, Milamber <milamber@apache.org
> > > > > > > > > > > > >                   
> > > > > > > > > > > > >                     
> > > > > > > > > > > > >                         
> > > > > > > > 
> > > > > > > >               
> > > > > > > > > > wrote:
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > 
> > > > > > > > > > > > >                         
> > > > > > > > > > > > > > Le 21/05/2012 22:12, Philippe Mouawad a ecrit :
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > > > I read recently this little comparison :
> > > > > > > > > > > > > > > https://github.com/excilys/gatling/wiki/Benchmarks<
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > > > > > https://github.com/excilys/gatling/wiki/Benchmark>
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > I reviewed the test plan that was used to make the test.
> > > > > > > > > > > > > > > It seems to me the test is little biased:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >    - View Results Tree is in test plan (as it uses a lot of
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > memory,
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > it's
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > big issue)
> > > > > > > > > > > > > > >    - View Results in Table (same thing)
> > > > > > > > > > > > > > >    - 3 Non Standard JMeter listener, so It's not pure JMeter:
> > > > > > > > > > > > > > >       - jp@gc - Transactions per Second
> > > > > > > > > > > > > > >       - jp@gc - Response Times Over Time
> > > > > > > > > > > > > > >       - jp@gc - Active Threads Over Time
> > > > > > > > > > > > > > >    - Default XML output seems to have been used , it's
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > against
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > best
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > > > > practices
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Besides the following conclusions that seem to me non
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > scientific
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > and
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > purely
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > subjective:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >    - The testers, new to both Gatling and JMeter found that
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > JMeter
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > was
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > > harder to learn and use than Gatling to create the
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > simulations,
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > > > despite the
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > use of a proxy.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I will only look at other ideas expressed:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >    - JMeter creates one thread per user simulated. If there
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > is
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > not
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > enough
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > memory allocated to the JVM, it can crash trying to create
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > these
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > > > threads
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > - This need to be detailed, cause either it fails with
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > OOM
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > and
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > it's
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > not during thread creation, either it fails with
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > "unable
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > to
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > create
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > native thread"
> > > > > > > > > > > > > > >    - For instance, JMeter could not run 1500 users with 512
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > MB
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > (what
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > was
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > used for Gatling even with 2000 users); OutOfMemoryErrors
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > are
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > > > recorded in
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > the table as *OOM*
> > > > > > > > > > > > > > >       - => I made a Test with up to 2000 Threads with 512 m
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > without
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > any
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > crash, it depends on Test and on application
> > > > > > > > > > > > > > >    - Another problem occurred with the 2000 users
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > simulations;
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > it
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > seems
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > that JMeter can not simulate more than 1514 users
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > independently
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > from
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > memory that was allocated to the JVM
> > > > > > > > > > > > > > >       - => I made a Test with up to 2000 Threads with 512 m
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > without
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > any
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > crash, so assertion is false, it depends on Test and on
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > > > application
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > As it's difficult to install the application used for test
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > (last
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > version
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > does not seem to work as expected) , if they provide a
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > working
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > WAR
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > > > against
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > a local Postgres DB I will be happy to test with it.
> > > > > > > > > > > > > > > But in current state, application seems to be packaged for
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > cloud
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > or H2
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > > local DB, I didn't want to spend too much time setting up
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > application
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > as
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > don't know its real status.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I just tried to run Test Plan against a blank tomcat to
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > verify
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > what
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > they
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > say about Thread Creation, I didn't find any issue on this.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > So I decided to make a very simple scenario test on Tomcat
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > Examples
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > (It
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > goes to Session Example, adds attribute, go back to index, go
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > back
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > to
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > > Session Example, test contains Response assertion for each
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > Request).
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >                             
> > > > > > > > > > > > > > Please, indicate the Tomcat version used? it's the same
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > machine?
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > config
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > editor?
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > Tomcat apache-tomcat-6.0.24
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > -Xms256m -Xmx1024m
> > > > > > > > > > > > >    <Connector port="8080" protocol="HTTP/1.1"
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > >                   
> > > > > > > > > > > > >                     
> > > > > > > > > > > > >                         
> > > > > > > > > > > > 
> > > > > > > > > > > >                       
> > > > > > > > > > 
> > > > > > > > > >                   
> > > > > > > > 
> > > > > > > >               
> > > > > > > compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json"
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > > compression="off"
> > > > > > > > > > > > >                socketBuffer="8"
> > > > > > > > > > > > >                maxThreads="400"
> > > > > > > > > > > > >               connectionTimeout="20000"
> > > > > > > > > > > > >               redirectPort="8443" />
> > > > > > > > > > > > > Set session timeout in web.xml  to 1min
> > > > > > > > > > > > > java version "1.6.0_29"
> > > > > > > > > > > > > Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
> > > > > > > > > > > > > Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed
> > > > > > > > > > > > >                   
> > > > > > > > > > > > >                     
> > > > > > > > > > > > >                         
> > > > > > > mode)
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > > Mac OS 10.6.8
> > > > > > > > > > > > > JMeter  and Tomcat on same machine
> > > > > > > > > > > > > No particular OS Tuning
> > > > > > > > > > > > > 8GO RAM
> > > > > > > > > > > > > No swap, nothing runnning except these 2
> > > > > > > > > > > > > Tomcat CPU around 5%
> > > > > > > > > > > > > Memory around 50 mo
> > > > > > > > > > > > > 
> > > > > > > > > > > > > JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)?
> > > > > > > > > > > > >                   
> > > > > > > > > > > > >                     
> > > > > > > > > > > > >                         
> > > > > > > > > > > > > > Same config
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > No TCP tuning
> > > > > > > > > > > > > 
> > > > > > > > > > > > >                   
> > > > > > > > > > > > >                     
> > > > > > > > > > > > >                         
> > > > > > > > > > > > > > Can you post the jmx file?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Will it be accepted on this list ?
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > Best to create a Wiki page and attach the test file there.
> > > > > > > > > > > > 
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > 
> > > > > > > > > > > > >                         
> > > > > > > > > > > > > > Milamber
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > It is not at all representative but it is a way for me to
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > check
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > potential
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > issues in JMeter and performance changes accross versions.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I ran the test with 1500 VU using JMeter 2.5.1,  2.7 (current
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > trunk)
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > with
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > -Xmx512m, 10 minutes run and CSV output against a local
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > Tomcat
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > (I
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > restard
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > tomcat between tests and control its health):
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >    - I noticed that current trunk version behaves much
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > better in
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > terms of
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > memory than 2.5.1 or 2.6:
> > > > > > > > > > > > > > >       - In 2.5.1 :
> > > > > > > > > > > > > > >       - GC activity is much higher with around 5 GC CPU peaks
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > every 2
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > > minutes,  and 20 FULL GC of 700 to 800 ms each
> > > > > > > > > > > > > > >          - Throughput: 97,71%
> > > > > > > > > > > > > > >          - Pauses : 13,69s
> > > > > > > > > > > > > > >          - Mém : 391M/min
> > > > > > > > > > > > > > >          - Full GC tend to be much more frequent at end of
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > test
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > > > > - 2.7:
> > > > > > > > > > > > > > >          -  no GC CPU peak, 1 FULL GC
> > > > > > > > > > > > > > >          - Throughput:98.54
> > > > > > > > > > > > > > >          - Pauses : 8.9s
> > > > > > > > > > > > > > >          - 1108m /min
> > > > > > > > > > > > > > >          - Summary:
> > > > > > > > > > > > > > >       - 25.1:
> > > > > > > > > > > > > > >          - 164676 samples in 605,1s
> > > > > > > > > > > > > > >          - 272,2/s
> > > > > > > > > > > > > > >          - Avg:    97
> > > > > > > > > > > > > > >          - 2.7:
> > > > > > > > > > > > > > >          - 165367 in 605.0s
> > > > > > > > > > > > > > >          - 273.3/s
> > > > > > > > > > > > > > >          - Avg:   228
> > > > > > > > > > > > > > >          - I also noticed results have a much better look:
> > > > > > > > > > > > > > >       - in 2.5.1, Transactions/s are around 300 / sec during
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > 4
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > > > minutes,
> > > > > > > > > > > >                 
> > > > > > > > > > > >                   
> > > > > > > > > > > >                       
> > > > > > > > > > > > > > > then drop to 200/s, go up to 400/s , then down to 260/s
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > and
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > > > > > > finally 200/sec
> > > > > > > > > > > > > > >       - in 2.7, Transactions stay around 300/sec
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Actions:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >    - I think it would be useful to have some reference
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > application
> > > > > > > >         
> > > > > > > >           
> > > > > > > >               
> > > > > > > > > > on
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > we could test JMeter behaviour
> > > > > > > > > > > > > > >    - What would the best place to put these comparisons ?
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > wiki ?
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > which
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > > indicators should we put ?
> > > > > > > > > > > > > > >    - Further testing should be done against a richer
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > application
> > > > > > >       
> > > > > > >         
> > > > > > >             
> > > > > > > > > > that
> > > > > > > > > >             
> > > > > > > > > >               
> > > > > > > > > >                   
> > > > > > > > > > > > > > could
> > > > > > > > > > > > > >                     
> > > > > > > > > > > > > >                       
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > > > be deployed locally
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > Philippe
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >                       
> > > > > > > > > > > > > > >                         
> > > > > > > > > > > > > > >                             
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > >                           
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Cordialement.
> > > > > > > > > > > > > Philippe Mouawad.
> > > > > > > > > > > > >                   
> > > > > > > > > > > > >                     
> > > > > > > > > > > > >                         
> > > > > > > > > > > > 
> > > > > > > > > > > >                       
> > > > > > > > > > > --
> > > > > > > > > > > Cordialement.
> > > > > > > > > > > Philippe Mouawad.
> > > > > > > > > > >               
> > > > > > > > > > >                 
> > > > > > > > > > >                     
> > > > > > > > > > 
> > > > > > > > > >                   
> > > > > > > > 
> > > > > > > >               
> > > > > > > 
> > > > > > >             
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >     
> > > > > >       
> > > > > >           
> > > > > 
> > > > >         
> > > > 
> > > >       
> > 
> > 
> >   
> 



Mime
View raw message