jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebb <>
Subject Re: Graphite Listener
Date Wed, 01 Jan 2014 19:04:41 GMT
On 1 January 2014 15:08, Philippe Mouawad <> wrote:
> On Tue, Dec 31, 2013 at 9:00 PM, Vladimir Sitnikov <
>> wrote:
>> >> I believe JMeter plugin-like interface between injector and performance
>> >> repository would enable live monitoring of sytem under test.
>> sebb>Exactly. The JMeter architecture is designed to allow for easy
>> integration of 3rd party plugins.
>> I need to double-check, however I did not manage to specify "path to
>> plugins.jar folder" via command line option.
>> Ultimately I would love to have JMeter installation and plugins in
>> completely separate folders. That simplifies "jmeter upgrade", "check
>> what plugins are installed", "compose test harness from maven"
>> usecases.
>> Add this to
> plugin_dependency_paths which will contains the path to third party jar
> user.classpath which will contain path to dependencies of third party jar
>> Are there "asynchronous output" interfaces from JMeter?
> in remote testing you are in async mode with:
> mode=StrippedAsynch
> or
> mode= Asynch
>> Is there a way to send listener result via regular samplers?
> Why would you like to do that ?
> sebb> Proper analysis should take place offline after the test has
>> completed.
>> Very true.
>> However, it is quite important to perform online analysis to be able
>> to adjust the test.
>> Say, adjust the load, fix bugs in script, correct system configuration,
>> etc.
>> >> One can parse raw csv/xml results and upload for the analysis, however
>> it
>> >> is likely to create big latency gap between collection and the
>> >> visualization.
>> >
>> sebb> Is that really a problem for most users?
>> How do we measure that?
> A vote would be a way to sort it out.

Perhaps, though it's very difficult to know whether a vote is representative.

>> Here are the most relevant scenarios for our company:
>> 1) Durability testing. Say, you launch 1..7days long test script.
>> It is crucial to know if the system is yet stable.
>> That includes multiple KPIs (throughput, latency, failrate%, memory
>> consumption, gc, cpu%), and the request processing KPIs are not the
>> least important ones.
>> In case JMeter sampler info is not ready until the whole test is
>> finished it is major drawback.
> Agree

There are ways round this.
The Generate Summary Results Listener was designed for just this sort
of scenario.
I was involved in testing a large system where the test ran for
several days, and that was ideal for checking that the test was still
running well.

Obviously it only provides basic info, but sufficient to check that
the test is generally behaving (or not).
Its big advantage is the low overhead and simplicity.

>> 2) Automated scalability testing. Say you want to identify maximum
>> load the system will sustain. One way to identify it is to gradually
>> increase the load and see if the system is stable (e.g. queues do not
>> build up, failrate is 0, response times are stable, etc).
>> Having data displayed in near-realtime helps a lot.
>> Especially, when you run test suite in different environment (e.g.
>> acceptance of new software/hardware)
> Agree

We used the Generate Summary Results Listener for that as well.

>> 3) Tight-schedule testing. When performing load testing at customer
>> environment (e.g. acceptance of production environment), it is
>> important to make informed decisions. Is good to see if your test
>> works as expected when the test is running, not when you've done 4
>> hours of testing and analyzing afterwards.
> Agree
>> 4) Red-green detection during regular 30-60min testing. Our scenarios
>> involve multiple jmx scripts and lots of samples.
>> We use JMeter GUI only for script development. We use just console for
>> load testing to avoid injector slowness/out of memory/etc.
>> Currently it is hard to tell if the test goes as expected: failrate%,
>> number of samples, response times, etc.

Most of that is available from the Generate Summary Results Listener already.

>> Agree
>> sebb> That would be a sensible addition to JMeter to allow performance data
>> > to be readily saved to an arbitrary repo.
>> True.
>> OOB samplers might work (e.g. http sampler) might work for the "output
>> interfaces".
>> For instance "http sampler" under "stats listener" posts the result of
>> "stats listener" to the configured end-point.
>> However, I believe this kind of integration should be asynchronous to
>> avoid impact result collection on the test scenario. If trying to use
>> sync result posting, we could result into multi-second test hang due
>> to lag of performance repository receiver.
>> Currently the Graphite Listener patch it is implemented this way:
> 1) Synchronously, success , failures, min and max are updated (very fast
> computations), percentile related data is also computed (Looking at code of
> commons-math3 I don't see why it would not perform fine). If  further tests
> show the percentile computation is costly , then we could delay this to the
> 2) thread below. But the tests I did did not show problem.
> 2) And in the background, a task runs every minute to send current computed
> results to Graphite server , resetting datas except for percentile related
> ones.

Seems to me all this could be done by having a generic interface to
enable the raw results to be saved to whatever backend is required.

What I don't agree with is yet more code that is very specific to a
single 3rd party software solution.

Whatever is added needs to be
- generally useful to the JMeter population. Niche requirements should
be met by 3rd party add-ons developed for the purpose.
- use a generic API, for example like JSR223, JDBC etc.

>> Regards,
>> Vladimir Sitnikov
> --
> Cordialement.
> Philippe Mouawad.

View raw message