jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe Mouawad <philippe.moua...@gmail.com>
Subject Re: Bug 56119 - dealing with idle connection timeout and dropped connection
Date Mon, 31 Mar 2014 21:00:49 GMT
Hello,
As discussed, I wrote this :

   - https://wiki.apache.org/jmeter/JMeterSocketClosed

Your review is welcome.

Regards

Philippe



On Fri, Mar 21, 2014 at 12:48 PM, sebb <sebbaz@gmail.com> wrote:

> On 21 March 2014 10:14, Philippe Mouawad <philippe.mouawad@gmail.com>
> wrote:
> > Hello,
> > Regarding this item, I think we should make retry behaviour (or stale
> > check) reflect what browser do.
>
> Maybe, but one on the reasons we changed the code was that JMeter was
> making additional unexpected requests, putting more load on the
> server.
> That was mainly due to the unconditional stale checking.
>
> I think we need to consider collecting statistics on retries - perhaps
> an extra field in the sample result?
> Or perhaps a sub-sample?
>
> And the behaviour should ideally be configurable.
> In this case a property would be OK as I think the behaviour does not
> need to be host-specific.
>
> > It seems browsers by default do some retries but I was not yet able to
> find
> > a reference of this behaviour.
>
> Yes, we need to find out what sort of errors cause the browser to
> retry, and how many times.
> However, note that browsers may use asynchronous I/O which would allow
> them to detect connection failure much earlier.
> With synch I/O it's not generally possible to tell if the connection
> dropped before the request was completely sent.
> So one cannot determine if a non-idempotent request should be auto-retried.
> [That's where the conditional stale check is needed]
>
> > Regards
> > Philippe
> >
> >
> > On Wed, Feb 26, 2014 at 2:59 AM, sebb <sebbaz@gmail.com> wrote:
> >
> >> On 23 February 2014 04:38, sebb <sebbaz@gmail.com> wrote:
> >> > On 19 February 2014 21:51, sebb <sebbaz@gmail.com> wrote:
> >> >> On 19 February 2014 21:03, Philippe Mouawad <
> philippe.mouawad@gmail.com>
> >> wrote:
> >> >>> On Wed, Feb 19, 2014 at 9:58 PM, sebb <sebbaz@gmail.com>
wrote:
> >> >>>
> >> >>>> On 19 February 2014 18:49, sebb <sebbaz@gmail.com> wrote:
> >> >>>> > It looks as though the problem reported in Bug 56119 was
due to
> the
> >> >>>> > server dropping connections that have been idle too long.
> >> >>>> >
> >> >>>> > There may also be servers that only allow a connection
to be
> reused
> >> a
> >> >>>> > certain number of times (this does not seem to have been
the case
> >> >>>> > here).
> >> >>>> >
> >> >>>> > This email is to discuss what JMeter could perhaps do
to make it
> >> >>>> > easier to test such servers.
> >> >>>> >
> >> >>>> > The two existing work-rounds are:
> >> >>>> > - disable Keep-Alive
> >> >>>> > - enable staleCheck
> >> >>>> >
> >> >>>> > Neither is ideal; the first is awkward to use, and staleCheck
can
> >> >>>> > generate unnecessary additional traffic (which is why
it was
> >> disabled
> >> >>>> > in 2.11).
> >> >>>> >
> >> >>>> > I can think of two possible approaches:
> >> >>>> >
> >> >>>> > 1) proactively shut connections. This would be easy for
servers
> that
> >> >>>> > limit reuse.
> >> >>>> > Just count reuses and turn off keep-alive when a specified
limit
> is
> >> >>>> reached.
> >> >>>> > Not so easy for idle timeouts; one cannot retroactively
disable
> >> >>>> keep-alive.
> >> >>>> >
> >> >>>> > 2) Deal with the disconnects when they occur.
> >> >>>> > The code needs to distinguish which errors are retriable,
and may
> >> need
> >> >>>> > to distinguish at what point the failure occurs. For example,
> even a
> >> >>>> > POST ought to be retriable if JMeter is unable to send
any data
> on
> >> the
> >> >>>> > connection.
> >> >>>> >
> >> >>>> > Also need to consider how one might report retries.
> >> >>>> > I think the tester needs to be able to find out if additional
> >> requests
> >> >>>> > have been made by JMeter.
> >> >>>>
> >> >>>> Further testing against the ASF servers shows that HC 4.2.x
does
> >> >>>> handle idle timeouts without needing to use the staleCheck
option.
> >> >>>> This relies on the server sending a header of the form:
> >> >>>>
> >> >>>> Keep-Alive: timeout=5, max=100
> >> >>>>
> >> >>>> In this case, the connection is automatically recreated if
> necessary
> >> >>>> when the next sampler runs.
> >> >>>> If the server fails to send the header, then the connection
may be
> >> >>>> dropped unexpectedly (which is what was happening with Bug
56119).
> >> >>>> So another approach might be to allow an optional keep-alive
> timeout
> >> >>>> in case the server does not provide one.
> >> >>>>
> >> >>>> Or we could take the view that there is nothing to fix in JMeter.
> >> >>>> The Keep-Alive header is there for a reason, it tells the client
> when
> >> >>>> it is safe to reuse the connectiion.
> >> >>>> If the server fails to send it, then it is broken, and so the
> failed
> >> >>>> samples are to be expected.
> >> >>>>
> >> >>>
> >> >>> I think we need to make something at least for servers like Amazon
> S3
> >> which
> >> >>> close connections after number of uses.
> >> >>> Did you check to see if this kind of server send a keep alive
> header ?
> >> >>
> >> >> I just tested again with jmeter.a.o.
> >> >> It returns headers of the form:
> >> >>
> >> >> Keep-Alive: timeout=5, max=100
> >> >> Connection: Keep-Alive
> >> >> ...
> >> >> Keep-Alive: timeout=5, max=99
> >> >> Connection: Keep-Alive
> >> >> ...
> >> >> etc
> >> >> ...
> >> >> Keep-Alive: timeout=5, max=1
> >> >> Connection: Keep-Alive
> >> >> ...
> >> >> Connection: close
> >> >>
> >> >> So the HC connection manager does not need to keep track of the
> >> >> remaining re-use count; the server disconnects at the end of the last
> >> >> request.
> >> >> Nice and simple.
> >> >>
> >> >> I assume S3 does the same as jmeter.a.o if it is well-behaved.
> >> >
> >> > I asked on the HC dev list, and found out that the Keep-Alive header
> >> > is optional.
> >> > So servers that don't send it are not misconfigured - though of course
> >> > it helps the client if they send the header.
> >> >
> >> > Also, the HC default retry processing does handle the disconnect case,
> >> > i.e. where the server disconnects whilst starting to send the next
> >> > request.
> >> > Setting retry count to 1 allows the request to be retried.
> >> >
> >> > JMeter previously used to enable retries, but we found this had
> >> > adverse effects, as it could generate extra requests.
> >> > It turns out that the default retry handler always retries failed
> >> > idempotent requests (e.g. GET), even if they were successfully sent.
> >> > This is perhaps what caused the additional traffic.
> >> >
> >> > So I think we need an amended version of the retry handler which only
> >> > retries failed sends, if the retry count is 0.
> >> > We can use the HC version for retry counts > 0.
> >>
> >> Unfortunately this does not work, as the disconnect is only detected
> >> once the request has been sent.
> >>
> >> However, it is possible to specify an idle timeout to deal with a
> >> missing Keep-Alive header.
> >> I've tried that and it works, but we need some decisions on how to
> >> implement it.
> >> I'll send a separate e-mail for that.
> >>
> >> It may also be possible to code a conditional stale connection check
> >> which is not applied every single time.
> >> e.g. it could be done after N requests or when the connection has been
> >> idle for T seconds.
> >> I'll look into this a bit more and report back in another mail.
> >>
> >> >>> Anyway on my side I think what has been changed in 2.11 should
not
> be
> >> >>> reverted, because for servers correctly configured you don't get
> these
> >> >>> errors, I made 3 campaigns on different servers  with 2.11 and
never
> >> got
> >> >>> this kind of issues.
> >> >>
> >> >> Agreed, no need to revert.
> >> >>
> >> >>> But maybe we should document it better somewhere.
> >> >>
> >> >> Yes, the error and likely cause should be documented.
> >> >> Probably easiest to start as a Wiki page.
> >> >>
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>



-- 
Cordialement.
Philippe Mouawad.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message