mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Chen" <chenp...@gmail.com>
Subject Re: Permanent solution for OOM errors
Date Tue, 08 May 2007 23:37:36 GMT
Hi, folks,

Here is what I did to solve the OOM problem caused by writeRequest Queues
piling up
when many SocketConnector sessions bombard a service provider thus making
it very slow in responding.

It is the sum of all writeRequest queues that blows up the memory.

The idea is simple, but, alas, it is not that generic. But you get the idea
when reading
the code comments.

Not sure what's the best way/place to modify the Mina src to create global
views
across sessions?

Thanks for comments


package ....

import org.apache.mina.common.IoFilterAdapter;
import org.apache.mina.common.IoSession;

/**
 * Prevent out of memory caused by output write queue piling up
 *
 * This filter is attached on a Mina SocketConnector session to:
 *
 * - Check outputQ quota of a service provider, the quota implicitly
indicates the max capacity of the
 *   service provider
 *
 * This filter complements the Mina ReadThrottleFilter which I attach to
each acceptor opened by my
 * Mina-based server, and assign a max quota to each. I preconfigure them to
make sure the
 * total of all acceptor quota values don't exceed my system memory limit.
 */
public class WriteThrottleFilter extends IoFilterAdapter {

    /** Write a request to a service provider via a SocketConnector */
    @Override
    public void filterWrite(NextFilter nextFilter, IoSession session,
WriteRequest writeRequest) throws Exception {

        // A codec turned it to my Request obj already
        Request req = (Request) message;
        int reqSize = req.size();
        String svc = req.getServiceName();

        // Check outputQ quota for the svc
        //

        // Make sure we don't exceed max capacity pre-configured for this
service provider
        // There can be many SocketConnectors sending reqs to the service
provider. When
        // it is slow, all write queues connected to this service pile up in
Mina. Thus I
        // need a global view instead of a single
session.getScheduledWriteBytes()

        if (reqSize + svcProviders.outQueueSize(svc) >
svcProviders.outQueueQuota(svc)) {
            // Mark the service as busy, reply an error response to the
client
            return;
        }

        svcProviders.incrOutQueue(svc, reqSize);

        nextFilter.filterWrite(session, writeRequest);
    }

} // End of class

On 5/3/07, Martin Ritchie <ritchiem@apache.org> wrote:
>
> That is great. I look forward to using the new filter as we have been
> having a lot more OOM problems on our clients under load testing
> (although that is as much a fault of the test app).
>
> On 03/05/07, Vinod Panicker <vinod.p@gmail.com> wrote:
> > I've already done that. It's being tested right now. We'll be
> > contributing that as soon as we are happy about the performance under
> > high load.
> >
> > On 5/2/07, Martin Ritchie <ritchiem@apache.org> wrote:
> > > On 30/04/07, Paul Chen <chenpaul@gmail.com> wrote:
> > > > I'm refining that to make it more presentable and generic, will
> publish
> > > that
> > > > in 1 week or 2.
> > > >
> > > > On 4/30/07, Gaston Dombiak <gaston@jivesoftware.com> wrote:
> > > > >
> > > > > Hey Paul,
> > > > >
> > > > > That is exactly what I was planning to implement for our case. Can
> you
> > > > > send me your modified version? I would like to include it in
> Openfire.
> > > > >
> > > > > Thanks,
> > > > >
> > > > >   -- Gato
> > > > >
> > > > > -----Original Message-----
> > > > > From: Paul Chen [mailto:chenpaul@gmail.com]
> > > > > Sent: Monday, April 30, 2007 1:07 PM
> > > > > To: dev@mina.apache.org
> > > > > Subject: Re: Permanent solution for OOM errors
> > > > >
> > > > > I actually *tweaked* the read throttle to take the output Queue
> size
> > > > > into account to avoid the slow write syndrome. Maybe I should call
> > > > > it traffic control throttle instead.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On 4/29/07, Vinod Panicker <vinod.p@gmail.com> wrote:
> > > > > >
> > > > > > On 4/29/07, Paul Chen <chenpaul@gmail.com> wrote:
> > > > > > > After curbing the read rate (one quota per acceptor) at
a
> > > > > conservative
> > > > > > rate,
> > > > > > > the write problem is automatically gone.
> > > > > >
> > > > > > Not really.  The read quota can be applied per acceptor, but
the
> > > > > > server might have connectors that could be writing to slower
> clients,
> > > > > > leading to another OOM.
> > >
> > > Hi,
> > >
> > > I've been watching this thread grow but not had time to contribute
> > > (Sorry about that).
> > >
> > > The Qpid project is hoping to move the latest released version of mina
> > > and so I would be happy to take another look at the
> > > WriteThrottleFilter (on DIRMINA-302) to see how it can be tidied up.
> > > Specifically the filter currently limits by number of requests whereas
> > > the number of bytes on the queue would be far better solution.
> > >
> > > --
> > > Martin Ritchie
> > >
> >
>
>
> --
> Martin Ritchie
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message