mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gaston Dombiak" <gas...@jivesoftware.com>
Subject RE: Huge SessionBuffer event queue empties too slowly
Date Thu, 11 Jan 2007 17:28:26 GMT
Hey Paul,

Nice analysis. I can think of a few things you may want to try:

1) Branch 1.1 (and I guess that trunk too) are using a
ConcurrentLinkedQueue instead of an Array. So performance with big
queues should be better.
2) Based on your description I infer that you are using a single
connection that in turn uses a single thread (from the pool) to process
incoming traffic. While the processing thread is busy events (e.g.
incoming data) gets queued. Therefore:
2.1) verify that the time it takes to process the incoming data is as
fast as possible. The faster is process incoming data then the smaller
the queue will get
2.2) change your thread model (e.g. add another ExecutorFilter after
your ExecutorFilter) so that you have a pool of threads processing the
events queue.
2.3) add more connections between your 2 applications so that more
threads from the ExecutorFilter get used.

Note: Options 2.2 and 2.3 should be profiled to ensure that speed is
really improving compared to current situation. I think that ThreadModel
is also related to the number of CPUs that you have so that context
switching is not an overhead and having several threads is a benefit. :)

Regards,

  -- Gato

-----Original Message-----
From: Paul Mallach [mailto:pm@ariva.de] 
Sent: Thursday, January 11, 2007 9:05 AM
To: dev@mina.apache.org
Subject: Huge SessionBuffer event queue empties too slowly

Hi.

We're using MINA as the network library for our quote (as in stock
exchange 
market data) server. 

I'm currently debugging a problem that seems to occur when a lot of data
is 
sent in a short amount of time: We have two quote servers that replicate
data 
between them. So when one server starts up, it acts as a client to the
other 
server and requests a complete data-snapshot.

The active server then calls session.write(Quote q) for about a million
quote 
objects. Every few thousand calls we use WriteFuture.join(), so that the

client is not overloaded. The client received the data without a hitch.

The problem is, that all the data seems to remain in memory on the
server even 
after it has been sent to the client: The java heap grows by about 600 
megabyte. After a few hours however the heap usage has dropped down
again.

I have tried to analyze heap dumps before and after the replication
process: 
After the replication is completed (client has received all the data),
all 
ByteBuffers etc. created during sending are still in memory. In
particular I 
found that there is an instance of 
org.apache.mina.filter.executor.ExecutorFilter$SessionBuffer that has
the 
status "not completed" and that contains more than 2 million elements in
it's 
event queue. These seem to be split about halfway between 
org.apache.mina.filter.codec.ProtocolCodecFilter$MessageByteBuffer
(first) 
and org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
(later 
on). These entries seem to correspond to the objects that have
previously 
been sent to the client.

Heap usage return to normal after a few hours. Is this an expected
behaviour 
or am I doing something wrong? Is there a way to make those entries
disappear 
sooner?

One of Mina's threads seems to be working on an event queue: 

"AnonymousIoService-1" daemon prio=10 tid=0x00002aab3dbcd400 nid=0x1334 
runnable [0x0000000042d55000..0x0000000042d55dc0]
   java.lang.Thread.State: RUNNABLE
        at 
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent
(AbstractIoFilterChain.java:531)
        at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100
(AbstractIoFilterChain.java:52)
        at 
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Sent(AbstractIoFilterChain.java:783)
        at 
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilt
er.java:248)
        at 
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run
(ExecutorFilter.java:305)
        at 
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWor
ker(ThreadPoolExecutor.java:987)
        at 
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker
.run(ThreadPoolExecutor.java:528)
        at java.lang.Thread.run(Thread.java:619)

I digged around in the mina's source and found, that ExecuteFilter uses
an 
ArrayList to queue events and calls  ArrayList.remove(0) to get the
first 
element.  

As a test I created an ArrayList with 2 million Integer objects. Just
removing 
the first 10000 elements in this way takes between 45 and 95 seconds. 

Of course it will get faster when the list shrinks, but I would still 
estimate, that just removing the elements from the list might take more
than 
an hour. Could a LinkedList be used instead of the ArrayList? As far as
I can 
see the only operations used on the list are append at the end and
remove at 
the beginning, all of which would be constant time operations for a 
LinkedList.

We are using mina-core-1.0.0.jar. The used IoHandler is directly derived
from 
IoHandlerAdapter. The only filter that is manually inserted is a 
ProtocolCodecFilter with my custom Decoder and Encoder implementation.

bye, Paul.

-- 
ARIVA.DE AG, Paul Mallach, Produktentwicklung
Walkerdamm 17, 24103 Kiel
Tel.: +(49) 0431/97108-23, Fax: +(49) 0431/97108-29
E-Mail: pm@ariva.de, Web: www.ariva.de

Mime
View raw message