mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeroen Brattinga <jeroen.bratti...@gmail.com>
Subject Re: Mina throughput
Date Thu, 23 Aug 2007 16:24:44 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Well, I would first check out the base header buffer, like so:<br>
<br>
ByteBuffer message;<br>
<br>
message.order(ByteOrder.LITTLE_ENDIAN);<br>
byte type = message.get();<br>
long length = message.getUnsignedInt();&nbsp;&nbsp; &nbsp;&nbsp; &nbsp;&nbsp;
// assuming length is
more than one byte, since you mention endianness<br>
char[] headServiceName = 6 x message.getChar();<br>
...etc.<br>
<br>
After checking type and header information, you could pass on the body
to another class.<br>
If the body is BIG_ENDIAN, you could simply do
body.order(ByteOrder.BIG_ENDIAN) and start reading.<br>
<br>
You normally want to read the recieved buffer with a state machine
(since you never know you have received the entire message).<br>
Every field could be a state (Enum is handy in this case, or take a
look at the state machine mentioned in these Mina threads), e.g.<br>
<br>
while message.remaining() {<br>
&nbsp;&nbsp;&nbsp; switch(state) {<br>
&nbsp;&nbsp;&nbsp; case readType: <br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; type = message.get();<br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; if type checks out, state =
MessageState.readLength;<br>
&nbsp;&nbsp;&nbsp; case readLength:<br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp; length = messag.getUnsignedInt();<br>
<br>
etc.<br>
<br>
(note: don't use the Enum identifier in the cases, just the last part!)<br>
You could check for buffer underflows (when you try to read data you
haven't yet got) and (if necessary) skip over parts you don't need
(corrupt data, duplicate messages etc.).<br>
<br>
I hope this makes it a little bit clearer how powerful the Mina
ByteBuffer can be. Read the JavaDoc for more cool things!<br>
<br>
<br>
Jeroen Brattinga<br>
<br>
<br>
mat wrote:
<blockquote
 cite="mid44ee70730708211731n4cafafbete5197038c32a1a71@mail.gmail.com"
 type="cite">
  <pre wrap="">On 8/22/07, Jeroen Brattinga <a class="moz-txt-link-rfc2396E" href="mailto:jeroen.brattinga@gmail.com">&lt;jeroen.brattinga@gmail.com&gt;</a>
wrote:

  </pre>
  <blockquote type="cite">
    <pre wrap="">Well, I just use the (Mina) ByteBuffer (see
<a class="moz-txt-link-freetext" href="http://mina.apache.org/report/1.1/apidocs/org/apache/mina/common/ByteBuffer.html">http://mina.apache.org/report/1.1/apidocs/org/apache/mina/common/ByteBuffer.html</a>)
methods to get each field (e.g. getUnsignedInt, getString, getChar). I
have one class that does the basic unpacking (header, CRC, payload size
etc.), and some more specific classes that worry about the payload. I didn't
find it cumbersome, although a struct would have been easier.

But now that it's implemented, I only deal with nice and easy message
objects that contain the processed data (using the
DemuxingProtocolCodecFactory).

Could you give a more detailed specification of one of your packets? It
would give us some more insight into the problem.

    </pre>
  </blockquote>
  <pre wrap=""><!---->
Thanks. Jeroen
My protocol is something like:
byte(type of message) + byte(length of body-little-endian) + head + body

head:
#define Length 450
typedef struct _tagHeader
{
 char cServiceName[6];
 char cCompress[1];
 char cComprLevel[1];
 unsigned int A;
 unsigned int B;
 ...
}Header

body:
typedef struct _tagBody
{
 unsigned int iLength;
 char cMsg[MaxBodyLength];
}Body;

What bothered me is:
1) convert each integer field from little-endian to big-endian
2) use system.arraycopy in order to get each byte array and convert a byte
array to string

It takes really a lot of CPU time and memory for decode and encode.


  </pre>
  <blockquote type="cite">
    <pre wrap="">Jeroen Brattinga


mat wrote:

The reason why I raised this thread is: recently I am really bothered with
following issue.

My mina component has to connect to a legacy server written in C++. C++
server just simply sends out the struct as a message. I think for a C++
program it is just simple return the struct pointer and use each field.
However, to my java program that is a disaster(also really lower the
performance). I have to use all the system.arraycopy to make up each
field.(Actually it causes OOM). I don't know if you guys ever faced this
kind of problem. How to solve it?


On 8/11/07, Emmanuel Lecharny <a class="moz-txt-link-rfc2396E" href="mailto:elecharny@gmail.com">&lt;elecharny@gmail.com&gt;</a>
<a class="moz-txt-link-rfc2396E" href="mailto:elecharny@gmail.com">&lt;elecharny@gmail.com&gt;</a>
wrote:


Hi guys,

when guessing relative performances of Java/C++ on a network
environment, please keep in mind that data processing will be
processed orders or magnitude faster than simple network handling by
the underlying layer. Thinking that Java is slower than C++ to handle
messages on a network based application because it does not have
pointers is out of base.

FYI, I have tested our LDAP server on my laptop, sending Search
requests through MINA (Apache Directory Server is based on MINA, 1.0.3
version), and I got something like 5000 req/s (a request is around
1kb), assuming that almost all the time is spent internally to the
server itself, not in the MINA layer. Btw, LDAP messages are binary,
but that does not mean it's easier to decode them in C/C++ than in
Java (I would say that the complexity is exactly the same for both
languages).

I don't want to start a flame war, but I encourage anyone who want to
compare Java and C++ to compare things that are comparable, and not
blind guess what can be slow or fast in both languages.

My 2cts
Emmanuel

On 8/11/07, mat <a class="moz-txt-link-rfc2396E" href="mailto:forum.maillist@gmail.com">&lt;forum.maillist@gmail.com&gt;</a>
<a class="moz-txt-link-rfc2396E" href="mailto:forum.maillist@gmail.com">&lt;forum.maillist@gmail.com&gt;</a>
wrote:


Actually the windows IOCP server is written in C++ and it is running on


a pc


server. I believe the reasons why Java can't reach that performance
sometimes due to lack of structure(pointer). I find it is really


cumbersome


job to decode a binary message in Java. I don't know if you guys have


better


solution to decode the binary format and manipulate the each field in


the


binary message.

On 8/11/07, Michael Grundvig <a class="moz-txt-link-rfc2396E" href="mailto:mike@electrotank.com">&lt;mike@electrotank.com&gt;</a>
<a class="moz-txt-link-rfc2396E" href="mailto:mike@electrotank.com">&lt;mike@electrotank.com&gt;</a>
wrote:


I don't have the exact numbers but I know on a big Linux box (8
processors,
8 gb ram) with a switched gigabit backbone we have seen greater then
45,000 - 50,000 messages per second sustained. Ultimately the problem
becomes a matter of garbage collector churn rather then IO overhead.


On


 Windows machines we could get only to a fraction of that. We believe


the


 underlying I/O differences between Windows and *nix become really


obvious


 when you get to higher message counts.

Michael

----- Original Message -----
From: "mat" <a class="moz-txt-link-rfc2396E" href="mailto:forum.maillist@gmail.com">&lt;forum.maillist@gmail.com&gt;</a>
<a class="moz-txt-link-rfc2396E" href="mailto:forum.maillist@gmail.com">&lt;forum.maillist@gmail.com&gt;</a>
To: "dev" <a class="moz-txt-link-rfc2396E" href="mailto:dev@mina.apache.org">&lt;dev@mina.apache.org&gt;</a>
<a class="moz-txt-link-rfc2396E" href="mailto:dev@mina.apache.org">&lt;dev@mina.apache.org&gt;</a>
Sent: Friday, August 10, 2007 11:24 AM
Subject: Mina throughput




Does anyone have the throughput test by raw socket communication
(keep-alive
mode)? My colleague wrote a windows IOCP server whose throughput


could


 reach1.8m/sec. (5000message/sec in intranet).



        --
Regards,
Cordialement,
Emmanuel L&eacute;charnywww.iktek.com


    </pre>
  </blockquote>
  <pre wrap=""><!---->
  </pre>
</blockquote>
</body>
</html>

Mime
View raw message