mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <akaras...@apache.org>
Subject Re: IoBuffers of bytebuffers
Date Sat, 20 Aug 2011 10:59:05 GMT
On Sat, Aug 20, 2011 at 2:46 AM, Emmanuel Lecharny <elecharny@gmail.com>wrote:

> On 8/19/11 2:26 PM, Alan D. Cabrera wrote:
>> On Aug 19, 2011, at 5:19 AM, Emmanuel Lecharny wrote:
>>  On 8/19/11 2:16 PM, Alan D. Cabrera wrote:
>>>> I'm wondering.  Do you guys think it's a good idea?  It seems to make
>>>> things pretty complicated and adds another dimension to groking the behavior
>>>> of your service.  I'm not sure that it's necessary.
>>> Definitively a bad idea.
>>> What we need is an abstraction on top of an Array of ByteBuffer (the Java
>>> NIO class), which extends the size by adding new ByteBuffer on the fly.
>>> The array must behave exactly as the ByteBuffer.
>>> I wrote such a class 2 years ago, but I can't find the code. Will look
>>> again on my USB keys this week-end.
>> So, what is the scenario that we're trying to support?  I imagine
>> appending headers to binary data would be one.  In this case is an Array of
>> ByteBuffers really needed?  Why not just send down one ByteBuffer for the
>> header and another for the body that was sent to you?
> Julien and Steve laready responded with clear examples where expandable BB
> are necessary. I do think this is a common case, assuming you can perfectly
> receive the data byte by byte, and you want to store those bytes in a place
> which does not need to be reallocated every time.
> The current IoBuffer, when full, is copied in a new loBuffer which size is
> doubled. If you are expecting big PDUs, you might allocate huge Iobuffer for
> nothing, wasting space. Plus you have to copy all the old IoBuffer into the
> new one. Not really a good thing.
> *if* we are using DirectBuffer, we even might want to use fixed size
> IoBuffer (say 1k), and store them in a queue of avilable free buffers for
> reuse, sparing the cost of allocating them (to be validated, I'm not sure
> that the cost of managing concurrent access to this queue does not
> overweight the cost of allocating a new direct buffer)
> For HeapBuffers, i'm quite sure that we should allocate new ByteBuffer with
> the exact size of the received data (if it's 10 bytes, then the new
> ByteBuffer will be 10 byte slong. If it's bigger, then fine). This
> additional ByteBuffer can be appended a the end of the ByteBuffer array, and
> the filter will be able to process the list when it has everything needed.
> Julien mentionned the Cumulative decoder, which can be used in may cases :
> - we are waiting for a \n (HTTP protocol)
> - we are waiting for a closing tag (XML)
> - we are dealing with LV (Length/value) PDU, and the received value is not
> fully received (we are expecting Length bytes)
> - we are dealing with fixed size PDU
> One more thing to consider : at some point, we may want to flush the
> ByteBuffer to disk, if we are waiting for a huge PDU, to avoid sucking all
> the memory : this is typically the case in LDAP when receiving a JpegPhot
> attribute, inside a PDU (we don't know that it's a JpegPhoto attribute until
> we have read and processed teh full PDU). With a proxy on top of the
> ByteBuffer, we can eventually hide the fact that the bytes are on disk.

Although these PDU's are large you can store arbitrary data as well, who
knows what kinds of data and what scenarios users will come up with. It
would be very nice if MINA allowed some way behind the scenes to stream
large PDU's to disk and transparently allow upstream consumers to read from
disk as if it were coming directly from the wire. However this might be
something added on top of the framework WDYT?

Best Regards,
-- Alex

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message