mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan D. Cabrera" <l...@toolazydogs.com>
Subject Re: IoBuffers of bytebuffers
Date Sat, 20 Aug 2011 15:27:01 GMT

On Aug 20, 2011, at 3:59 AM, Alex Karasulu wrote:

> On Sat, Aug 20, 2011 at 2:46 AM, Emmanuel Lecharny <elecharny@gmail.com>wrote:
> 
>> On 8/19/11 2:26 PM, Alan D. Cabrera wrote:
>> 
>>> On Aug 19, 2011, at 5:19 AM, Emmanuel Lecharny wrote:
>>> 
>>> On 8/19/11 2:16 PM, Alan D. Cabrera wrote:
>>>> 
>>>>> I'm wondering.  Do you guys think it's a good idea?  It seems to make
>>>>> things pretty complicated and adds another dimension to groking the behavior
>>>>> of your service.  I'm not sure that it's necessary.
>>>>> 
>>>> Definitively a bad idea.
>>>> 
>>>> What we need is an abstraction on top of an Array of ByteBuffer (the Java
>>>> NIO class), which extends the size by adding new ByteBuffer on the fly.
>>>> 
>>>> The array must behave exactly as the ByteBuffer.
>>>> 
>>>> I wrote such a class 2 years ago, but I can't find the code. Will look
>>>> again on my USB keys this week-end.
>>>> 
>>> So, what is the scenario that we're trying to support?  I imagine
>>> appending headers to binary data would be one.  In this case is an Array of
>>> ByteBuffers really needed?  Why not just send down one ByteBuffer for the
>>> header and another for the body that was sent to you?
>>> 
>> 
>> Julien and Steve laready responded with clear examples where expandable BB
>> are necessary. I do think this is a common case, assuming you can perfectly
>> receive the data byte by byte, and you want to store those bytes in a place
>> which does not need to be reallocated every time.
>> 
>> The current IoBuffer, when full, is copied in a new loBuffer which size is
>> doubled. If you are expecting big PDUs, you might allocate huge Iobuffer for
>> nothing, wasting space. Plus you have to copy all the old IoBuffer into the
>> new one. Not really a good thing.
>> 
>> *if* we are using DirectBuffer, we even might want to use fixed size
>> IoBuffer (say 1k), and store them in a queue of avilable free buffers for
>> reuse, sparing the cost of allocating them (to be validated, I'm not sure
>> that the cost of managing concurrent access to this queue does not
>> overweight the cost of allocating a new direct buffer)
>> 
>> For HeapBuffers, i'm quite sure that we should allocate new ByteBuffer with
>> the exact size of the received data (if it's 10 bytes, then the new
>> ByteBuffer will be 10 byte slong. If it's bigger, then fine). This
>> additional ByteBuffer can be appended a the end of the ByteBuffer array, and
>> the filter will be able to process the list when it has everything needed.
>> 
>> Julien mentionned the Cumulative decoder, which can be used in may cases :
>> - we are waiting for a \n (HTTP protocol)
>> - we are waiting for a closing tag (XML)
>> - we are dealing with LV (Length/value) PDU, and the received value is not
>> fully received (we are expecting Length bytes)
>> - we are dealing with fixed size PDU
>> 
>> One more thing to consider : at some point, we may want to flush the
>> ByteBuffer to disk, if we are waiting for a huge PDU, to avoid sucking all
>> the memory : this is typically the case in LDAP when receiving a JpegPhot
>> attribute, inside a PDU (we don't know that it's a JpegPhoto attribute until
>> we have read and processed teh full PDU). With a proxy on top of the
>> ByteBuffer, we can eventually hide the fact that the bytes are on disk.
> 
> 
> Although these PDU's are large you can store arbitrary data as well, who
> knows what kinds of data and what scenarios users will come up with. It
> would be very nice if MINA allowed some way behind the scenes to stream
> large PDU's to disk and transparently allow upstream consumers to read from
> disk as if it were coming directly from the wire. However this might be
> something added on top of the framework WDYT?


This is my thinking as well.  I see a need to batch together byte buffers to be more efficient
when TCP_NO_DELAY has been set but the scenarios listed above can be implemented without a
more complex byte buffer, i.e. the byte data should be translated to higher order data structures
opportunistically.


Regards,
Alan


Mime
View raw message