mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Google Protocol Buffer codec...
Date Fri, 09 Jan 2009 07:22:03 GMT
On Thu, Jan 8, 2009 at 10:55 PM, Tomasz Blachowicz
<tblachowicz@gmail.com> wrote:
> On Thu, Jan 8, 2009 at 6:16 AM, Ashish <paliwalashish@gmail.com> wrote:
>> Some observation
>> 1. the sandbox folder has whole of MINA code. Will be great if we can
>> remove other files
> The filter codec I've develop to handle Google Protocol Buffers essentially
> lives in his own module mina-filter-codec-protobuf and depends only on
> mina-core module. So, if you want to focus only on the code I've
> contributed, please consider mina-filter-codec-protobuf  module.

[ashish] This is fine :-)

>> AFAIK, protocol buffer codec is a binary data binding framework,
>> something similar to XML-Data binding framework's like JAXB, Castor,
>> XMLBeans. Google's implementation reminds of CORBA IDL's
>> There is basic operational similarity of Google Protocol Buffer codec
>> with prefixed string decoder. We have a length and then actual
>> messages. It will be worthwhile to explore if we can align the google
>> codec with prefixed string codec.

> Yes, the prefixed string codec is similar to the one I've developed for
> protobuf messages. However there are two things to be considered before we
> would start any alignment:
> 1. Prefixed string codec has been implemented in mina-core module and
> protobuf codec resides in it's own *optional* module.

[ashish] Well alignment is desirable, its good to have similar codes
behave in same way and coded in same pattern.
Its easy for Users to understand.

> 2. Prefixed string codec uses fixed format to encode length of the message
> as protobuf codec uses varint way to encode the integer values (
> http://code.google.com/apis/protocolbuffers/docs/encoding.html)<http://code.google.com/apis/protocolbuffers/docs/encoding.html>.
> This way of encoding is pretty nifty and can save bandwidth if you are
> sending many lightweight messages. For instance value less than 128 requires
> only two bytes to encode and values less than 16K requires 3. What is more,
> prefixed string codec uses method on IoBuffer implementation to determine
> the size, while protobuf codec determines the size internally using varints
> encoding.

[ashish] The idea is u get length before message, how u determine
length can vary. BER encoding has TLV encoding,
something similar. Well not actually questioning how Google's protocol behaves.

> There is one more situation that must be though about is piggybacked
>> messages. the implementation can receive a packet with one message
>> ending and another message starting (again by nature of TCP). So
>> returning true from decoder may result in an exception (more data
>> remaining) [Please correct me if I am wrong]
> This is not a problem as when the decoder receives a buffer of say 64 bytes
> and reads only 50 of them to decode the message (and returns true and then
> the framework will present remaining 14 bytes in the next call to the decode
> method. Please refer to the test cases I've developed.

[ashish] Well it was more towards how CumulativeProtocolDecoder works.
I thought if you have more data in buffer and you return true from
doDecode(), it shall throw an Exception. Seems my understanding is
incorrect, the documentation states "if there is any data left that
cannot be decoded, we store it in a buffer in the session and next
time this decoder is invoked the session buffer gets appended to".
Ignore my comment. Will have to rectify my implementation :-)

>> Also, is it worthwhile to have a mechanism to add extensions into
>> ExtensionRegistry? or the library is doing it.
> I spent some time playing around with Google Protocol Buffers and in my
> opinion the extensions is one of the strongest features of the library.
> Because of the fact that the protobuf is not self-descriptive protocol
> (as opposite to Java serialization) the meta data about the message carried
> over the wire is very limited and have to be provided to the decoder by
> the application that consumes the messages. However I'm open to all the
> advices and suggestions how we might do it better :)

[ashish] Well Here it more usability perspective. Again don't
understand the mechanism completely. It more or less like you know
what objects you are going to decode for your application. So you
preload them some how. In MPXJ, lib adds appropriate decoder in a map
and returns the requested one. So we could replicate the same, if this
is how protocol buffer work :-)

>> Would like to spend some more time with the code before commenting
>> more on this. Though it will be worthwhile to have this codec with us.
>> Once we review the code, lets vote for the same.
> Any comments are welcome and highly appreciated!

[ashish] This is more for me, will learn from your code :-)

>> Shall try to run this implementation and validate it against my
>> possible use case. Though would have loved to have an implementation
>> where I don't have to define .proto files. (I know the implementation
>> doesn't work that ways)
> Well, in my opinion it'd be madness to write the Java class for the message
> by hand. What I can do is I can provide you with the maven code snippet that
> will allow you automatically generate Java code from .proto files while you
> build the application. How does it sound?

[ashish] Hmm not convinced on this. My targets are to create DHCP,
SMPP, DNS and TFTP servers. Now Google's implementation doesn't help
me here in any ways. This was the primary reason why I didn't went
further into buffers. Anyways that's off topic.

For MINA, having this codec is a big plus :-)

View raw message