directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wes McKean" <>
Subject Re: Re: [asn.1] Decoder service design ( was: Paper on original SNACC)
Date Fri, 23 Jan 2004 15:15:44 GMT
That's a good question.  My guess is that there would be a pool of threads doing the decoding,
like the current architecture.  Is it safe to assume that it is OK for a thread to block while
waiting for a complete ASN.1 messaeg?  I think so.

I've been giving this alot of thought.  It would be easy to parse the incomming message into
a TLV tree.  The parser could do this without having to know *what* the message is.  This
might make it a lot easier to encode a state driven parser/decoder.  Once the complete message
is parsed, it could then be passed off to something that would try to understand.  I think
this approach will work fine, and it would fulfill our desire to have a non-blocking decoder.

What do you think?


---------- Original Message ----------------------------------
From: Alex Karasulu <>
Reply-To: "Apache Directory Developers List" <>
Date:  Fri, 23 Jan 2004 9:09:20 -0500

>How many threads are used while decoding a message in toto?
>> From: "Wes McKean" <>
>> Date: 2004/01/22 Thu PM 12:11:40 EST
>> To: "Apache Directory Developers List" <>
>> Subject: Re: [asn.1] Decoder service design ( was: Paper on original SNACC)
>> TLV = Tag Length Value, the basic ASN.1 construct.  I'm edumacated :-)
>> I'm gonna read this in more detail later, but I wanted to respond right away to the
question of streams versus ByteBuffers...
>> What I did in my stuff, for testing, was create a sink.  The sink acted like an InputStream
for the thread reading the data, and it was a non-blocking repository for the thread doing
the reading.  In other words...
>> The reading thread reads byte buffer, finds the sink for the client connection, and
ads the byte buffer to its queue.  done.
>> The thread doing the decoding reads from the input stream, and blocks when it runs
out of data.  When the reader thread drops another byte buffer into the sink/queue, the decoder
thread takes off again.
>> This is a lot simpler than writing a decoder that works like a state machine.  It
kind of hides the source of the input, and allows the decoder to deal with a stream.
>> Not saying this is the way to go, but this was my idea from earlier...
>> Wes
>> ---------- Original Message ----------------------------------
>> From: Alex Karasulu <>
>> Reply-To: "Apache Directory Developers List" <>
>> Date:  Thu, 22 Jan 2004 11:10:32 -0500
>> >Wes,
>> >
>> >> From: "Wes McKean" <>
>> >> I don't have permission to access the snacc.pdf.  What do I do?
>> >
>> >Sorry dude I just set the perms to allow access.  Give it another 
>> >try.
>> >
>> >> I'm going to go ahead and get started on the BER encoder/decoder.  For now,
>> >> I will keep the interface the same as the current one.  Namely, the
>> >> encoder/decoder will read from and write to a stream.  The primary
>> >> difference is I will detect non-blocking streams.  If anyone has any idea
>> >> this, let me know.  It might truly be difficult to implement the decoder
>> >> non-blocking, if that means decoding TLVs in chunks.  
>> >
>> >TLVs? Forgive me I'm bad with acronyms.  Oh and another thing take a look at
>> >the new front end stuff in eve/frontend/common/api where there are events 
>> >defined.  Namely take a look at the InputEvent which carries with it a 
>> >direct nio buffer.  Look towards defining your interfaces around this.
>> >
>> >> That would require the decoder to almost be like a state machine.  
>> >> If we decide that is the route to go, then fine with me.  For getting 
>> >> started, I will use just a regular old stream.
>> >
>> >I've given limited time to thinking about this one.  Basically we have
>> >chunks of data that will be delivered over to the decoder in the form
>> >of input events associated with a client.  The InputEvent (caught a bug)
>> >should extend the ClientEvent (which it currently does not - my bad!) 
>> >which has the ClientKey associated with it.  The InputEvent is a SEDA
>> >event.
>> >
>> >So what you do with the buffer content is upto you but here are some
>> >thoughts.  I'm thinking out loud here.  
>> >
>> >First of let me state a couple things about the existing state of 
>> >the frontend.  Don't worry about manipulating the byte buffer with 
>> >a concern for effecting other peices of code that work with the same 
>> >input event.  When you access the buffer you really have a read only 
>> >copy of the buffer which is backed by the direct buffer allocated 
>> >from the buffer pool.  Meaning you can advance and read and do a 
>> >bunch o stuff to change where you are in the buffer but you cannot 
>> >affect the content to mess with other code that may also have the 
>> >event delivered to it.  So advancing in the buffer will not throw of
>> >other listeners that have the event delivered to them.
>> >
>> >You got the buffer.  What you do with it is upto you.  
>> >
>> >Now the decoder service may eventually be used to decode more than
>> >just input comming in from a client stream down the road.  Just wanted
>> >to mention this but I don't think you should factor it into what you're
>> >dealing with right now.  This is for decoding stuff through the LdapContext
>> >JNDI interface which still use encoded ASN.1 for Controls.  I'll deal
>> >with this later but just thought I'd mention it.
>> >
>> >Ok we want to basically not block or rather dedicate a thread while 
>> >processing an entire message.  That would mean two threads used per
>> >request.  One thread to deliver the event, and another to read and 
>> >decode it.  So this is the main problem.
>> >
>> >Perhaps we can setup a small framework here inside the decoder with
>> >a callback.  When a new message is detected, a non-direct buffer can 
>> >be assigned to span the time required to decode the request.  This
>> >buffer collects the data coming in from the direct buffer in the
>> >InputEvent.  The input buffers are copied into the collector.  Now
>> >if the decoder was a state machine then we would not have to do a
>> >copy right?  The state machine based decoder could read from the 
>> >direct buffer and between chunks freeze its state until the next
>> >chunk appears.  That would be the best approach as you mentioned but
>> >I don't know how complex that would be.  And when processing a chunk
>> >the state machine should return even when it has not reached a 
>> >terminal state.
>> >
>> >Now if we continue on the less efficient collector buffer approach 
>> >then you need to have the decoder detect the start of a new PDU and
>> >the end of a PDU (Protocol Data Unit an ASN.1 message envelope).  It 
>> >must detect the start of the PDU to create and assign a collector 
>> >buffer to the current request.  It must detect the end of the PDU
>> >to basically trigger the true decoding machinery to slurp up the
>> >content in the collector buffer and decode it which can be done in
>> >one step and can be blocking.  You can use 1.4 regex functionality
>> >to detect the PDU start sequence and the end sequence of bytes 
>> >comming in as you transfer bytes from the direct buffer into the
>> >collector buffer.  Again this is sounding convoluted too.
>> >
>> >What are your thoughts?  There has got to be a better way perhaps
>> >we just need to sleep on it.
>> >
>> >Alex
>> >
>> >
>> >
>> >
>> >
>> >
>> >

View raw message