directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <>
Subject Re: Re: [asn.1] Decoder service design ( was: Paper on original SNACC)
Date Thu, 01 Jan 1970 00:00:00 GMT

How many threads are used while decoding a message in toto?


> From: "Wes McKean" <>
> Date: 2004/01/22 Thu PM 12:11:40 EST
> To: "Apache Directory Developers List" <>
> Subject: Re: [asn.1] Decoder service design ( was: Paper on original SNACC)
> TLV = Tag Length Value, the basic ASN.1 construct.  I'm edumacated :-)
> I'm gonna read this in more detail later, but I wanted to respond right away to the question
of streams versus ByteBuffers...
> What I did in my stuff, for testing, was create a sink.  The sink acted like an InputStream
for the thread reading the data, and it was a non-blocking repository for the thread doing
the reading.  In other words...
> The reading thread reads byte buffer, finds the sink for the client connection, and ads
the byte buffer to its queue.  done.
> The thread doing the decoding reads from the input stream, and blocks when it runs out
of data.  When the reader thread drops another byte buffer into the sink/queue, the decoder
thread takes off again.
> This is a lot simpler than writing a decoder that works like a state machine.  It kind
of hides the source of the input, and allows the decoder to deal with a stream.
> Not saying this is the way to go, but this was my idea from earlier...
> Wes
> ---------- Original Message ----------------------------------
> From: Alex Karasulu <>
> Reply-To: "Apache Directory Developers List" <>
> Date:  Thu, 22 Jan 2004 11:10:32 -0500
> >Wes,
> >
> >> From: "Wes McKean" <>
> >> I don't have permission to access the snacc.pdf.  What do I do?
> >
> >Sorry dude I just set the perms to allow access.  Give it another 
> >try.
> >
> >> I'm going to go ahead and get started on the BER encoder/decoder.  For now,
> >> I will keep the interface the same as the current one.  Namely, the
> >> encoder/decoder will read from and write to a stream.  The primary
> >> difference is I will detect non-blocking streams.  If anyone has any idea on
> >> this, let me know.  It might truly be difficult to implement the decoder as
> >> non-blocking, if that means decoding TLVs in chunks.  
> >
> >TLVs? Forgive me I'm bad with acronyms.  Oh and another thing take a look at
> >the new front end stuff in eve/frontend/common/api where there are events 
> >defined.  Namely take a look at the InputEvent which carries with it a 
> >direct nio buffer.  Look towards defining your interfaces around this.
> >
> >> That would require the decoder to almost be like a state machine.  
> >> If we decide that is the route to go, then fine with me.  For getting 
> >> started, I will use just a regular old stream.
> >
> >I've given limited time to thinking about this one.  Basically we have
> >chunks of data that will be delivered over to the decoder in the form
> >of input events associated with a client.  The InputEvent (caught a bug)
> >should extend the ClientEvent (which it currently does not - my bad!) 
> >which has the ClientKey associated with it.  The InputEvent is a SEDA
> >event.
> >
> >So what you do with the buffer content is upto you but here are some
> >thoughts.  I'm thinking out loud here.  
> >
> >First of let me state a couple things about the existing state of 
> >the frontend.  Don't worry about manipulating the byte buffer with 
> >a concern for effecting other peices of code that work with the same 
> >input event.  When you access the buffer you really have a read only 
> >copy of the buffer which is backed by the direct buffer allocated 
> >from the buffer pool.  Meaning you can advance and read and do a 
> >bunch o stuff to change where you are in the buffer but you cannot 
> >affect the content to mess with other code that may also have the 
> >event delivered to it.  So advancing in the buffer will not throw of
> >other listeners that have the event delivered to them.
> >
> >You got the buffer.  What you do with it is upto you.  
> >
> >Now the decoder service may eventually be used to decode more than
> >just input comming in from a client stream down the road.  Just wanted
> >to mention this but I don't think you should factor it into what you're
> >dealing with right now.  This is for decoding stuff through the LdapContext
> >JNDI interface which still use encoded ASN.1 for Controls.  I'll deal
> >with this later but just thought I'd mention it.
> >
> >Ok we want to basically not block or rather dedicate a thread while 
> >processing an entire message.  That would mean two threads used per
> >request.  One thread to deliver the event, and another to read and 
> >decode it.  So this is the main problem.
> >
> >Perhaps we can setup a small framework here inside the decoder with
> >a callback.  When a new message is detected, a non-direct buffer can 
> >be assigned to span the time required to decode the request.  This
> >buffer collects the data coming in from the direct buffer in the
> >InputEvent.  The input buffers are copied into the collector.  Now
> >if the decoder was a state machine then we would not have to do a
> >copy right?  The state machine based decoder could read from the 
> >direct buffer and between chunks freeze its state until the next
> >chunk appears.  That would be the best approach as you mentioned but
> >I don't know how complex that would be.  And when processing a chunk
> >the state machine should return even when it has not reached a 
> >terminal state.
> >
> >Now if we continue on the less efficient collector buffer approach 
> >then you need to have the decoder detect the start of a new PDU and
> >the end of a PDU (Protocol Data Unit an ASN.1 message envelope).  It 
> >must detect the start of the PDU to create and assign a collector 
> >buffer to the current request.  It must detect the end of the PDU
> >to basically trigger the true decoding machinery to slurp up the
> >content in the collector buffer and decode it which can be done in
> >one step and can be blocking.  You can use 1.4 regex functionality
> >to detect the PDU start sequence and the end sequence of bytes 
> >comming in as you transfer bytes from the direct buffer into the
> >collector buffer.  Again this is sounding convoluted too.
> >
> >What are your thoughts?  There has got to be a better way perhaps
> >we just need to sleep on it.
> >
> >Alex
> >
> >
> >
> >
> >
> >
> >

View raw message