directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Karasulu" <>
Subject [LDAP] [Client] Client side Cursors can help w/ LDAP paging and notification
Date Sun, 23 Mar 2008 01:30:26 GMT
While implementing the Cursor pattern in the JDBM based partition, some
interesting ideas came to mind regarding the new LDAP Client API we've been
working on.  Up to now we have simply defined what Entries look like and
have called it the Entry API instead.  Eventually this will grow into a full
client as we combine more of the pieces together.  JNDI is a less than
optimal API for LDAP.

Cursors are special, in contrast to the NamingEnumerations we use, because
they can be positioned after creation.  Cursors can be positioned using the
beforeFirst(), first(), last() and afterLast() methods at any point during
their use.  Furthermore, Cursors are bidirectional; they can be traversed in
both directions with calls to next() and previous() to navigate results.
Callers can advance a Cursor forward or reverse at any time after creation.

The Partition interface in ApacheDS will soon expose search results by
returning a Cursor<ServerEntry> instead of a NamingEnumeration.  Depending
on the filter used, this is a composite Cursor which leverages partition
indices to position itself without having to buffer results.  This allows
the server to pull entries satisfying the search scope and filter one at a
time, and return it to the client without massive latency, or memory
consumption.  It also means we can process many more concurrent requests as
well as process single requests faster.  In addition a resultant search
Cursor can be advanced or position using the methods described above just by
having nested Cursors based on indicies advance or jump to the appropriate
Index positions.  We already have some of these footprint benefits with
NamingEnumerations, however the positioning and advancing capabilities are
not present with NamingEnumerations.

During the course of this work, I questioned whether or not client side
Cursors would be possible.  Perhaps not under the current protocol without
some controls or extended operations.  Technical barriers in the protocol
aside, I started to dream about how this neat capability could impact
clients.  With it, clients can advance or position themselves over the
results of a search as they like.  Clients may even be able to freeze a
search in progress, by having the server tuck away the server side Cursor's
state, to be revisited later.  For lack of terms I've likened this to a form
of asynchronous bidirectional LDAP search. This would eliminate the need to
bother with paging controls.  It could even be used to eliminate the thread
per search problem associated with persistent search.  OK, let me stop
dreaming and start looking at reality so we can determine if this is even a

So these characteristics of a Cursor have a profound impact on the semantics
of a search operation - not talking about the protocol yet.  I'm referring
to search as seen from the perspective of client callers using the Cursor:
the front end.  As stated search operations can be initiated and shelved to
persist the state of the search by tucking away the Cursor in the connection
session.  A Cursor for a search will automatically track it's position.

However the protocol imposes some limitations on being able to leverage
these capabilities across the network on an LDAP client.  A search request
begins the search, and entry responses are received from the server, until
the server returns a search response done operation which  signals the end
of the search operation.  During this sequence, without creative extended
operations, or controls, there's little the client can do to influence the
entries returned by the server or throttle the rate of return.  Of course
size and time limits can be set on the search request but after issuing the
search, these cannot be altered.  Because the LdapMessage envelop contains a
messageId, and all responses contain the messageId of the request they
correspond to, the protocol allows for multiple requests to be issued in
parallel.  Even if client API's do not allow for it, this is certainly

Although I've long forgotten how the paging control works exactly, I still
have a rough idea: forgive me for my laziness and if I'm missing something.
A control specifies some number of results to return per page, and the
server complies by limiting the search to that number then capping off the
search operation with a search result done.  Cookies in the request and
response controls are used to track the progress, so another search request
for the next page returns the next page rather than initiating the search
from the start.  This breaks a big search up into many smaller search
requests.  This way the client has the ability to intervene in what
otherwise would be a long flood of results in a single large search
operation.  If this page control could also convey information about
positioning, and directionality, along with a page size set to 1, we could
implement client side Cursors with the same capabilities they posses on the
server.  Paging search results effectively has the server tucking away the
search Cursor state into the client session and pulling it out again to
continue.  This is how we would implement this control today (that is if
anyone gets the time to do so :) ).

Persistent search is unrelated however I'd like to explore whether or not
there's some possible synergy/relationship between it and paging.
(Persistent search IMHO is poorly implemented but we can deal with it). It
is intended for receiving change notifications.  A persistent search control
is issued on a search request which may return a bunch of entry responses
for the filter if requested, but the most notable thing is that the search
does not end with a search result done response.  The operation persists to
return entries satisfying the filter along with a response control
containing the change type when entries satisfying the filter change
accordingly.  Clients usually need to assign a thread to listen for such
responses.  Smart clients will use a single thread instead of one per
persistent search.  Even smarter clients will use a single thread to listen
for search responses on persistent search requests and for unsolicited
notifications.  Regardless once the persistent search request is issued,
there's no way the client can stop it until size and time limits are
reached.  These are parameters in the control sent on the search request.
Of course persistent search requests can use a size limit of 1 and clients
can request another persistent search after an event is received to have
more control.  Regardless a change many not occur for a while in which case
fine tuning with the time limit will help.

It's a bit crazy to think what would happen if both these controls are used
together on the same search request.  I guess it all depends on the server
implementation in the end.  Not sure if anyone would even want to do this.
Regardless of the pain it would entail, I think this situation can be
managed to work in the server.  Now where does this lead us tho?  Perhaps
the Cursor interface could be enhanced to support a listener to
asynchronously notify users of changes to the underlying results.  The
Cursor can then be reset (or just repositioned).  Of course this is all
presuming the Cursor was created to traverse results.  Instead a Cursor
might just be created to iterate over only changes, but does this make
sense?  Whatever the answer, at least we can know when underlying results
have changed to invalidate the Cursor on the client.   This is probably a
bullsh*t idea but it's entertaining to think about.

BTW change notifications are probably best implemented as a combination of
search and extended operations through unsolicited notifications.   The
client issues a search request with a control similar to the persistent
search request control.  Instead of 'persisting' the search, the search
returns immediately with a search result done response using a result code
to indicate whether or not the server will honor the request to be notified
of changes.  Then the client is done registering the request to be
notified.  Whenever the server detects changes that satisfy the changeType,
scope and filter of the notification registration, it sends an unsolicited
extended response to the client.  The payload carries information about the
change which took place similar to the way search entry results do with the
response control of persistent search.   The client can issue a
deregistration message to the server to stop receiving these notifications
using an ExtendedRequest.  The server would respond to this with an
ExtendedResponse.  IMO this is a much better mechanism with full control
over the subscription and notification process.


View raw message