directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayyagarikiran <>
Subject Re: [DRS] thoughts about implementation
Date Mon, 02 Feb 2009 11:07:56 GMT

Emmanuel Lecharny wrote:
> Hi,
>>    IMO this marking of journal should happen in the DirectoryService's
>> sync() operation.
> As the sync can occur on a quite big interval of time (let say 15 secs
> by default), if we depend on it to add a checkpoint, that mean we may
> lose 15 secs of modifications. It may be seen as accepteblae, but
> IMHO, the idea is to store logs as fast as possible, and when a
> modification is considered as done, then add a checkpoint. This will
> limitate the potential loss of information.

sorry if my initial description gave a different perspective. The data *will* be written to
journal as soon as an
operation(add/del/mod) is done, the 'mark' operation is just to put a checkpoint in the journal
to tell us that the DIT
data present till this checkpoint is flushed to jdbm files on disk. So a journal *always*
contains the data irrespective
of the state of jdbm file(s).

But I want to put the 'mark' only when jdbm files are synced so that we can figure out which
records weren't flushed to
the jdbm file(s).

>>    A change to the partition's sync() method to call DirectoryService's
>> sync() which intern calls (each) partition's
>>    commit() (a new method) would help. A special flag like 'isDirty' in the
>> partition will allow us to avoid calling
>>    commit() on each partition.
>>    Any other ideas about how best we can maintain a checkpoint/mark after
>> *any* sync operation on DIT data?.
> when the mod is considered done (even if not written into the
> backend), we should consider the operation valid, and then write it on
> the log, and when done, adding a checkpoint. That should be done for
> every modification, whatever it costs.
yep this is the case

> IMO, JDBM should be used to help access to modifications operation,
> but the master table should not contain the real data, but just a
> pointer to another file in which modification are written in a
> sequential way. We just keep an offset into the Mastertable. If
> something goes really wrong, we can rebuild the master table and all
> the indexes from this sequential file.
yeah, it would be great if we can avoid having the complete data in a sequential file
and btw, the howl journal file holds some data in clear text format which might be a security
(yes, we do have cryptography but thats a different thing altogether ;) )

> This can be discussed, I'm just speaking my mind here, not imposing
> any solution.

thanks Emmanuel

Kiran Ayyagari

View raw message