directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <>
Subject Re: [Mavibot] BulkLoad
Date Fri, 20 Jun 2014 15:32:42 GMT
On Fri, Jun 20, 2014 at 6:20 PM, Howard Chu <> wrote:

> Emmanuel L├ęcharny wrote:
>> Hi guys,
>> many thanks Kiran for the OOM fix !
>> That's one step toward a fast load of big database load.
>> The next steps are also critical. We are currently limited by the memory
>> size as we store in memory the DNs we load. In order to go one step
>> farther, we need to implement a system where we can prcoess a ldif file
>> with no limitation due to the available memory.
>> That supposes we prcoess the ldif file by chunks, and once the chuks are
>> sorted, then we process them as a whole, pulling one element from each
>> of the sorted list of DN and picking the smallest to inject it into the
>> BTree.
> Why do you store the DNs in memory? Why are you sorting them?
sorting the DNs with the assumption that given input LDIF may contain
entries in random order
in ApacheDS each entry contains 'entryParentID' attribute linking to it's
parent entry's ID
The DNs are to held in memory briefly until this relationship is built
using the DN and the
generated entryUUIDs.

> --
>   -- Howard Chu
>   CTO, Symas Corp. 
>   Director, Highland Sun
>   Chief Architect, OpenLDAP

Kiran Ayyagari

View raw message