directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Lecharny (JIRA)" <j...@apache.org>
Subject [jira] (DIRSERVER-2116) ApacheDS failed to start after every reboot and throwing error ERR_250_ENTRY_ALREADY_EXISTS dc=example,dc=com already exists!
Date Tue, 31 Jan 2017 09:54:43 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml"> 
    <head> 
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
        <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"
/> <base href="https://issues.apache.org/jira" /> 
        <title>Message Title</title> 
    </head> 
    <body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px;
line-height: 1.429"> 
        <table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
            <!-- header here --> 
            <tr> 
                <td id="header-pattern-container" style="padding: 0px; border-collapse:
collapse; padding: 10px 20px"> 
                    <table id="header-pattern" cellspacing="0" cellpadding="0" border="0"
style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                        <tr> 
                            <td id="header-avatar-image-container" valign="top" style="padding:
0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px">
<img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-elecharny-785b4707-bafe-4ca1-9bc1-45eb330bfc46"
height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" /> 
                            </td> 
                            <td id="header-text-container" valign="middle" style="padding:
0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover"
rel="elecharny" id="email_elecharny" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=elecharny"
style="color:#3b73af;; color: #3b73af; text-decoration: none">Emmanuel Lecharny</a>
<strong>commented</strong> on <a href="https://issues.apache.org/jira/browse/DIRSERVER-2116"
style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-static-bug-836a1517-1702-4a79-8860-c4c0237aac96"
height="16" width="16" border="0" align="absmiddle" alt="Bug" /> DIRSERVER-2116</a>

                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="email-content-container" style="padding: 0px; border-collapse:
collapse; padding: 0 20px"> 
                    <table id="email-content-table" cellspacing="0" cellpadding="0" border="0"
width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt;
border-spacing: 0; border-collapse: separate"> 
                        <tr> 
                            <!-- there needs to be content in the cell for it to render
in some clients --> 
                            <td class="email-content-rounded-top mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color:
#fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc;
border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px;
line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table class="page-title-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt"> 
                                    <tr> 
                                        <td style="vertical-align: top;; padding: 0px;
border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule:
exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header"
style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise:
2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.apache.org/jira/browse/DIRSERVER-2116"
style="color: #3b73af; text-decoration: none">Re: ApacheDS failed to start after every
reboot and throwing error ERR_250_ENTRY_ALREADY_EXISTS dc=example,dc=com already exists!</a>
</span> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td id="text-paragraph-pattern-top" class="email-content-main
mobile-expand  comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left:
1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0
15px 0 16px; background-color: #fff; border-bottom: none; padding-bottom: 0"> 
                                <table class="text-paragraph-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 2px"> 
                                    <tr> 
                                        <td class="text-paragraph-pattern-container mobile-resize-text
" style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> 
                                            <p style="margin: 10px 0 0 0">Dear Steve,</p>

                                            <p style="margin: 10px 0 0 0">we knows that
is frustrating. If it were simple, it would have been fixed 3 years ago. Ok,that does not
help :/ Let me give you a bit of background for you to understand the complexity of this issue.</p>

                                            <p style="margin: 10px 0 0 0">LDAP update
requests ends up with B-tree being updated, many B-trees. For even a single attribute being
changed in an existing entry. For instance, when you add an entry, we have to update all those
indexes (B-trees) :</p> 
                                            <ul> 
                                                <li>
                                                    master table which contains the entry
itself (but its DN)
                                                </li> 
                                                <li>
                                                    RDN index - which will be updated as many
times as we have RDNs in the DN, so if your entry is 'cn=johnDoe,ou=people,dc=example,dc=com',
with 'dc=example,dc=com' being your partition's suffix, we will update the index 3 times,
as the suffix is considered as one single RDN -
                                                </li> 
                                                <li>
                                                    present index, which is used when you
use filters like '(cn=*)'
                                                </li> 
                                                <li>
                                                    ObjectClass index
                                                </li> 
                                                <li>
                                                    entryCSN index (used for replication)
                                                </li> 
                                                <li>
                                                    entryUUID index (each entry has a unique
UUID across all the partitions)
                                                </li> 
                                            </ul> 
                                            <p style="margin: 10px 0 0 0">and potentially
more indexes, depending on the entry's content : alis, oneAlias, subAlias, administrativeRole.
Add to that all the user's defined indexes.</p> 
                                            <p style="margin: 10px 0 0 0">That being
said, you can understand that all of those index <b>MUST</b> be updated as a whole,
or none of them should be written. We do need transactions for that purpose, and we don't
have transaction, yet. Well, kind of.</p> 
                                            <p style="margin: 10px 0 0 0">JDBM, the
main b-tree implementation we are binding <tt>ApacheDS</tt> with since day one
has a very poor support for transactions, and worse than that, it does not support cross-b-tree
transactions. So it ends up with a system that may break if we have a crash, or concurrent
updates.</p> 
                                            <p style="margin: 10px 0 0 0">We have spent
months trying to get this fix in a ugly way (adding locks all over the server, trying to leverage
JDBM transaction to some extent, addinga repair mode...), but at the end, we knew that we
would need something different. Using BerkeleyDB JE wasn't an option - even though Kiran wrote
a partition for it -, for a simple reason : incompatible licenses...</p> 
                                            <p style="margin: 10px 0 0 0">So to put
things in perspective, here is what we know we should do, and we knew it back in 2006 (yes,
11 years ago) for different reasons : we need a MVCC system that support cross b-tree transactions.
It is what Mavibot was created for in 2012 (bottom line, we tried to implement MVCC on top
of JDBM, which was not a real success). Here is a ref to the mail where we discussed about
Mavibot : <a href="https://lists.apache.org/thread.html/72d9ba2fc6b567471464c385cdfcf80bf2882cd5e7825e3fa5bdc773@1340293586@%3Cdev.directory.apache.org%3E"
class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://lists.apache.org/thread.html/72d9ba2fc6b567471464c385cdfcf80bf2882cd5e7825e3fa5bdc773@1340293586@%3Cdev.directory.apache.org%3E</a></p>

                                            <p style="margin: 10px 0 0 0">For the record,
we talked about MVCC in 2006 at Austin, where CouchDB was presented. Alex new we we going
to need transaction in the server, and I realized that MVCC was the way to go for LDAP (concurrent
reads with no locks, one single writer).</p> 
                                            <p style="margin: 10px 0 0 0">As a matter
of fact, OpenLDAP and more specifically Howard Chu started to work on LMDB, which would become
the de facto backend for OpenLDAP back in 2011. That was funny, because we didn't heard about
it before the end of 2012... LMDB is exactly what Mavibot will be : MVCC with cross-b-tree
transactions (exept that LMDB is a bit more than that, typically it's based on a Memory Mapped
File and does not use cache, something we have to use in Mavibot, for the simple reason that
serializing/deserializing objects in Java is way more complex than mapping a C struct on top
of a byte[], and it's also way slower.</p> 
                                            <p style="margin: 10px 0 0 0">That being
said, writing a new backend is not an easy thing. The very first version was released on 06/Jun/2013
(<a href="http://directory.apache.org/mavibot/download-old-versions.html" class="external-link"
rel="nofollow" style="color: #3b73af; text-decoration: none">http://directory.apache.org/mavibot/download-old-versions.html</a>),
and it didn't had transaction support. Although it does work, and it is supported by ApacheDS.
It even gives ApacheDS performance a real boost, with a 5x performance increase when it comes
to updates.</p> 
                                            <p style="margin: 10px 0 0 0">So why didn't
we make Mavibot the default database ? Well, simply because it was not ready for production
:</p> 
                                            <ul class="alternate" type="square"> 
                                                <li>
                                                    the 'reclaimer' (see it as the database
GC) was not functional yet. That makes the database growing very quickly, making it unusable
in production - we are talking about Gb of data being accumulated for a few thousands of entries
-
                                                </li> 
                                                <li>
                                                    transaction weren't implemented properly
                                                </li> 
                                            </ul> 
                                            <p style="margin: 10px 0 0 0">Bottom line,
a new version of Mavibot is being design, and as a mater of fact, I rebooted the effort 3
weeks ago, and the current status is work in progress. Adding transactions is challenging,
but the result is astonishing. Not that it makes the database safe against crashes - which
is by design in MVCC anyway -, but because it makes LDAP data safe against a crash, even if
it's in the middle of an update. But it also speed up updates enormously, as we don't write
anything on disk before the txn is completed, saving potentially 70% of the writes needed
for an update. That has the potential of speeding up ApacheDS by a factor 3, for updates (reads
will not be impacted, well, sort of).</p> 
                                            <p style="margin: 10px 0 0 0">As I said,
reads will not be impacted, but it's not exactly true. The current ApacheDS implementation
uses locks that block reads when a write is done (same story : trying to make the dabase safe
against concurrent access). Having MVCC with transaction would mean we can remove all thse
stupid locks, speeding up reads, too (but to a lesser extend).</p> 
                                            <p style="margin: 10px 0 0 0">One last advantage
: with Mavibot, we already have a bulk loader that can be use to create a database without
having to load the entries one by one in ApacheDS. That means you will be able to inject 1M
entries in a matter of seconds, instead of hours (or even days..).</p> 
                                            <p style="margin: 10px 0 0 0">Ok, now, were
are we ? Not so far, but not ready yet. Even if I complete the Mavibot transaction implementation
next week, we will need a lot of checks to be sure it's safe. It might take weeks, months.
</p> 
                                            <p style="margin: 10px 0 0 0">But this is
not the only problem : I do have a day job, and a family. I don't sleep a lot, but still,
it's hard to work 1 hour on this proect between midnight and 1am in the morning, and to be
efficient. Currently, I'm mainly programming on this stuff during week-ends.</p> 
                                            <p style="margin: 10px 0 0 0">Mavibot is
not such a big piece of code (25 000 slocs), but it's a complex one. Integrating it in ApacheDS
is not exatcly a piece of cake, but it's already partially done. Also keep in mind that ApacheDS
is quite a big baby : 200 000 slocs for the server, 200 000 slocs for the LDAP API which it
uses, 45 000 slocs for MINA, the underlying NIO framework, not mentionning the Studio's 200
000 slocs . We are facing an nearly 1 million slocs project...<br /> It takes a hell
of a time to correctly work this out in a way it does not break everything.</p> 
                                            <p style="margin: 10px 0 0 0">Last, not
least, I'm doing this for the fun of it, and EVERYBODY is very welcome to join the effort,
especially those depending on this piece of software. I mean, it's open source, and if it's
broken, you can help fixing it. Whining does not really help, but I can understand the exasperation
one can feel when a critical bug is postponed version after version...</p> 
                                            <p style="margin: 10px 0 0 0">So keep faith,
and if you have time and energy, feel free to join teh effort !</p> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table id="actions-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 1px"> 
                                    <tr> 
                                        <td id="actions-pattern-container" valign="middle"
style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align:
middle; padding-left: 0"> 
                                            <table align="left" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                                                <tr> 
                                                    <td class="actions-pattern-action-icon-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align:
middle"> <a href="https://issues.apache.org/jira/browse/DIRSERVER-2116#add-comment"
target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img
class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-538f9fce-eb3e-4928-9d10-2b21ae0ef092"
alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align:
middle" /> </a> 
                                                    </td> 
                                                    <td class="actions-pattern-action-text-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left:
5px"> <a href="https://issues.apache.org/jira/browse/DIRSERVER-2116#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>

                                                    </td> 
                                                </tr> 
                                            </table> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <!-- there needs to be content in the cell for it to render in
some clients --> 
                        <tr> 
                            <td class="email-content-rounded-bottom mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height:
5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px
solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius:
5px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="footer-pattern" style="padding: 0px; border-collapse: collapse;
padding: 12px 20px"> 
                    <table id="footer-pattern-container" cellspacing="0" cellpadding="0"
border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">

                        <tr> 
                            <td id="footer-pattern-text" class="mobile-resize-text" width="100%"
style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height:
18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
                                 This message was sent by Atlassian JIRA <span id="footer-build-information">(v6.3.15#6346-<span
title="dbc023dd75cecacf443c4b235f66124b15f5c5fe" data-commit-id="dbc023dd75cecacf443c4b235f66124b15f5c5fe}">sha1:dbc023d</span>)</span>

                            </td> 
                            <td id="footer-pattern-logo-desktop-container" valign="top"
style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">

                                <table style="border-collapse: collapse; mso-table-lspace:
0pt; mso-table-rspace: 0pt"> 
                                    <tr> 
                                        <td id="footer-pattern-logo-desktop-padding" style="padding:
0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop"
src="cid:jira-generated-image-static-footer-desktop-logo-e318c394-e38a-4eb7-add5-16e908284e30"
alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />

                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
        </table>   
    </body>
</html>
Mime
View raw message