directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel L├ęcharny <>
Subject [Mavibot] Update
Date Mon, 23 Jan 2017 09:12:49 GMT
Hi guys,

just to inform you that this week-end, I spent time on refactoring the
mavibot code base to implement transaction support. Here is a quick
heads up :

- every read and write will be done in a transaction

- we will have Read transactions and Write trasactions

- only one write transaction can run, the other will have to wait for
the previous one's completion

- I will remove the multi-value support from the core code : it will
have to be handled by the application

The whole idea is that many updates may be applied during one write
transaction, with all the modified pages being hold in memory until the
commit. That will save us a hell lot of disk writes (I woud assume that
nearly 60 to 70 % of the writes will be avoided, speeding up the write
rate by the same amount).

The consequence is that every function will have to take an extra
parameter, a Transaction.

The code should be ready soon, it's not that complex.

The next step will be to implement the reclaimer on top of the
transaction system. I still have to see if this will be the last task
when doing an update (but the pb is that we may wait a long time with
dead pages pending), or if we need a dedicated thread for this (as we do
in the current implementation). All in all, this should not be
complicated, as we already have it working. The only thing that might be
reworked is the algorithm in charge to find the pages that can be
reclaimed. The brutal algorithm just reclaim pages that are older that
the oldest revision in use, a more subtile, but more complex, algorithm
would be able to reclaim pages when a revision is released, even if we
still have older reads going on - we discussed that with Kiran two years
ago... -

The last step will be to implement a correct cache system, because we
currently have to deserialize pages when we fetch them by their offset.
I still have to think about it, but it's an improvement of a working
system, so it may come after the current work (even if it will be
absolutely required on a working system, in rder to have decent
performances. But correctness comes first !)

I hope I can have a running version soon, but I can't set a deadline,
being quite busy atm.

I'll keep you posted.

Thanks !

Emmanuel Lecharny

View raw message