ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Goncharuk <agoncha...@apache.org>
Subject Changes in data loader approach
Date Wed, 04 Feb 2015 07:58:51 GMT
Hello fellow Igniters,

I was thinking about reworking the isolated mode of Ignite cache data
loader:

In the majority of use-cases isolated mode of data loader is used for
initial cache load. This mode requires that there are no concurrent cache
updates happening during the load process. Nevertheless, in transactional
caches the performance is still not good enough because updates are still
running in transactional manner.

I suggest to slightly change the semantics of isolated mode and make it
provide the same guarantees that loadCache() method provides. Namely,
non-null value will be set to a cache entry if the entry is new (i.e. has
start version)

In this case:

   - Load be run in non-transactional manner and will be very fast
   - Load will be consistent with any ongoing cache update regardless
   whether it is a transactional or an atomic cache
   - Load will not overwrite or delete any existing cache data

Because of the latter property I think it makes sense to rename 'isolated'
property to 'allowOverwrite' and make it false by default.

Quick testing shows that with this change Ignite will be able to load 200M
entries in ~10 minutes (even faster with off-heap).

Any thoughts or feedback is much appreciated.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message