ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject memory-only mode for Ignite indexes
Date Sat, 05 May 2018 00:45:44 GMT
Igniters,

One of the main complaints I hear from users is that whenever the
persistence is turned on, index creation can really slow down the
performance, because of massive amounts of writes to disk. The reason
Ignite is writing indexes to disk is to support fast restarts - nothing
needs to be rebuilt on startup, and Ignite can become operational right
away.

However, as far as I can tell, most users care about faster operations
after the system is started and much less about the startup speed. What if
we added a mode where we do not persist indexes at all? This way data
ingestion and overall throughput will significantly increase (of course, at
the cost of startup type getting longer because we have to rebuild the
indexes).

There are 2 ways to achieve this in Ignite. The simplest way is not mark
index pages dirty in memory, so they will never participate in
check-pointing process. We also have to make sure that index pages never
get evicted form memory. This can be done fairly quickly. The disadvantage
of this approach is that if indexes fill up most of the memory, then it
will be very difficult to find a page to evict, which may hurt the
performance.

The other way is to have a separate in-memory off-heap region for indexes.
This region should never be persisted. It maybe somewhat bigger
refactoring, as we currently do not separate between index and data pages.
However, the advantage of this approach is that this region can be flushed
to disk practically as is during a graceful shutdown of the node, and hence
shorten the restart time.

I think we should start from the 1st approach and then think about the 2nd
one. What do you think?

D.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message