ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Lantukh <ilant...@gridgain.com>
Subject Rebalancing - how to make it faster
Date Thu, 22 Mar 2018 15:38:33 GMT
Igniters,

I've spent some time analyzing performance of rebalancing process. The
initial goal was to understand, what limits it's throughput, because it is
significantly slower than network and storage device can theoretically
handle.

Turns out, our current implementation has a number of issues caused by a
single fundamental problem.

During rebalance data is sent in batches called
GridDhtPartitionSupplyMessages. Batch size is configurable, default value
is 512KB, which could mean thousands of key-value pairs. However, we don't
take any advantage over this fact and process each entry independently:
- checkpointReadLock is acquired multiple times for every entry, leading to
unnecessary contention - this is clearly a bug;
- for each entry we write (and fsync, if configuration assumes it) a
separate WAL record - so, if batch contains N entries, we might end up
doing N fsyncs;
- adding every entry into CacheDataStore also happens completely
independently. It means, we will traverse and modify each index tree N
times, we will allocate space in FreeList N times and we will have to
additionally store in WAL O(N*log(N)) page delta records.

I've created a few tickets in JIRA with very different levels of scale and
complexity.

Ways to reduce impact of independent processing:
- https://issues.apache.org/jira/browse/IGNITE-8019 - aforementioned bug,
causing contention on checkpointReadLock;
- https://issues.apache.org/jira/browse/IGNITE-8018 - inefficiency in
GridCacheMapEntry implementation;
- https://issues.apache.org/jira/browse/IGNITE-8017 - automatically disable
WAL during preloading.

Ways to solve problem on more global level:
- https://issues.apache.org/jira/browse/IGNITE-7935 - a ticket to introduce
batch modification;
- https://issues.apache.org/jira/browse/IGNITE-8020 - complete redesign of
rebalancing process for persistent caches, based on file transfer.

Everyone is welcome to criticize above ideas, suggest new ones or
participate in implementation.

-- 
Best regards,
Ilya

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message