ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Vinogradov ...@apache.org>
Subject Crash recovery speed-up #3, Cellular Switch
Date Wed, 06 May 2020 10:24:01 GMT
Igniters,

PME-free switch [1] (since 2.8) skips PME on node left when possible
(baseline + fully rebalanced cluster).
This means we already wait for nothing (except recovery) to perform the
switch.
This optimization allows continuing already started operations during or
after the switch if they are not affected by failed primary.
But upcoming operations still can't be started until the switch is finished
cluster-wide.

Let me propose an additional optimization - Cellular switch.
Cellular Affinity [2] means that nodes combined into virtual cells where,
for each partition, backups located at the same cell with primaries.
The simplest way to gain Cellular Affinity is to use backup filters [3].

Cellular Affinity allows to finish the switch outside the affected cell
instantly with the following assumptions:
- Replicated caches should be recovered first since every node affected (as
a backup) by any failed primary.
  But, it is expected that replicated caches effectively read-only (has
extremely rare updates), so, nothing to wait here.
- Upcoming replicated transactions (with non-failed primaries) can be
started but can't be committed until switch finished cluster-wide.
- Upcoming transactions related to the broken cell will wait for cell
recovery (cluster-wide switch finish).

... and this means:
In addition to PME-free switch, where we able to continue already started
operations during or after the switch, now we also able to perform most of
the upcoming operations during the switch.

In other words, Cellular switch has little effect on the operation's
latency, when operation not related to the failed cell.

According to benchmark [4] which checks "how fast upcoming transactions
(started after switch start) can be committed when we have thousands of
prepared transactions (prepared before switch start)", we have 5326 ms [5]
operation's latency on master and 65 ms [6] with the proposed fix, which is
~100 times faster.

Fix [7] (as a part of IEP-45 [8]) ready to be reviewed.
Waiting for your review!


[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Non-blocking-PME-Phase-One-Node-fail-tp43531p44586.html
[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up#IEP-45:CrashRecoverySpeed-Up-Cellularswitch
[3]
https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5#file-bench-java-L417
[4]
https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5
[5]
https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-master-txt-L15
[6]
https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-fix-txt-L15
[7] https://issues.apache.org/jira/browse/IGNITE-12617
[8]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message