ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexei Scherbakov <alexey.scherbak...@gmail.com>
Subject Re: [DISCUSS] Data loss handling improvements
Date Thu, 07 May 2020 08:36:10 GMT
Yes, it will work this way.

чт, 7 мая 2020 г. в 10:43, Anton Vinogradov <av@apache.org>:

> Seems I got the vision, thanks.
> There should be only 2 ways to reset lost partition: to gain an owner from
> resurrected first or to remove ex-owner from baseline (partition will be
> rearranged).
> And we should make a decision for every lost partition before calling the
> reset.
>
> On Wed, May 6, 2020 at 8:02 PM Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > ср, 6 мая 2020 г. в 12:54, Anton Vinogradov <av@apache.org>:
> >
> > > Alexei,
> > >
> > > 1,2,4,5 - looks good to me, no objections here.
> > >
> > > >> 3. Lost state is impossible to reset if a topology doesn't have at
> > least
> > > >> one owner for each lost partition.
> > >
> > > Do you mean that, according to your example, where
> > > >> a node2 has left, soon a node3 has left. If the node2 is returned
to
> > > >> the topology first, it would have stale data for some keys.
> > > we have to have node2 at cluster to be able to reset "lost" to node2's
> > > data?
> > >
> >
> > Not sure if I understand a question, but try to answer using an example:
> > Assume 3 nodes n1, n2, n3, 1 backup, persistence enabled, partition p is
> > owned by n2 and n3.
> > 1. Topology is activated.
> > 2. cache.put(p, 0) // n2 and n3 have p->0, updateCounter=1
> > 3. n2 has failed.
> > 4. cache.put(p, 1) // n3 has p->1, updateCounter=2
> > 5. n3 has failed, partition loss is happened.
> > 6. n2 joins a topology, it has stale data (p->0)
> >
> > We actually have 2 issues:
> > 7. cache.put(p, 2) will success, n2 has p->2, n3 has p->0, data is
> diverged
> > and will not be adjusted by counters rebalancing if n3 is later joins a
> > topology.
> > or
> > 8. n3 joins a topology, it has actual data (p->1) but rebalancing will
> not
> > work because joining node has highest counter (it can only be a demander
> in
> > this scenario).
> >
> > In both cases rebalancing by counters will not work causing data
> divergence
> > in copies.
> >
> >
> > >
> > > >> at least one owner for each lost partition.
> > > What the reason to have owners for all lost partitions when we want to
> > > reset only some (available)?
> > >
> >
> > It's never were possible to reset only subset of lost partitions. The
> > reason is to make guarantee of resetLostPartitions method - all cache
> > operations are resumed, data is correct.
> >
> >
> > > Will it be possible to perform operations on non-lost partitions when
> the
> > > cluster has at least one lost partition?
> > >
> >
> > Yes it will be.
> >
> >
> > >
> > > On Wed, May 6, 2020 at 11:45 AM Alexei Scherbakov <
> > > alexey.scherbakoff@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > I've almost finished a patch bringing some improvements to the data
> > loss
> > > > handling code, and I wish to discuss proposed changes with the
> > community
> > > > before submitting.
> > > >
> > > > *The issue*
> > > >
> > > > During the grid's lifetime, it's possible to get into a situation
> when
> > > some
> > > > data nodes have failed or mistakenly stopped. If a number of stopped
> > > nodes
> > > > exceeds a certain threshold depending on configured backups, count a
> > data
> > > > loss will occur. For example, a grid having one backup (meaning at
> > least
> > > > two copies of each data partition exist at the same time) can
> tolerate
> > > only
> > > > one node loss at the time. Generally, data loss is guaranteed to
> occur
> > if
> > > > backups + 1 or more nodes have failed simultaneously using default
> > > affinity
> > > > function.
> > > >
> > > > For in-memory caches, data is lost forever. For persistent caches,
> data
> > > is
> > > > not physically lost and accessible again after failed nodes are
> > returned
> > > to
> > > > the topology.
> > > >
> > > > Possible data loss should be taken into consideration while designing
> > an
> > > > application.
> > > >
> > > >
> > > >
> > > > *Consider an example: money is transferred from one deposit to
> another,
> > > and
> > > > all nodes holding data for one of the deposits are gone.In such a
> > case, a
> > > > transaction temporary cannot be completed until a cluster is
> recovered
> > > from
> > > > the data loss state. Ignoring this can cause data inconsistency.*
> > > > It is necessary to have an API telling us if an operation is safe to
> > > > complete from the perspective of data loss.
> > > >
> > > > Such an API exists for some time [1] [2] [3]. In short, a grid can be
> > > > configured to switch caches to the partial availability mode if data
> > loss
> > > > is detected.
> > > >
> > > > Let's give two definitions according to the Javadoc for
> > > > *PartitionLossPolicy*:
> > > >
> > > > ·   *Safe* (data loss handling) *policy* - cache operations are only
> > > > available for non-lost partitions (PartitionLossPolicy != IGNORE).
> > > >
> > > > ·   *Unsafe policy* - cache operations are always possible
> > > > (PartitionLossPolicy = IGNORE). If the unsafe policy is configured,
> > lost
> > > > partitions automatically re-created on the remaining nodes if needed
> or
> > > > immediately owned if a last supplier has left during rebalancing.
> > > >
> > > > *That needs to be fixed*
> > > >
> > > > 1. The default loss policy is unsafe, even for persistent caches in
> the
> > > > current implementation. It can result in unintentional data loss and
> > > > business invariants' failure.
> > > >
> > > > 2. Node restarts in the persistent grid with detected data loss will
> > > cause
> > > > automatic resetting of LOST state after the restart, even if the safe
> > > > policy is configured. It can result in data loss or partition desync
> if
> > > not
> > > > all nodes are returned to the topology or returned in the wrong
> order.
> > > >
> > > >
> > > > *An example: a grid has three nodes, one backup. The grid is under
> > load.
> > > > First, a node2 has left, soon a node3 has left. If the node2 is
> > returned
> > > to
> > > > the topology first, it would have stale data for some keys. Most
> recent
> > > > data are on node3, which is not in the topology yet. Because a lost
> > state
> > > > was reset, all caches are fully available, and most probably will
> > become
> > > > inconsistent even in safe mode.*
> > > > 3. Configured loss policy doesn't provide guarantees described in the
> > > > Javadoc depending on the cluster configuration[4]. In particular,
> > unsafe
> > > > policy (IGNORE) cannot be guaranteed if a baseline is fixed (not
> > > > automatically readjusted on node left), because partitions are not
> > > > automatically get reassigned on topology change, and no nodes are
> > > existing
> > > > to fulfill a read/write request. Same for READ_ONLY_ALL and
> > > READ_WRITE_ALL.
> > > >
> > > > 4. Calling resetLostPartitions doesn't provide a guarantee for full
> > cache
> > > > operations availability if a topology doesn't have at least one owner
> > for
> > > > each lost partition.
> > > >
> > > > The ultimate goal of the patch is to fix API inconsistencies and fix
> > the
> > > > most crucial bugs related to data loss handling.
> > > >
> > > > *The planned changes are:*
> > > >
> > > > 1. The safe policy is used by default, except for in-memory grids
> with
> > > > enabled baseline auto-adjust [5] with zero timeout [6]. In the latter
> > > case,
> > > > the unsafe policy is used by default. It protects from unintentional
> > data
> > > > loss.
> > > >
> > > > 2. Lost state is never reset in the case of grid nodes restart
> (despite
> > > > full restart). It makes real data loss impossible in persistent grids
> > if
> > > > following the recovery instruction.
> > > >
> > > > 3. Lost state is impossible to reset if a topology doesn't have at
> > least
> > > > one owner for each lost partition. If nodes are physically dead, they
> > > > should be removed from a baseline first before calling
> > > resetLostPartitions.
> > > >
> > > > 4. READ_WRITE_ALL, READ_ONLY_ALL is a subject for deprecation because
> > > their
> > > > guarantees are impossible to fulfill, not on the full baseline.
> > > >
> > > > 5. Any operation failed due to data loss contains
> > > > CacheInvalidStateException as a root cause.
> > > >
> > > > In addition to code fixes, I plan to write a tutorial for safe data
> > loss
> > > > recovery in the persistent mode in the Ignite wiki.
> > > >
> > > > Any comments for the proposed changes are welcome.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> org.apache.ignite.configuration.CacheConfiguration#setPartitionLossPolicy(PartitionLossPolicy
> > > > partLossPlc)
> > > > [2] org.apache.ignite.Ignite#resetLostPartitions(caches)
> > > > [3] org.apache.ignite.IgniteCache#lostPartitions
> > > > [4]  https://issues.apache.org/jira/browse/IGNITE-10041
> > > > [5]
> org.apache.ignite.IgniteCluster#baselineAutoAdjustEnabled(boolean)
> > > > [6] org.apache.ignite.IgniteCluster#baselineAutoAdjustTimeout(long)
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>


-- 

Best regards,
Alexei Scherbakov

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message