cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
Date Fri, 09 Dec 2016 22:59:58 GMT


Paulo Motta commented on CASSANDRA-12888:

Thanks for your investigation Benjamin. As discussed on the mailing list and CASSANDRA-12905
we cannot ensure view consistency with sstable-based streaming of base/MVs in all scenarios,
specially with repair, so I'm afraid this is not a viable solution just yet (needs to be further
discussed). An alternative is to provide an option for repair to allow repairing base and
MV separately when you *know what you are doing*©.

While we work on improving MV streaming overhead in a separate ticket, I'd like to focus here
on fixing the problem with MV/CDC repair stated in the description. What do you think of the
previous proposal of just skipping the write path for base table mutations and keeping the
sstables? While this is still a  bit expensive it will allow users to incrementally repair
moderately-sized MVs, specially after CASSANDRA-12905.

Your observations from large scale MV streaming in the context of bootstrap will be pretty
useful, would you mind adding them to the follow-up streaming improvement ticket for MVs?

By the way, congrats for the baby! :-)

> Incremental repairs broken for MVs and CDC
> ------------------------------------------
>                 Key: CASSANDRA-12888
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>            Reporter: Stefan Podkowinski
>            Assignee: Benjamin Roth
>            Priority: Critical
>             Fix For: 3.0.x, 3.x
> SSTables streamed during the repair process will first be written locally and afterwards
either simply added to the pool of existing sstables or, in case of existing MVs or active
CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we must put all
partitions through the same write path as normal mutations. This also ensures any 2is are
also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through the CommitLog
so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental repairs, as we
loose the {{repaired_at}} state in the process. Eventually the streamed rows will end up in
the unrepaired set, in contrast to the rows on the sender site moved to the repaired set.
The next repair run will stream the same data back again, causing rows to bounce on and on
between nodes on each repair.
> See linked dtest on steps to reproduce. An example for reproducing this manually using
ccm can be found [here|]

This message was sent by Atlassian JIRA

View raw message