cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Lynch (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra
Date Tue, 03 Apr 2018 19:28:00 GMT


Joseph Lynch commented on CASSANDRA-14346:

{quote}You're free to spend time implementing any solution for this you’d like. Just keep
in mind that it’s unlikely to get committed until there’s been a wider discussion on how
we should be approaching this problem as a project. You have 2 people on this ticket concerned
that this might not be the right way to approach the problem of scheduling repairs.
My understanding is that there are two alternative proposals, please correct me if I'm missing
 # Cassandra ships with repair scheduling in a sidecar process (either with or without reliable
 # We continue doing nothing and the community just solves this in different ways.

The current production implementation we have internally is basically #1, it would be _easier_
for us to open source it as a separate service or jar. When we talked about this at NGCC (and
later conversations) [~kohlisankalp] and others had objections to us open sourcing it as yet
another external tool, so we changed directions to working on porting into Cassandra itself.
We were further convinced by our production experience with a sidecar based scheduler that
indicates this is not a particularly reliable way to build the scheduler due to numerous IPC
difficulties (which I have described in detail above and in the design document). To be honest
much I don't think we _need_ up front consensus on sidecar vs in-process since it is reasonably
easy to port the code so that we can switch between them easily (except the table config,
that's the one thing we'd need to figure out).

I don't view #2 (status quo) as a good solution. Do you?
{quote}You also haven't addressed my comments about why ipc difficulties and being in process
making it easier to detect problems aren't strong arguments for doing this in process.
I thought I did, let me try again. If we're inside Cassandra then repair state (status of
repair) and repair scheduling state (status of cluster repair) are shared in the same process,
so we can easily call methods, setup thread heartbeats, etc ... If we are in separate processes
then not only do we have to deal with differing process lifecycles, but every piece of communication
must go through JMX (or not yet existing reliable IPC); this slows down iteration and limits
repair scheduling capability and reliability. My understanding of your disagreement is that
we _could_ theoretically have reliable IPC but I just don't agree and welcome any concrete
proposal that provides the same level of reliability that in-process inter-thread communication
{quote}Discussing alternate approaches with an open mind is probably a better use of time.
I am trying to be very open in discussing alternate approaches, and I'm happy to take the
sidecar implementation we have an put it literally anywhere (including just on Netflix's Priam
project). I think the only thing that would change significantly is where the per table configuration
lives, and I have offered an option that might be compatible with either design where we put
the table config in a Cassandra table if we wanted to go that way. That also may be more convenient
for people that already have external tools since interfacing with Cassandra tables is much
easier than with table metadata. If you or others have an alternative suggestion I of course
welcome that.

> Scheduled Repair in Cassandra
> -----------------------------
>                 Key: CASSANDRA-14346
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Repair
>            Reporter: Joseph Lynch
>            Priority: Major
>              Labels: CommunityFeedbackRequested
>             Fix For: 4.0
>         Attachments: ScheduledRepairV1_20180327.pdf
> There have been many attempts to automate repair in Cassandra, which makes sense given
that it is necessary to give our users eventual consistency. Most recently CASSANDRA-10070,
CASSANDRA-8911 and CASSANDRA-13924 have all looked for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), which
we spoke about last year at NGCC. Given the positive feedback at NGCC we focussed on getting
it production ready and have now been using it in production to repair hundreds of clusters,
tens of thousands of nodes, and petabytes of data for the past six months. Also based on feedback
at NGCC we have invested effort in figuring out how to integrate this natively into Cassandra
rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our implementation into
Cassandra, and have created a [design document|]
showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would be greatly
appreciated about the interface or v1 implementation features. I have tried to call out in
the document features which we explicitly consider future work (as well as a path forward
to implement them in the future) because I would very much like to get this done before the
4.0 merge window closes, and to do that I think aggressively pruning scope is going to be
a necessity.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message