cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Brown (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8304) Explore evicting replacement state sooner
Date Sat, 22 Nov 2014 06:30:33 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221847#comment-14221847
] 

Jason Brown commented on CASSANDRA-8304:
----------------------------------------

As a slightly different tack, let's consider this: we've hit some bumps with this uncoordinated
garbage collection where we expect things to line up nicely and timing issues/races to not
get in our way, but we have'n t been that lucky.  So, let's consider two possible ideas:

1) don't bother garbage collecting dead nodes. We've fallen into this gap of wall clock time
between nodes and expect things to line up and things not bleed over, but #8260 shows us up
again. Perhaps we can rename the quarantine to "dead pile", and just ignore anything in there.
I can't image the memory requirements to be that great, and it'd be cleaned out with a restart.
As for the system.peers table, we could add an extra column that indicates the node has been
replaced/shot/evicted/whatever. Of course, this puts a slightly higher burden on clients who
read from system.peers to discover the cluster to check an extra field for 'dead' status.

2) coordinating the garbage collection. Instead of expecting each individual to drop the quarantined
nodes independently, we could have a node (perhaps the replacement node, or pick something
via paxos) send a message throughout the system to indicate that everyone drop any quarantined
peer data. What I imagine is a broadcast primitive unlike what we have currently in cassandra,
but something more like Plumtree (http://www.gsd.inesc-id.pt/~jleitao/pdf/srds07-leitao.pdf)
where a message is disseminated rapidly throughout the cluster based on broadcast trees (in
a gossip-style manner, of course). This way we have a degree of coordination on the garbage
collection (one node shouting "bring out yer dead"). Of course, this idea is not completely
timing/race-proof, and I haven't fully baked the idea myself yet, and it's a mechanism we
do not have, but I think there's some value in ignoring the garbage (a la my first idea),
and then coordinating the garbage collection.

Even if neither of these thoughts are usable, hoping we can generate some new, interesting
ideas.

> Explore evicting replacement state sooner
> -----------------------------------------
>
>                 Key: CASSANDRA-8304
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8304
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>             Fix For: 3.0
>
>
> As a follow-up to CASSANDRA-8260 Tyler suggests we evict and quarantine in step 2.  I
don't want to change any core gossip logic in 2.0 at this point, but his theory seems feasible
and we should explore it for 3.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message