cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-8316) "Did not get positive replies from all endpoints" error on incremental repair
Date Thu, 18 Dec 2014 13:14:15 GMT


Marcus Eriksson commented on CASSANDRA-8316:

To summarize this;
1. we had a bug in compaction marking that could make a node end up in an infinite loop, fixed
in branch linked above
2. we allowed multiple repairs over the same sstables, fixed
3. we had a situation where we didn't remove the parent repair sessions, fixed

And, to describe the "final" problem:

# Node A sends a PrepareMessage to overloaded Node B
# B starts preparing
# A times out waiting for B to prepare
# B finishes preparing and marks a bunch of sstables as being repaired
# User retries the repair on node A
# B gets the new PrepareMessage but sees that the sstables it wants to repair are already
being repaired, and refuses to start

One solution could be to have A send out a cancel message, another solution could be to have
B remove any parent repair sessions after 5 (or something) minutes if it hasn't received a
validation message before that. Need [~yukim] input.

>  "Did not get positive replies from all endpoints" error on incremental repair
> ------------------------------------------------------------------------------
>                 Key: CASSANDRA-8316
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: cassandra 2.1.2
>            Reporter: Loic Lambiel
>            Assignee: Marcus Eriksson
>             Fix For: 2.1.3
>         Attachments: 0001-patch.patch, 8316-v2.patch, CassandraDaemon-2014-11-25-2.snapshot.tar.gz,
> Hi,
> I've got an issue with incremental repairs on our production 15 nodes 2.1.2 (new cluster,
not yet loaded, RF=3)
> After having successfully performed an incremental repair (-par -inc) on 3 nodes, I started
receiving "Repair failed with error Did not get positive replies from all endpoints." from
nodetool on all remaining nodes :
> [2014-11-14 09:12:36,488] Starting repair command #3, repairing 108 ranges for keyspace
xxxx (seq=false, full=false)
> [2014-11-14 09:12:47,919] Repair failed with error Did not get positive replies from
all endpoints.
> All the nodes are up and running and the local system log shows that the repair commands
got started and that's it.
> I've also noticed that soon after the repair, several nodes started having more cpu load
indefinitely without any particular reason (no tasks / queries, nothing in the logs). I then
restarted C* on these nodes and retried the repair on several nodes, which were successful
until facing the issue again.
> I tried to repro on our 3 nodes preproduction cluster without success
> It looks like I'm not the only one having this issue:
> Any idea?
> Thanks
> Loic

This message was sent by Atlassian JIRA

View raw message