cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Horwich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8120) Failed creating a merkle tree during repair.
Date Wed, 21 Jan 2015 14:29:34 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285672#comment-14285672
] 

Adam Horwich commented on CASSANDRA-8120:
-----------------------------------------

Hi, I spotted this today on quite a few ranges during a -par -local repair on a large column
family. We're using LeveledCompaction and Cassandra 2.1.2

I suspect this is also affecting the bootstrap issues I've been experiencing in https://issues.apache.org/jira/browse/CASSANDRA-8620

The advice seems to be to do an offline sstablescrub then perform a repair to repopulate the
removed out-of-order rows. But it is unclear how the sstable could get itself into this state,
and whether it will happen again.

> Failed creating a merkle tree during repair.
> --------------------------------------------
>
>                 Key: CASSANDRA-8120
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8120
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 12.04.5 LTS.  3.2.0-69-virtual.
>            Reporter: jason milliron
>
> Seeing this while running nodetool repair on several nodes of our new (not an upgrade)
2.1.0 cluster.  Multiple keyspaces.  Doesn't hang the repair job.  Storage is SSD ephemeral
(EC2).  No storage errors in syslog.  
> {noformat}
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,140 Validator.java:242 - Failed creating
a merkle tree for [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]],
/10.37.666.175 (see log for details)
> INFO  [AntiEntropySessions:37] 2014-10-13 22:20:51,141 RepairSession.java:260 - [repair
#308e2750-5327-11e4-925c-65a001aa6d80] new session: will sync ip-10-37-666-175.ec2.internal/10.37.666.175,
/10.167.666.137, /10.164.666.81 on range (6870883176570587821,6871062738154464368] for censored.[censored]
> INFO  [AntiEntropyStage:1] 2014-10-13 22:20:51,142 ColumnFamilyStore.java:856 - Enqueuing
flush of censored: 3227760 (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:2295] 2014-10-13 22:20:51,145 Memtable.java:326 - Writing
Memtable-censored@2052537453(380369 serialized bytes, 8235 ops, 0%/0% of on/off-heap limit)
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,148 RepairSession.java:303 - [repair
#1523dfa0-5327-11e4-925c-65a001aa6d80] session completed with the following error
> org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80
on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,148 CassandraDaemon.java:166 - Exception
in thread Thread[ValidationExecutor:39,1,main]
> java.lang.AssertionError: row DecoratedKey(-4468391015221140553, 00040000000100000800000000005abc2d00)
received out of order wrt DecoratedKey(-4436255891196926644, 0004000000010000080000000000617f4400)
>         at org.apache.cassandra.repair.Validator.add(Validator.java:136) ~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:916)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:100)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:560)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,149 CassandraDaemon.java:166 - Exception
in thread Thread[AntiEntropySessions:34,5,RMI Runtime]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair
#1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]]
Validation failed in /10.37.666.175
>         at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.0.jar:2.1.0]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80
on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         ... 3 common frames omitted
> INFO  [MemtableFlushWriter:2295] 2014-10-13 22:20:51,180 Memtable.java:360 - Completed
flushing /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/censored-censored-ka-3185-Data.db
(126124 bytes) for commitlog position ReplayPosition(segmentId=1412194655943, position=28904247)
> INFO  [RepairJobTask:3] 2014-10-13 22:20:51,259 RepairJob.java:145 - [repair #308e2750-5327-11e4-925c-65a001aa6d80]
requesting merkle trees for censored (to [/10.167.666.137, /10.164.666.81, ip-10-37-666-175.ec2.internal/10.37.666.175])
> INFO  [AntiEntropyStage:1] 2014-10-13 22:20:51,351 RepairSession.java:171 - [repair #308e2750-5327-11e4-925c-65a001aa6d80]
Received merkle tree for censored from /10.167.666.137
> INFO  [AntiEntropyStage:1] 2014-10-13 22:20:51,446 RepairSession.java:171 - [repair #308e2750-5327-11e4-925c-65a001aa6d80]
Received merkle tree for censored from /10.164.666.81
> INFO  [ValidationExecutor:39] 2014-10-13 22:20:51,452 SSTableReader.java:383 - Opening
/ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3173
(87123 bytes)
> INFO  [ValidationExecutor:39] 2014-10-13 22:20:51,453 SSTableReader.java:383 - Opening
/ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3181
(7010 bytes)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message