cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-2968) AssertionError during compaction of CF with counter columns
Date Mon, 01 Aug 2011 17:24:10 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-2968:
----------------------------------------

    Attachment: 2968.patch

This is actually a pretty stupid bug (not that there is smart bug): the old NodeId for the
local node were read from the system table in reversed order while they shouldn't. The wrong
path was then taken based on that mistake. No data was lost due to that (i.e, the total value
of the counters is preserved), but non-sensical counter context were created (hence triggering
the assertion).

Fixing the root cause is pretty straightforward. Fixing the nonsensical counter contexts is
more subtle, but it is doable up to the fact that the local NodeId on the node(s) where the
assertion is triggered will have to be renewed. Attaching a patch that does both (fixing root
cause and repairing the bad data). Also add two unit tests, one for the root cause and one
to check that the bad data repair code does what it is supposed to do.

After applying that patch (or upgrading on a release shipping it), you will (potentially)
need to restart the node with the -Dcassandra.renew_counter_id=true (compaction will still
fail if you don't but with a message saying that you should restart with the startup flag).

> AssertionError during compaction of CF with counter columns
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-2968
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2968
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.8.2
>         Environment: CentOS release 5.6
>            Reporter: Taras Puchko
>            Assignee: Sylvain Lebresne
>             Fix For: 0.8.3
>
>         Attachments: 2968.patch, AffiliateActivity-g-147-Data.db, AffiliateActivity-g-147-Index.db,
AffiliateActivity-g-195-Data.db, AffiliateActivity-g-195-Index.db
>
>
> Having upgraded from 0.8.0 to 0.8.2 we ran nodetool compact and got
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.lang.AssertionError
>         at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
>         at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
>         at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>         at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>         at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>         at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>         at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>         at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>         at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>         at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>         at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>         at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>         at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>         at sun.rmi.transport.Transport$1.run(Transport.java:159)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>         at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.AssertionError                                                 
                                                                                         
                                                              
>         at org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterContext.java:593)
                                                                                         
                                                
>         at org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:237)
                                                                                         
                                                          
>         at org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:256)
                                                                                         
                                                          
>         at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:88)
                                                                                         
                                                     
>         at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:140)
                                                                                         
                                 
>         at org.apache.cassandra.db.compaction.CompactionIterator.getReduced(CompactionIterator.java:123)
                                                                                         
                                          
>         at org.apache.cassandra.db.compaction.CompactionIterator.getReduced(CompactionIterator.java:43)
                                                                                         
                                           
>         at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:74)
                                                                                         
                                                      
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
                                                                                         
                                                 
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
                                                                                         
                                                          
>         at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
                                                                                         
                                         
>         at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
                                                                                         
                                                
>         at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:569)
                                                                                         
                     
>         at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506)
                                                                                         
                                          
>         at org.apache.cassandra.db.compaction.CompactionManager$4.call(CompactionManager.java:319)
                                                                                         
                                                
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)       
                                                                                         
                                                              
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)                 
                                                                                         
                                                              
>         ... 3 more

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message