cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Morton (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3870) Internal error processing batch_mutate: java.util.ConcurrentModificationException on CounterColumn
Date Wed, 08 Feb 2012 09:45:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203424#comment-13203424
] 

Aaron Morton commented on CASSANDRA-3870:
-----------------------------------------

I've done some digging and this is what I think is happening...

StorageProxy.counterWriteOnCoordinatorPerformer() creates a counterWriteTask and put it in
the local MUTATION TP. This will run the mutation and then put a task in REPLICATE_ON_WRITE
to send if the counter write to other nodes, if a node is down hints a scheduled and AbstractWriteResponseHandler.addFutureForHint()
is called with the hint mutation future. 

However the response handler was originally created in the request thread, which may now have
called get() to wait on it. This may miss hints that have not already been added or may get
the CME in the error here. 

It looks like this is only a problem for counter writes where the coordinator is a replica,
and then only in certain circumstances. Counters where another replica is the leader look
ok and so do regular writes.

I was thinking:
* Change the AWRH.hintFutures to a CopyOnWriteArrayList (to be cautious) and
* Change AWRH.get() to wait on hints after the normal request has complet instead of before.
 
I'll try to pick it up again tomorrow.
                
> Internal error processing batch_mutate: java.util.ConcurrentModificationException on
CounterColumn
> --------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3870
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3870
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.7
>         Environment: Debian 6.0 x64
> x64 Sun Java 6u26
> Cassandra 1.0.7
> JNA
> 2 DCs, 1 ring/DC, 8 nodes/ring, RF=3/DC, Random partitioner.
> Disk access auto (mmap)
>            Reporter: Viktor Jevdokimov
>            Assignee: Aaron Morton
>
> Cassandra throws an exception below while performing batch_mutate with counter column
insertion mutation to increment column with 1:
> ERROR [Thrift:134] 2012-02-03 15:51:02,800 Cassandra.java (line 3462) Internal error
processing batch_mutate
> java.util.ConcurrentModificationException
>         at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>         at java.util.AbstractList$Itr.next(AbstractList.java:343)
>         at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:532)
>         at org.apache.cassandra.service.AbstractWriteResponseHandler.waitForHints(AbstractWriteResponseHandler.java:89)
>         at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:58)
>         at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:201)
>         at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:639)
>         at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:590)
>         at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:598)
>         at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Column family definition:
> create column family CountersColumnFamily1
>   with column_type = 'Standard'
>   and comparator = 'BytesType'
>   and default_validation_class = 'BytesType'
>   and key_validation_class = 'BytesType'
>   and rows_cached = 1000000.0
>   and row_cache_save_period = 0
>   and row_cache_keys_to_save = 2147483647
>   and keys_cached = 0.0
>   and key_cache_save_period = 14400
>   and read_repair_chance = 0.1
>   and gc_grace = 43200
>   and min_compaction_threshold = 4
>   and max_compaction_threshold = 32
>   and replicate_on_write = true
>   and row_cache_provider = 'SerializingCacheProvider'
>   and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message