cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Morton (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-4601) Ensure unique commit log file names
Date Sun, 02 Sep 2012 22:39:07 GMT
Aaron Morton created CASSANDRA-4601:
---------------------------------------

             Summary: Ensure unique commit log file names
                 Key: CASSANDRA-4601
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4601
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Sun JVM 1.6.33 / Ubuntu 10.04.4 LTS 
            Reporter: Aaron Morton
            Assignee: Aaron Morton
            Priority: Critical


The commit log segment name uses System.nanoTime() as part of the file name. There is no guarantee
that successive calls to nanoTime() will return different values. And on less than optimal
hypervisors this happens a lot. 

I observed the following in the wild:

{code:java}
ERROR [COMMIT-LOG-ALLOCATOR] 2012-08-31 15:56:49,815 AbstractCassandraDaemon.java (line 134)
Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.lang.AssertionError: attempted to delete non-existing file CommitLog-13926764209796414.log
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:68)
        at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172)
        at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223)
        at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.lang.Thread.run(Unknown Source)
{code}

My _assumption_ is that it was because of duplicate file names. As this is on a hypervisor
that is less than optimal. 
 
After a while (about 30 minutes) mutations stopped being processed and the pending count sky
rocketed. I _think_ this was because log writing was blocked trying to get a new segment and
writers could not submit to the commit log queue. The only way to stop the affected nodes
was kill -9. 

Over about 24 hours this happened 5 times. I have deployed a patch that has been running for
12 hours without incident, will attach. 

The affected nodes could still read, and I'm checking logs to see how the other nodes handled
the situation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message