cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zach Aller (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart
Date Tue, 22 Apr 2014 05:24:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976386#comment-13976386
] 

Zach Aller edited comment on CASSANDRA-7042 at 4/22/14 5:23 AM:
----------------------------------------------------------------

Doing nothing they just show up in the logs during normal use our tables are all ttl'd set
to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the
issue, here is a error for the 2.0.7 version not even sure if the error is related to the
space growth but it is the only bit of info in the logs that might suggest it. Willing to
help you guys out anyway I can would love to get this fixed its pretty much our last major
issue with keeping our cluster stable.

ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception
in thread Thread[CompactionExecutor:112,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db
(No such file or directory)
        at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
        at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
        at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
        at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)


was (Author: zaller):
Doing nothing they just show up in the logs during normal use our tables are all ttl'd set
to about a day with around 30GB or so of data churn. Upgraded to 2.0.7 and still having the
issue, here is a error for the 2.0.7 version. Willing to help you guys out anyway I can would
love to get this fixed its pretty much our last major issue with keeping our cluster stable.

ERROR [CompactionExecutor:112] 2014-04-21 21:26:35,109 CassandraDaemon.java (line 198) Exception
in thread Thread[CompactionExecutor:112,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-580812-Data.db
(No such file or directory)
        at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
        at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
        at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
        at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)

> Disk space growth until restart
> -------------------------------
>
>                 Key: CASSANDRA-7042
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>            Reporter: Zach Aller
>         Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot 2014-04-18
at 11.47.30 AM.png, after.log, before.log
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only thing that
seems to fix it is a restart of cassandra this happens about every 3-5 hrs we will grow from
about 350GB to 650GB with no end in site. Once we restart cassandra it usually all clears
itself up and disks return to normal for a while then something triggers its and starts climbing
again. Sometimes when we restart compactions pending skyrocket and if we restart a second
time the compactions pending drop off back to a normal level. One other thing to note is the
space is not free'd until cassandra starts back up and not when shutdown.
> I will get a clean log of before and after restarting next time it happens and post it.
> Here is a common ERROR in our logs that might be related
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java (line 196)
Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
(No such file or directory)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
>         at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
>         at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
>         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
>         at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
(No such file or directory)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(Unknown Source)
>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
>         at org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
>         ... 17 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message