cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Morton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2156) Compaction Throttling
Date Sat, 09 Apr 2011 02:35:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017794#comment-13017794
] 

Aaron Morton commented on CASSANDRA-2156:
-----------------------------------------

>From a discussion on the user list http://www.mail-archive.com/user@cassandra.apache.org/msg12027.html

CompactionManager.submitSSTableBuild() and submitIndexBuild() are used when receiving streams
from other nodes. But they do not use the CompactionIterator() so are not covered by this
ticket.

Want to create another ticket just for those tasks or reopen CASSANDRA-1882 and punt it to
a future version?

> Compaction Throttling
> ---------------------
>
>                 Key: CASSANDRA-2156
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2156
>             Project: Cassandra
>          Issue Type: New Feature
>            Reporter: Stu Hood
>            Assignee: Stu Hood
>             Fix For: 0.8
>
>         Attachments: 0001-Throttle-total-compaction-to-a-configurable-throughput.txt,
for-0.6-0001-Throttle-compaction-to-a-fixed-throughput.txt, for-0.6-0002-Make-compaction-throttling-configurable.txt
>
>
> Compaction is currently relatively bursty: we compact as fast as we can, and then we
wait for the next compaction to be possible ("hurry up and wait").
> Instead, to properly amortize compaction, you'd like to compact exactly as fast as you
need to to keep the sstable count under control.
> For every new level of compaction, you need to increase the rate that you compact at:
a rule of thumb that we're testing on our clusters is to determine the maximum number of buckets
a node can support (aka, if the 15th bucket holds 750 GB, we're not going to have more than
15 buckets), and then multiply the flush throughput by the number of buckets to get a minimum
compaction throughput to maintain your sstable count.
> Full explanation: for a min compaction threshold of {{T}}, the bucket at level {{N}}
can contain {{SsubN = T^N}} 'units' (unit == memtable's worth of data on disk). Every time
a new unit is added, it has a {{1/SsubN}} chance of causing the bucket at level N to fill.
If the bucket at level N fills, it causes {{SsubN}} units to be compacted. So, for each active
level in your system you have {{SubN * 1 / SsubN}}, or {{1}} amortized unit to compact any
time a new unit is added.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message