cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6109) Consider coldness in STCS compaction
Date Tue, 01 Oct 2013 20:49:32 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783326#comment-13783326
] 

Tyler Hobbs commented on CASSANDRA-6109:
----------------------------------------

bq. The latter is better if you have cold data that may become hot again... but it's confusing
if you have a workload such that you can't keep up with all compaction, but you can keep up
with hot sstable. (Compaction backlog stat becomes useless since we fall increasingly behind.)

The pending compactions stat is already pretty wonky, so I'm not sure we should prioritize
keeping that sane.

Option 1 (don't compact cold sstables) seems dangerous as a first step compared to option
2, especially because it's hard to decide what is "cold".  Prioritizing compaction of hotter
sstables seems like the better first step.

When comparing hotness of sstables, I think a good measure is {{avg_reads_per_sec / number_of_keys}}
rather than just {{avg_reads_per_sec}} so that large sstables aren't over-weighted.  When
I mention the hotness of a bucket of sstables below, I'm talking about the sum of the hotness
measure across the individual sstables.

For prioritizing compaction of hotter sstables, it seems like there are a few levels this
can operate at:
# Picking sstable members for compaction buckets
# Picking the most "interesting" bucket to submit to the compaction executor (currently the
smallest sstables are considered the most interesting)
# At the compaction executor level, prioritizing tasks in the queue (the queue is not currently
prioritized)

(1) seems like the most difficult point to make good decisions at.  I can imagine a scheme
like dropping members that are below {{2 * stdev}} of the mean hotness for the bucket working
decently, but some of the efficiency of compacting many sstables at once is lost, and some
of the drops would be poor when there is little variance among the sstables.

(2) would probably work well by itself, although, as discussed below, sstable overlap is a
better measure than hotness for this.

(3) requires (2) to be somewhat fair.  Each table submits its hottest buckets for compaction,
and the executor prioritizes the hottest buckets in the queue (regardless of which table they
came from).  There is a potential for starvation among colder tables when compaction falls
behind, but that may be mitigated by a few things:
* If the compaction of the hotter sstables is very effective at merging rows, the hotness
of future buckets for that table should be lower.  Since the hotness of a bucket is the sum
of its members, if four totally overlapping sstables are merged into one sstable, the hotness
of the new sstable should be 1/4 of the hotness of the previous bucket.  I'll point out that
tracking how much overlap there is among sstables would be a much better measure than hotness
for picking which compactions to prioritize; in the worst case here (no overlap), the hotness
of the newly compacted sstable could be the same as the bucket it came from.
* If we were willing to discard cold items in the queue when hotter items came in and the
queue was full, colder tables would eventually submit new tasks with more sstables in them
(thus having greater hotness).

While I'm thinking about it, do we have any tickets or features in place to track sstable
overlap (beyond average number of sstables hit per read at the table level)?

> Consider coldness in STCS compaction
> ------------------------------------
>
>                 Key: CASSANDRA-6109
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6109
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Tyler Hobbs
>             Fix For: 2.0.2
>
>
> I see two options:
> # Don't compact cold sstables at all
> # Compact cold sstables only if there is nothing more important to compact
> The latter is better if you have cold data that may become hot again...  but it's confusing
if you have a workload such that you can't keep up with *all* compaction, but you can keep
up with hot sstable.  (Compaction backlog stat becomes useless since we fall increasingly
behind.)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message