cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (Commented) (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-2503) Eagerly re-write data at read time ("superseding")
Date Wed, 23 Nov 2011 08:37:40 GMT


Sylvain Lebresne commented on CASSANDRA-2503:

Two small nits:
* I would prefer using {{sstablesIterated > cfs.getMinimumCompactionThreshold()}} rather
than {{>=}}. I guess I'm afraid that this 'limit worst-case' would get trigger too often.
Typically for minThreshold == 2, hoisting as soon as we hit more than 1 sstable feels a bit
too much. Again, I have no big argument, it just feels a tad more reasonable with >.
* We probably should use {{!CFS.isCompactionDisabled()}} instead of {{cfs.getMinimumCompactionThreshold()
> 0}}

But those minor nits apart, patch lgtm. +1 with or without the changes.
> Eagerly re-write data at read time ("superseding")
> --------------------------------------------------
>                 Key: CASSANDRA-2503
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Stu Hood
>            Assignee: Jonathan Ellis
>              Labels: compaction, performance
>             Fix For: 1.1
>         Attachments: 2503-v2.txt, 2503-v3.txt, 2503.txt
> Once CASSANDRA-2498 is implemented, it will be possible to implement an optimization
to eagerly rewrite ("supersede") data at read time. If a successful read needed to hit more
than a certain threshold of sstables, we can eagerly rewrite it in a new sstable, and 2498
will allow only that file to be accessed. This basic approach would improve read performance
considerably, but would cause a lot of duplicate data to be written, and would make compaction's
work more necessary.
> Augmenting the basic idea, if when we superseded data in a file we marked it as superseded
somehow, the next compaction that touched that file could remove the data. Since our file
format is immutable, the values that a particular sstable superseded could be recorded in
a component of that sstable. If we always supersede at the "block" level (as defined by CASSANDRA-674
or CASSANDRA-47), then the list of superseded blocks could be represented using a generation
number and a bitmap of block numbers. Since 2498 would already allow for sstables to be eliminated
due to timestamps, this information would probably only be used at compaction time (by loading
all superseding information in the system for the sstables that are being compacted).
> Initially described on [1608|!default.jspa?id=12477095&commentId=12920353].

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message