cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Olsson (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-11412) Many sstablescanners opened during repair
Date Wed, 23 Mar 2016 18:37:25 GMT


Marcus Olsson commented on CASSANDRA-11412:

Hmm.. yes that seems to be a bug. Either we could implement hashCode() and equals() on all
scanners or we could change (or add a separate) {{AbstractCompactionStrategy.getScanners()}}
to take a collection of {{Range<Token>}}. I think the second option would be preferable
since we would only call {{SSTableReader.getScanner()}} once for each sstable. There is already
a {{getScanner()}} method that takes [a collection of ranges|]
in SSTableReader. This would also make more sense with LCS since as it is now even if we check
for equality the LeveledScanner might not always be equal for all ranges and create one scanner
per range either way.

This would still make it so that we have one scanner per sstable opened at the same time though(which
would pretty much be the same as pre-CASSANDRA-5220), except for when using leveled compaction.
So it might be good to only open the scanners when needed depending on how many sstables we
have. The implementation for that would probably be a bit more complex than for LCS since
in LeveledScanner we work with the assumption that all sstables in a level are non-overlapping
so we only open one scanner at a time. I think the partition iteration for overlapping sstables
could be done as:
# Sort all the sstables based on tokens
# Open the first sstable scanner
# Read/merge first partition from all open scanners (and close exhausted scanners) and keep
the first partition
# Compare the partition to the non-open sstables to see if they may contain a partition before
or at the same token as the previously read partition, open a scanner if that's the case and
then read the first partition
# Compare/merge all current partitions and return the first
# Continue at step 3 until all sstables are read

Unless I'm over-thinking this.

> Many sstablescanners opened during repair
> -----------------------------------------
>                 Key: CASSANDRA-11412
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Marcus Eriksson
> Since CASSANDRA-5220 we open [one sstablescanner per range per sstable|].
If compaction gets behind and you are running vnodes with 256 tokens and RF3, this could become
a problem (ie, {{768 * number of sstables}} scanners)
> We could probably refactor this similar to the way we handle scanners with LCS - only
open the scanner once we need it

This message was sent by Atlassian JIRA

View raw message