cassandra-pr mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iamaleksey <...@git.apache.org>
Subject [GitHub] cassandra pull request #239: [CASSANDRA-14556] Optimize Streaming
Date Mon, 23 Jul 2018 19:48:55 GMT
Github user iamaleksey commented on a diff in the pull request:

    https://github.com/apache/cassandra/pull/239#discussion_r204531887
  
    --- Diff: src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java ---
    @@ -114,13 +155,51 @@ public void write(StreamSession session, DataOutputStreamPlus out,
int version)
             CassandraStreamHeader.serializer.serialize(header, out, version);
             out.flush();
     
    -        CassandraStreamWriter writer = header.compressionInfo == null ?
    -                                       new CassandraStreamWriter(sstable, header.sections,
session) :
    -                                       new CompressedCassandraStreamWriter(sstable, header.sections,
    -                                                                           header.compressionInfo,
session);
    +        IStreamWriter writer;
    +        if (shouldStreamFullSSTable())
    +        {
    +            writer = new CassandraBlockStreamWriter(sstable, session, components);
    +        }
    +        else
    +        {
    +            writer = (header.compressionInfo == null) ?
    +                     new CassandraStreamWriter(sstable, header.sections, session) :
    +                     new CompressedCassandraStreamWriter(sstable, header.sections,
    +                                                         header.compressionInfo, session);
    +        }
             writer.write(out);
         }
     
    +    @VisibleForTesting
    +    public boolean shouldStreamFullSSTable()
    +    {
    +        return isFullSSTableTransfersEnabled && isFullyContained;
    +    }
    +
    +    @VisibleForTesting
    +    public boolean fullyContainedIn(List<Range<Token>> normalizedRanges,
SSTableReader sstable)
    +    {
    +        if (normalizedRanges == null)
    +            return false;
    +
    +        RangeOwnHelper rangeOwnHelper = new RangeOwnHelper(normalizedRanges);
    +        try (KeyIterator iter = new KeyIterator(sstable.descriptor, sstable.metadata()))
    +        {
    +            while (iter.hasNext())
    +            {
    +                DecoratedKey key = iter.next();
    +                try
    +                {
    +                    rangeOwnHelper.check(key);
    +                } catch(RuntimeException e)
    --- End diff --
    
    On a more general note, this is potentially quite an expensive thing to do, especially
for big sstables with skinny partitions, and in some cases this will introduce a performance
regression.
    
    The whole optimisation is realistically only useful for bootrstrap, decom, and rebuild,
with LCS (which is still plenty useful and impactful and worth having). But it wouldn't normally
kick in for regular repairs because of the full-cover requirement, and it won't normally kick
in for STCS until CASSANDRA-10540 (range aware compaction) is implemented. In those cases
having to read through the whole primary index is a perf regression that we shouldn't allow
to happen.
    
    The easiest way to avoid it would be to store sstable's effective token ranges in sstable
metadata in relation to the node's ranges, making this check essentially free. Otherwise we
should probably disable complete sstable streaming for STCS tables, at least until CASSANDRA-10540
is implemented. That however wouldn't address the regression to regular streaming, so keeping
ranges in the metadata would be my preferred way to go.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Mime
View raw message