cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Tunnicliffe (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-13304) Add checksumming to the native protocol
Date Mon, 13 Mar 2017 13:30:41 GMT


Sam Tunnicliffe updated CASSANDRA-13304:
    Status: Awaiting Feedback  (was: Open)

bq.TLS does solve but rotating of keys and assigning certs isn't a solved solution. Best to
be assured we don't corrupt people just because they don't have TLS.

Agreed and while using TLS should be a best practice, I don't like the idea of introducing
dependencies between features especially ones concerned with correctness.

A few headline comments on the patch: 
* I'm not particularly keen on the hijacking of the {{COMPRESSED}} header flag. Making compression
& checksumming properly orthogonal is possible, especially if we make it a mandatory requirement
for a given protocol version (I think we should). 
* I don't believe that bumping the version through to V6 is necessary. AFAIK, V5 is still
not released, so we could make this a feature/requirement of V5
* CRC32 is not threadsafe so as implemented, with singleton instances of the checksumming
compressors each with a single CRC32 instance, we're going to get corrupt checksums. 
* If the checksum implementation is switchable, then {{CHUNK_HEADER_OVERHEAD}} needs to be
calculated based on that implementation (or at least bumped to accomodate longs). 
In light of these last two points, I'd question whether we shouldn't just stick with CRC32,
especially as we use it extensively for checksums already (see also CASSANDRA-8684)

I've pushed a branch [here|] with proposals
to address the above points. Doing that included a fair bit of refactoring, so pushing a branch
just seemed easier than trying to describe everything in a review comment. Anyway, the changes
in this branch are really just suggestion so LMK what you think about them. 

We should also add some more comprehensive test coverage, particularly around the client/server
negotiation stage, handling of corrupt messages and other edge cases. I didn't add any tests
on my branch, but I believe the refactoring makes the components slightly more amenable to
unit testing too. 

Of course, the protocol spec will need to updated too if we get consensus. 

[~slebresne], [~aholmber], [~adutra] what do you all think of this?           

> Add checksumming to the native protocol
> ---------------------------------------
>                 Key: CASSANDRA-13304
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Michael Kjellman
>            Assignee: Michael Kjellman
>         Attachments: 13304_v1.diff
> The native binary transport implementation doesn't include checksums. This makes it highly
susceptible to silently inserting corrupted data either due to hardware issues causing bit
flips on the sender/client side, C*/receiver side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both client and
server know about a protocol version that supports checksums) -- and also adds checksumming
to clients that request compression.
> The serialized format looks something like this:
> {noformat}
>  *                      1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
>  *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Number of Compressed Chunks  |     Compressed Length (e1)    /
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * /  Compressed Length cont. (e1) |    Uncompressed Length (e1)   /
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Checksum of Lengths cont. (e1)|    Compressed Bytes (e1)    +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                      CRC32 Checksum (e1)                     ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                    Compressed Length (e2)                     |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                   Uncompressed Length (e2)                    |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                CRC32 Checksum of Lengths (e2)                 |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                     Compressed Bytes (e2)                   +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                      CRC32 Checksum (e2)                     ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                    Compressed Length (en)                     |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                   Uncompressed Length (en)                    |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                CRC32 Checksum of Lengths (en)                 |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                      Compressed Bytes (en)                  +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |                      CRC32 Checksum (en)                     ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame body itself
(and doesn't actually checksum lengths and headers). While it would be great to fully add
checksuming across the entire protocol, the proposed implementation will ensure we at least
catch corrupted data and likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor implementation
as it's been deprecated for a while -- is really slow and crappy compared to LZ4 -- and we
should do everything in our power to make sure no one in the community is still using it.
I left it in (for obvious backwards compatibility aspects) old for clients that don't know
about the new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized contents are
simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a FrameCompressor
inline. Unfortunately, we went with a decision to treat the frame body separately from the
header bits etc in a given message. So, instead we put a compressor implementation in the
options and then if it's not null, we push the serialized bytes for the frame body *only*
thru the given FrameCompressor implementation. The existing implementations simply provide
all the bytes for the frame body in one go to the compressor implementation and then serialize
it with the length of the compressed bytes up front.
> Unfortunately, this won't work for checksum'ing for obvious reasons as we can't naively
just checksum the entire (potentially) 256MB frame body and slap it at the end... so,
> The best place to start with the changes is in {{ChecksumedCompressor}}. I implemented
one single place to perform the checksuming (and to support checksuming) the actual required
chunking logic. Implementations of ChecksumedCompressor only implement the actual calls to
the given compression algorithm for the provided bytes.
> Although the interface takes a {{Checksum}}, right now the attached patch uses CRC32
everywhere. As of right now, given JDK8+ has support for doing the calculation with the Intel
instruction set, CRC32 is about as fast as we can get right now.
> I went with a 32kb "default" for the chunk size -- meaning we will chunk the entire frame
body into 32kb chunks, compress each one of those chunks, and checksum the chunk. Upon discussing
with a bunch of people and researching how checksums actually work and how much data they
will protect etc -- if we use 32kb chunks with CRC32 we can catch up to 32 bits flipped in
a row (but more importantly catch the more likely corruption where a single bit is flipped)
with pretty high certainty. 64kb seems to introduce too much of a probability of missing corruption.
> The maximum block size LZ4 operates on is a 64kb chunk -- so this combined with the need
to make sure the CRC32 checksums are actually going to catch stuff -- chunking at 32kb seemed
like a good reasonable value to use when weighing both checksums and compression (to ensure
we don't kill our compression ratio etc).
> I'm not including client changes here -- I asked around and I'm not really sure what
the policy there is -- do we update the python driver? java driver? how has the timing of
this stuff been handled in the past?

This message was sent by Atlassian JIRA

View raw message