cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Brown (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-4663) Streaming sends one file at a time serially.
Date Tue, 03 Jan 2017 20:49:58 GMT


Jason Brown commented on CASSANDRA-4663:

[~iksaif] Unfortunately, I believe this patch isn't going to do what you want. In the existing
code, {{connectionsPerHost}} is passed down to {{StreamCoordinator}}, where it is primarily
going to be used on the side that is transferring files
In {{StreamCoordinator#sliceSSTableDetails()}}, we essentially round robin the sstable chunks
across the number of connections we will use. It's also referenced from {{HostStreamingData#getOrCreateNextSession()}},
but that's not going to increase the inbound sessions (see next paragraph).

Your current patch sets the {{connectionsPerHost}} at the receiver end of the stream (even
though it would be the node that is initiating the stream session). As it's a bootstrapping
node that is requesting the ranges from the peer, and given the structure and protocol of
the current stream session implementation (session and protocol are essentially sequential,
single threaded, and expect a single socket), I think you'd need to get down into session
and protocol management code (and muck with a whole lot) in order to to parallelize from the
(non-initiating) sending node. 

I'm happy to be wrong about my understanding here, so if you've tested it out and have seen
good gains, please share :).

> Streaming sends one file at a time serially. 
> ---------------------------------------------
>                 Key: CASSANDRA-4663
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: sankalp kohli
>            Priority: Minor
>             Fix For: 3.x
>         Attachments: 0001-streaming-add-a-way-to-configure-the-number-of-conne.patch
> This is not fast enough when someone is using SSD and may be 10G link. We should try
to create multiple connections and send multiple files in parallel. 
> Current approach under utilize the link(even 1G).
> This change will improve the bootstrapping time of a node. 

This message was sent by Atlassian JIRA

View raw message