cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-11053) COPY FROM on large datasets: fix progress report and debug performance
Date Tue, 02 Feb 2016 09:53:40 GMT


Stefania commented on CASSANDRA-11053:

Another approach I am looking at is to continue reading in the parent process, possibly via
memory mapped files, and to only move the csv decoding to the worker processes. This would
be less disruptive in the existing design. I also note that we will still need to improve
worker processes performance as well, since they only spend about 30 seconds receiving, something
else needs to improve. Since most of the time consuming methods are in the driver I would
like to try and get the cythonized driver to work as well.

Sorry for the long chain of comments. However I would really appreciate any further ideas,
without taking too much of your time.

> COPY FROM on large datasets: fix progress report and debug performance
> ----------------------------------------------------------------------
>                 Key: CASSANDRA-11053
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Stefania
>            Assignee: Stefania
>             Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>         Attachments: copy_from_large_benchmark.txt, parent_profile.txt, worker_profiles.txt
> Running COPY from on a large dataset (20G divided in 20M records) revealed two issues:
> * The progress report is incorrect, it is very slow until almost the end of the test
at which point it catches up extremely quickly.
> * The performance in rows per second is similar to running smaller tests with a smaller
cluster locally (approx 35,000 rows per second). As a comparison, cassandra-stress manages
50,000 rows per second under the same set-up, therefore resulting 1.5 times faster. 
> See attached file _copy_from_large_benchmark.txt_ for the benchmark details.

This message was sent by Atlassian JIRA

View raw message