cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-7360) CQLSSTableWriter consumes all memory for table with compound primary key
Date Fri, 05 Sep 2014 15:59:29 GMT


Sylvain Lebresne updated CASSANDRA-7360:
    Attachment: 7360-v2.txt

bq. I would have wrapped it into a UncheckedException (e.g. SyncException) that I would have
caught in rawAddRow and rethrow as IOException as it is more direct

You're right, that's probably cleaner overall, made that change in the v2 attached.

bq. I think that in BufferedWriter.close() the code should probably try to call super.close
before trying to rethrow the exception

It's fixed by the virtue of BufferedWriter not overriding close() anymore.

bq. I have the following error message in my log when I run the Unit tests

That is something we need to fix but that's completely unrelated to this ticket so not gonna
fix it here. But created CASSANDRA-7889 so we don't forget to scratch this itch.

> CQLSSTableWriter consumes all memory for table with compound primary key
> ------------------------------------------------------------------------
>                 Key: CASSANDRA-7360
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Xu Zhongxing
>            Assignee: Sylvain Lebresne
>             Fix For: 2.0.11
>         Attachments: 7360-v2.txt, 7360.txt
> When using CQLSSTableWriter to write a table with compound primary key, if the partition
key is identical for a huge amount of records, the sync() method is never called, and the
memory usage keeps growing until the memory is exhausted. 
> Could the code be improved to do sync() even when there is no new row  created? The relevant
code is in and I am new
to the code and cannot produce a reasonable patch for now.
> The problem can be reproduced by the following test case:
> {code}
> import;
> import org.apache.cassandra.exceptions.InvalidRequestException;
> import;
> import java.util.UUID;
> class SS {
>     public static void main(String[] args) {
>         String schema = "create table test.t (x uuid, y uuid, primary key (x, y))";
>         String insert = "insert into test.t (x, y) values (?, ?)";
>         CQLSSTableWriter writer = CQLSSTableWriter.builder()
>             .inDirectory("/tmp/test/t")
>             .forTable(schema).withBufferSizeInMB(32)
>             .using(insert).build();
>         UUID id = UUID.randomUUID();
>         try {
>             for (int i = 0; i < 50000000; i++) {
>                 UUID id2 = UUID.randomUUID();
>                 writer.addRow(id, id2);
>             }
>             writer.close();
>         } catch (Exception e) {
>             System.err.println("hell");
>         }
>     }
> }
> {code}

This message was sent by Atlassian JIRA

View raw message