cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Lerer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7360) CQLSSTableWriter consumes all memory for table with compound primary key
Date Mon, 08 Sep 2014 11:19:29 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125416#comment-14125416
] 

Benjamin Lerer commented on CASSANDRA-7360:
-------------------------------------------

Just one nitpick: you do not need to create an ioe field as Throwable allow exception nesting.
You can just call the RuntimeException(Throwable cause) constructor via super(ioe) and retrieve
the Exception via getCause() in rawAddRow. 


> CQLSSTableWriter consumes all memory for table with compound primary key
> ------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7360
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7360
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Xu Zhongxing
>            Assignee: Sylvain Lebresne
>             Fix For: 2.0.11
>
>         Attachments: 7360-v2.txt, 7360.txt
>
>
> When using CQLSSTableWriter to write a table with compound primary key, if the partition
key is identical for a huge amount of records, the sync() method is never called, and the
memory usage keeps growing until the memory is exhausted. 
> Could the code be improved to do sync() even when there is no new row  created? The relevant
code is in SSTableSimpleUnsortedWriter.java and AbstractSSTableSimpleWriter.java. I am new
to the code and cannot produce a reasonable patch for now.
> The problem can be reproduced by the following test case:
> {code}
> import org.apache.cassandra.io.sstable.CQLSSTableWriter;
> import org.apache.cassandra.exceptions.InvalidRequestException;
> import java.io.IOException;
> import java.util.UUID;
> class SS {
>     public static void main(String[] args) {
>         String schema = "create table test.t (x uuid, y uuid, primary key (x, y))";
>         String insert = "insert into test.t (x, y) values (?, ?)";
>         CQLSSTableWriter writer = CQLSSTableWriter.builder()
>             .inDirectory("/tmp/test/t")
>             .forTable(schema).withBufferSizeInMB(32)
>             .using(insert).build();
>         UUID id = UUID.randomUUID();
>         try {
>             for (int i = 0; i < 50000000; i++) {
>                 UUID id2 = UUID.randomUUID();
>                 writer.addRow(id, id2);
>             }
>             writer.close();
>         } catch (Exception e) {
>             System.err.println("hell");
>         }
>     }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message