cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fernando Gonçalves (JIRA) <>
Subject [jira] [Commented] (CASSANDRA-7953) RangeTombstones not merging during compaction
Date Thu, 15 Oct 2015 12:58:06 GMT


Fernando Gonçalves commented on CASSANDRA-7953:

We have experienced the same behaviour described on ticket []:
"Once this happens in multiple sstables, compacting them causes the duplication to grow. The
more this occurs, the worse the problem gets."

Basically, when we run the repair in the node, the compaction process starts and never ends,
many pending tasks, and the number of sstables of one table grows exponentially (reaching
34k sstables). We use one column of the type map<text, text>, LeveledStrategyCompaction,
and many updates to this column. The memory consumption grows a lot too. We decide to stop
the repair process and kill the node, because the latency grown a lot too and impact the whole
cluster. But we need to run repair again, because we kill one node and put a new node on the
cluster, and another node was impacted from this bug again, and we need to repeated the process:
kill the repair process, kill the node, start a new node.

So we just created another table without using the collection map, but using blob type instead,
and migrate all the data for it - we are fine now: the repair process and compaction finished
successfully without big impact on performance.

Please, give attention for this ticket, I think that its a major issue!

> RangeTombstones not merging during compaction
> ---------------------------------------------
>                 Key: CASSANDRA-7953
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.1
>            Reporter: Marcus Olsson
>            Assignee: Branimir Lambov
>            Priority: Minor
>              Labels: compaction, deletes, tombstone
>             Fix For: 2.1.x, 2.2.x
>         Attachments: 0001-7953-v2.patch, CASSANDRA-7953-1.patch, CASSANDRA-7953.patch
> When performing a compaction on two sstables that contain the same RangeTombstone with
different timestamps, the tombstones are not merged in the new sstable.
> This has been tested using cassandra 2.1 with the following table:
> {code}
>   key text,
>   column text,
>   data text,
>   PRIMARY KEY(key, column)
> );
> {code}
> And then doing the following:
> {code}
> INSERT INTO test (key, column, data) VALUES ("1", "1", "1"); // If the sstable only contains
tombstones during compaction it seems that the sstable either gets removed or isn't created
(but that could probably be a separate JIRA issue).
> INSERT INTO test (key, column, data) VALUES ("1", "2", "2"); // The inserts are not actually
needed, since the deletes create tombstones either way.
> DELETE FROM test WHERE key="1" AND column="2";
> nodetool flush
> INSERT INTO test (key, column, data) VALUES ("1", "2", "2");
> DELETE FROM test WHERE key="1" AND column="2";
> nodetool flush
> nodetool compact
> {code}
> When checking with the SSTableExport tool two tombstones exists in the compacted sstable.
This can be repeated, resulting in more and more tombstones.

This message was sent by Atlassian JIRA

View raw message