cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
Date Mon, 15 Sep 2014 23:12:34 GMT


Benedict commented on CASSANDRA-7546:

Hi Graham,

I must admit I'm a bit confused, and it's partially self inflicted. In 2.1.1 we have changed
stress again from what we released in 2.1.0, and I can't tell which version you're referring
to, though it seems 2.1.1. Neither version has a 'visits' property in the yaml, but 2.1.1
supports -insert visits= revisit=, which are certainly functions worth exploring and I recommend
you use 2.1.1 for stress functionality either way. 

As far as using these functions are concerned, 'visits' splits a wide row up into multiple
inserts; if a visits value of 10 is produced, and there are on average 100 rows generated
for the partition, approximately 10 rows will be inserted, then the state of the partition
will be stashed away and the next insert that operates on that partition will pick up where
the previous one left off. Which partition is performed next is decided by the 'revisit' distribution,
which selects from the stash of partially completed inserts, with a value of 1 selecting the
most recently stashed (the max value of this distribution defines the total number of partitions
to stash); if it ever selects outside of the current stash a new partition is generated instead.

So the value for 'visits' is related to the number of unique clustering columns you generate
for each partition, whereas the value for revisit is determined by how diverse the data you
operate over in a given time window is.  

Separately, it's worth mentioning that offheap_objects is likely a better choice than offheap_buffers,
since it is considerably more memory dense.

> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>                 Key: CASSANDRA-7546
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: graham sanderson
>            Assignee: graham sanderson
>             Fix For: 2.1.1
>         Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt,
7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt,
hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png
> In order to preserve atomicity, this code attempts to read, clone/update, then CAS the
state of the partition.
> Under heavy contention for updating a single partition this can cause some fairly staggering
memory growth (the more cores on your machine the worst it gets).
> Whilst many usage patterns don't do highly concurrent updates to the same partition,
hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory
allocation rates can be seen (especially when the updates being hinted are small updates to
different partitions which can happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation whilst not
slowing down the very common un-contended case.

This message was sent by Atlassian JIRA

View raw message