cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Esken (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
Date Tue, 14 Mar 2017 10:05:41 GMT


Christian Esken commented on CASSANDRA-13265:

bq. This is not quite correct you can't count drainCount as dropped because some of the drained
messages may have been sent during iteration.
I looked in more detail, and I think this a flaw in the original code "suggested" me to do
this:  {{drainedMessages.clear()}} is called twice, and one time would be enough. IMO it would
be better to only keep the one at the end of the method and also do the drop-counting for
the drained messages there. This would also cover a rather exotic case of the {{catch (Exception
e)}} in the {{run()}} method. If an Exception is thrown, then there is a danger of nothing
being counted.

bq. Using a boxed integer
bq. You shouldn't need the check for null?
>From a brief check, this refers to a similar point. I saw many configuration options to
allow null and followed that route. I am absolutely happy to make it non-boxed.

bq. The right way to do it is create a branch for all the versions where this is going to
be fixed. Start at 2.2, merge to 3.0, merge to 3.11, then merge to trunk. 
At Github? I can do so. But no PR, right? I saw it mentioned that one should not open PR's
for Cassandra on Github as they cannot be handled (it's just a mirror).

> Expiration in OutboundTcpConnection can block the reader Thread
> ---------------------------------------------------------------
>                 Key: CASSANDRA-13265
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 1.8.0_112-b15)
> Linux 3.16
>            Reporter: Christian Esken
>            Assignee: Christian Esken
>             Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>         Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz,
> I observed that sometimes a single node in a Cassandra cluster fails to communicate to
the other nodes. This can happen at any time, during peak load or low load. Restarting that
single node from the cluster fixes the issue.
> Before going in to details, I want to state that I have analyzed the situation and am
already developing a possible fix. Here is the analysis so far:
> - A Threaddump in this situation showed  324 Threads in the OutboundTcpConnection class
that want to lock the backlog queue for doing expiration.
> - A class histogram shows 262508 instances of OutboundTcpConnection$QueuedMessage.
> What is the effect of it? As soon as the Cassandra node has reached a certain amount
of queued messages, it starts thrashing itself to death. Each of the Thread fully locks the
Queue for reading and writing by calling, making the situation worse and worse.
> - Writing: Only after 262508 locking operation it can progress with actually writing
to the Queue.
> - Reading: Is also blocked, as 324 Threads try to do, and fully lock
the Queue
> This means: Writing blocks the Queue for reading, and readers might even be starved which
makes the situation even worse.
> -----
> The setup is:
>  - 3-node cluster
>  - replication factor 2
>  - Consistency LOCAL_ONE
>  - No remote DC's
>  - high write throughput (100000 INSERT statements per second and more during peak times).

This message was sent by Atlassian JIRA

View raw message