cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaydeepkumar Chovatia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11551) Incorrect counting of pending messages in OutboundTcpConnection
Date Tue, 24 Apr 2018 18:19:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450341#comment-16450341
] 

Jaydeepkumar Chovatia commented on CASSANDRA-11551:
---------------------------------------------------

I think I've found the root cause for this issue, one possible reason is once we break at
[this point|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L270]
then {{currentMsgBufferCount}} doesn't get reset and at that time {{backlog.size()}} is 0
but {{currentMsgBufferCount}} could be > 0 and call to {{getPendingMessages}} at this time
will give > 0 pending message even though in reality it is none.

This problem will not happen most likely on {{trunk}} because it doesn't use {{currentMsgBufferCount}}
instead it directly deals with {{backlog.size()}} in [getPendingMessages|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/async/OutboundMessagingConnection.java#L659]

I've fixed this for older branches, please find fix here:
||3.11||3.0||2.2||
|[diff |https://github.com/apache/cassandra/compare/cassandra-3.11...jaydeepkumar1984:11551-3.11?expand=1]|[diff
|https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:11551-3.0?expand=1]|[diff|https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:11551-2.2?expand=1]|
|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-3.11.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/63]|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-3.0.svg?style=svg!
|https://circleci.com/gh/jaydeepkumar1984/cassandra/65]|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-2.2.svg?style=svg!
 |https://circleci.com/gh/jaydeepkumar1984/cassandra/66]|

Jaydeep

> Incorrect counting of pending messages in OutboundTcpConnection
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-11551
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11551
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Robert Stupp
>            Assignee: Jaydeepkumar Chovatia
>            Priority: Minor
>             Fix For: 2.2.x
>
>
> Somehow {{OutboundTcpConnection.getPendingMessages()}} seems to return a wrong number.
> {code}
> nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 1655
> Mismatch (Blocking): 0
> Mismatch (Background): 2
> Pool Name                    Active   Pending      Completed
> Large messages                  n/a         5              0
> Small messages                  n/a         0       31534100
> Gossip messages                 n/a         0         520393
> {code}
> Inspection of the heap dump of that node unveiled that all instances of {{OutboundTcpConnection.backlog}}
are empty but {{currentMsgBufferCount}} is {{1}} for 5 instances of {{OutboundTcpConnection}}.
> Maybe the cause is in {{OutboundTcpConnection.run()}} where {{drainedMessages.size()}}
is called twice but assumed that these are equal.
> /cc [~aweisberg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message