cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Lohfink (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-14867) Histogram overflows potentially leading to writes failing
Date Wed, 07 Nov 2018 00:48:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677468#comment-16677468
] 

Chris Lohfink commented on CASSANDRA-14867:
-------------------------------------------

That dropped messages logging runs at on a scheduled executor. It will not interrupt your
writes so it is mostly cosmetic. The fact that you had over 2^32 dropped messages is something
that you should be concerned about (something to move to mailing list).

The histogram throwing runtime exceptions has been recurring issue and is not really the right
approach. In some of the methods this instead returns max value and I think should just be
the result.

If these methods are going to throw exceptions they should not be runtime as the uses of it
rarely handle the condition.

> Histogram overflows potentially leading to writes failing
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-14867
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14867
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>         Environment: cassandra 3.11.1 on ubuntu 16.04
>            Reporter: David
>            Priority: Major
>
> I observed the following in cassandra logs on 1 host of a 6-node cluster:
> ERROR [ScheduledTasks:1] 2018-11-01 17:26:41,277 CassandraDaemon.java:228 - Exception
in thread Thread[ScheduledTasks:1,5,main]
> java.lang.IllegalStateException: Unable to compute when histogram overflowed
>  at org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot.getMean(DecayingEstimatedHistogramReservoir.java:472)
~[apache-cassandra-3.11.1.jar:3.11.1]
>  at org.apache.cassandra.net.MessagingService.getDroppedMessagesLogs(MessagingService.java:1263)
~[apache-cassandra-3.11.1.jar:3.11.1]
>  at org.apache.cassandra.net.MessagingService.logDroppedMessages(MessagingService.java:1236)
~[apache-cassandra-3.11.1.jar:3.11.1]
>  at org.apache.cassandra.net.MessagingService.access$200(MessagingService.java:87) ~[apache-cassandra-3.11.1.jar:3.11.1]
>  at org.apache.cassandra.net.MessagingService$4.run(MessagingService.java:507) ~[apache-cassandra-3.11.1.jar:3.11.1]
>  at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
~[apache-cassandra-3.11.1.jar:3.11.1]
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_172]
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_172]
>  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_172]
>  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_172]
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_172]
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
>  at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
>  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_172]
>  
> At the same time, this node was failing all writes issued to it. Restarting cassandra
on the node brought the cluster into a good state and we stopped seeing the histogram overflow
errors.
> Has this issue been observed before? Could the histogram overflows cause writes to fail?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message