cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkata Harikrishna Nukala (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-14647) Reading cardinality from Statistics.db failed
Date Fri, 01 Mar 2019 15:15:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781765#comment-16781765
] 

Venkata Harikrishna Nukala commented on CASSANDRA-14647:
--------------------------------------------------------

[~krummas] Updated the patch. 
||Branch||CircleCI||
|[14647-trunk\|https://github.com/apache/cassandra/compare/trunk...nvharikrishna:14647-trunk]|[link\|https://circleci.com/gh/nvharikrishna/cassandra/3#tests/containers/2]|

> Reading cardinality from Statistics.db failed
> ---------------------------------------------
>
>                 Key: CASSANDRA-14647
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14647
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Observability/Metrics
>         Environment: Clients are doing only writes with Local One, cluster consist of
3 regions with RF3.
> Storage is configured wth jbod/XFS on 10 x 1Tb disks
> IOPS limit for each disk 500 (total 5000 iops)
> Bandwith for each disk 60mb/s (600 total)
> OS is Debian linux.
>            Reporter: Vitali Djatsuk
>            Assignee: Venkata Harikrishna Nukala
>            Priority: Minor
>             Fix For: 3.0.x
>
>         Attachments: 14647-trunk-1.patch, cassandra_compaction_pending_tasks_7days.png
>
>
> There is some issue with sstable metadata which is visible in system.log, the messages
says:
> {noformat}
> WARN  [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading cardinality
from Statistics.db failed for /opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
> Although there is no such file. 
> The message has appeared after i've changed the compaction strategy from SizeTiered to
Leveled. Compaction strategy has been changed region by region (total 3 regions) and it has
coincided with the double client write traffic increase.
>  I have tried to run nodetool scrub to rebuilt the sstable, but that does not fix the
issue.
> So very hard to define the steps to reproduce, probably it will be:
>  # run stress tool with write traffic
>  # under load change compaction strategy from SireTiered to Leveled for the bunch of
hosts
>  # add more write traffic
> Reading the code it is said that if this metadata is broken, then "estimating the keys
will be done using index summary". 
>  [https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message