cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "NIcholas Rushton (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric
Date Wed, 18 Jan 2017 16:54:26 GMT
NIcholas Rushton created CASSANDRA-13133:

             Summary: Unclosed file descriptors when querying SnapshotsSize metric
                 Key: CASSANDRA-13133
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: CentOS 7
            Reporter: NIcholas Rushton
             Fix For: 3.9

Started to notice many open file descriptors (100k+) per node, growing at a rate of about
30 per minute in our cluster. After turning off our JMX exporting server which gets queried
every 30 seconds, the number of file descriptors remained static. 

Digging a bit further I ran a jmx dump tool over all the cassandra metrics and tracked the
number of file descriptors after each query, boiling it down to a single metric causing the
number of file descriptors to increase:


running a query a few times against this metric shows the file descriptors increasing after
each query:

for _ in {0..3} 
   java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> /dev/null; 
   sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk '{a[$(NF)]+=1}END{for(k in
a){print k, a[k]}}' | grep "events_by" 

> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71 33176
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71 33177
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71 33178
> /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71 33179

This message was sent by Atlassian JIRA

View raw message