cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <>
Subject [jira] Resolved: (CASSANDRA-1495) Server doesn't seem to close SSTables correctly, and ends up with too many file descriptors open
Date Mon, 13 Sep 2010 00:55:33 GMT


Brandon Williams resolved CASSANDRA-1495.

      Assignee: Brandon Williams
    Resolution: Duplicate

Closed as duplicate.  By the way, I did not search jira to find the issue, I knew that we'd
already fixed an FD leak since b1, so I looked at CHANGES.txt in trunk to find it.  Incidentally,
1410 was right after it, which is why I linked the wrong ticket the first time. :)

> Server doesn't seem to close SSTables correctly, and ends up with too many file descriptors
> ------------------------------------------------------------------------------------------------
>                 Key: CASSANDRA-1495
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7 beta 1
>         Environment: 0.7 beta1. Built from:
using dpkg-buildpackage from it. 
>            Reporter: Josep M. Blanquer
>            Assignee: Brandon Williams
>             Fix For: 0.7 beta 2
> Cassandra server accumulates many open file descriptors as the time progresses. Obviously
when it hits whatever limit the system allows, the service stops accepting messages.
> The exact trace is:
> WARN 05:41:47,929 Transport error occurred during acceptance of message.
> org.apache.thrift.transport.TTransportException: Too many open
>         at org.apache.thrift.transport.TServerSocket.acceptImpl(
>         at org.apache.thrift.transport.TServerSocket.acceptImpl(
>         at org.apache.thrift.transport.TServerTransport.accept(
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(
>         at org.apache.cassandra.thrift.CassandraDaemon.start(
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>         at java.lang.reflect.Method.invoke(
>         at
> Caused by: Too many open files
>         at Method)
>         at
>         at
>         at
>         at org.apache.thrift.transport.TServerSocket.acceptImpl(
>         ... 9 more
> I've increased the ulimit to 8K from the standard 1K ... but still gets hit. So it seems
that basically there are many fd's that never get closed.
> lsof shows that there are many fd's hanging on to SSTables...albeit it's only a small
subset of unique files. In my case there were only about 4 distinct SSTable files...but kept
opened hundreds of time each. 
> All of these files seem to be "Data" Filter or Index ones (as far as I remember)
> In case it matters:  DiskAccessMode 'auto' determined to be standard, indexAccessMode
is standard
> The service doesn't have a high rate of writes at the moment...I have 3 nodes, 1 being
receiving mostly the thrift entry point. I see the problem being more acute on the 2 nodes
that instead of being contacted by the clients, they just get the writes from the coordinator...(I
have RF=2). 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message