cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thibaut (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (CASSANDRA-2394) Faulty hd kills cluster performance
Date Thu, 05 May 2011 22:47:03 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029630#comment-13029630
] 

Thibaut edited comment on CASSANDRA-2394 at 5/5/11 10:45 PM:
-------------------------------------------------------------

I did grep the cassandra log for errors and no error showed up. 
There were errors in kern.log. (reading errors) On another server before, after seeing the
reading errors, I did try to copy the cassandra data directory and it caused input/output
errors. I didn't try this on the latest server where this error occured, but I suppose it
would have caused input/output errors as well.

Yes, it will still repond from any node (our application connects locally first, and these
nodes are also not replying), but very very very slowly (e.g. maybe 1% of normal cluster performance)
. Before it did show up on the entire cluster a massive commands/responds pending queue (>
1000, version 0.7.4 on all 20 nodes as far as I can remember). Normally the queue has about
0-5 entries at most. 
I didn't output it when it occured on the latest server. Iterating over a table with hector
will be continoulsy paused by timeout exceptions. Even if our application is turned off and
this is the only application running. I'm also using the dynamic snitch with 0.8 badness threshold.
Replication level is set to 3 (EDIT 3 that ist). So if one node goes down, it should still
work as 2 nodes are still available. 

And as I said before, stopping cassandra on the faulty node will nearly instantly return proper
cluster performance.

What can help you? Shall I do a jstack trace next time the error occurs? But this can take
some time until a hd dies. Do you know of a tool where I can simulate a hd error (or very
slow read)?


      was (Author: tbritz):
    I did grep the cassandra log for errors and no error showed up. 
There were errors in kern.log. (reading errors) On another server before, after seeing the
reading errors, I did try to copy the cassandra data directory and it caused input/output
errors. I didn't try this on the latest server where this error occured, but I suppose it
would have caused input/output errors as well.

Yes, it will still repond from any node (our application connects locally first, and these
nodes are also not replying), but very very very slowly (e.g. maybe 1% of normal cluster performance)
. Before it did show up on the entire cluster a massive commands/responds pending queue (>
1000, version 0.7.4 on all 20 nodes as far as I can remember). Normally the queue has about
0-5 entries at most. 
I didn't output it when it occured on the latest server. Iterating over a table with hector
will be continoulsy paused by timeout exceptions. Even if our application is turned off and
this is the only application running. I'm also using the dynamic snitch with 0.8 badness threshold.
Replication level is set to 4. So if one node goes down, it should still work as 2 nodes are
still available. 

And as I said before, stopping cassandra on the faulty node will nearly instantly return proper
cluster performance.

What can help you? Shall I do a jstack trace next time the error occurs? But this can take
some time until a hd dies. Do you know of a tool where I can simulate a hd error (or very
slow read)?

  
> Faulty hd kills cluster performance
> -----------------------------------
>
>                 Key: CASSANDRA-2394
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2394
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>            Priority: Minor
>             Fix For: 0.7.6
>
>
> Hi,
> About every week, a node from our main cluster (>100 nodes) has a faulty hd  (Listing
the cassandra data storage directoy triggers an input/output error).
> Whenever this occurs, I see many timeoutexceptions in our application on various nodes
which cause everything to run very very slowly. Keyrange scans just timeout and will sometimes
never succeed. If I stop cassandra on the faulty node, everything runs normal again.
> It would be great to have some kind of monitoring thread in cassandra which marks a node
as "down" if there are multiple read/write errors to the data directories. A single faulty
hd on 1 node shouldn't affect global cluster performance.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message