cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thibaut (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-2543) Node not responding, bringing down cluster, marked as up
Date Wed, 27 Apr 2011 12:48:03 GMT


Thibaut commented on CASSANDRA-2543:

I simply removed the table, and everything works fine again, even at smaller memory footprints.

I did backup the files, so I will try to create test case with only one table. But it will
take some time.

I did tune the memtable_throughput value and did get the OOM exception even with thrift &
gossip disabled. 

It's similar to
 I also wrote lot's of small entries (about 30-50KB per Entry).

> Node not responding, bringing down cluster, marked as up
> --------------------------------------------------------
>                 Key: CASSANDRA-2543
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>         Attachments: jstack
> I have one node which constantly hangs and brings done the entire cluster (not giving
any answers).
> If I restart the node, the node will hang after a certain number of time. I have no indication
> It's marked as up when executing the nodetool ring command.
> Executing the ring command on the node itself (without any traffic on the cluster) takes
at least 2 minutes to execute. The node takes about 50%-100% of cpu over all cpus.
> Netstats doesn't list anything interesting:
> /software/cassandra/bin/nodetool -h localhost netstats
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0          51064
> Responses                       n/a         0         530479
> I attached the jstack of the node. There are no indications that the node has faulty
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms5254M -Xmx5254M
-Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes
-cp /software/cassandra/bin/../conf:/software/cassandra/bin/../build/classes:/software/cassandra/bin/../lib/antlr-3.1.3.jar:/software/cassandra/bin/../lib/apache-cassandra-0.7.4.jar:/software/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/software/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/software/cassandra/bin/../lib/commons-cli-1.1.jar:/software/cassandra/bin/../lib/commons-codec-1.2.jar:/software/cassandra/bin/../lib/commons-collections-3.2.1.jar:/software/cassandra/bin/../lib/commons-lang-2.4.jar:/software/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:/software/cassandra/bin/../lib/guava-r05.jar:/software/cassandra/bin/../lib/high-scale-lib.jar:/software/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/software/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/software/cassandra/bin/../lib/jetty-6.1.21.jar:/software/cassandra/bin/../lib/jetty-util-6.1.21.jar:/software/cassandra/bin/../lib/jline-0.9.94.jar:/software/cassandra/bin/../lib/json-simple-1.1.jar:/software/cassandra/bin/../lib/jug-2.0.0.jar:/software/cassandra/bin/../lib/libthrift-0.5.jar:/software/cassandra/bin/../lib/log4j-1.2.16.jar:/software/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/software/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/software/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/software/cassandra/bin/../lib/snakeyaml-1.6.jar

This message is automatically generated by JIRA.
For more information on JIRA, see:

View raw message