hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Li <fancye...@gmail.com>
Subject two rs nodes crashed
Date Mon, 12 May 2014 07:09:57 GMT
by reading log, it seems the region server suffered from gc pause time.
my region server jvm arguments:
r -XX:OnOutOfMemoryError=kill -9 %p -Xmx6000m -server -XX:NewSize=512m
-XX:MaxNewSize=1024m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=70


2014-05-12 13:31:52,216 INFO  [MemStoreFlusher.0] util.FSUtils:
FileSystem doesn't support getDefaultBlockSize
2014-05-12 13:31:52,637 INFO  [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=71654928,
memsize=64.6m, hasBloomFilter=false, into tmp file
hdfs://192.168.10.48:8020/hbase/data/default/vc2.out_link/7152a7fe13b6befa889799ef9c5742d0/.tmp/8d72b3a10983466f8095e187afb65866
2014-05-12 13:31:52,642 DEBUG [MemStoreFlusher.0]
regionserver.HRegionFileSystem: Committing store file
hdfs://192.168.10.48:8020/hbase/data/default/vc2.out_link/7152a7fe13b6befa889799ef9c5742d0/.tmp/8d72b3a10983466f8095e187afb65866
as hdfs://192.168.10.48:8020/hbase/data/default/vc2.out_link/7152a7fe13b6befa889799ef9c5742d0/cf/8d72b3a10983466f8095e187afb65866
2014-05-12 13:31:52,646 INFO  [MemStoreFlusher.0] regionserver.HStore:
Added hdfs://192.168.10.48:8020/hbase/data/default/vc2.out_link/7152a7fe13b6befa889799ef9c5742d0/cf/8d72b3a10983466f8095e187afb65866,
entries=349073, sequenceid=71654928, filesize=21.1m
2014-05-12 13:31:52,646 INFO  [MemStoreFlusher.0]
regionserver.HRegion: Finished memstore flush of ~64.6m/67700088,
currentsize=0.0/0 for region
vc2.out_link,\xD4,1399794095886.7152a7fe13b6befa889799ef9c5742d0. in
444ms, sequenceid=71654928, compaction requested=true
2014-05-12 13:31:52,647 DEBUG [MemStoreFlusher.0]
regionserver.CompactSplitThread: Small Compaction requested: system;
Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0,
merge_queue=0
2014-05-12 13:31:52,647 DEBUG
[regionserver60020-smallCompactions-1399597823428]
compactions.RatioBasedCompactionPolicy: Selecting compaction from 5
store files, 0 compacting, 5 eligible, 10 blocking
2014-05-12 13:31:52,647 DEBUG
[regionserver60020-smallCompactions-1399597823428]
compactions.ExploringCompactionPolicy: Exploring compaction algorithm
has selected 0 files of size 0 starting at candidate #-1 after
considering 6 permutations with 0 in ratio
2014-05-12 13:31:52,647 DEBUG
[regionserver60020-smallCompactions-1399597823428]
compactions.RatioBasedCompactionPolicy: Not compacting files because
we only have 0 files ready for compaction. Need 3 to initiate.
2014-05-12 13:31:52,647 DEBUG
[regionserver60020-smallCompactions-1399597823428]
regionserver.CompactSplitThread: Not compacting
vc2.out_link,\xD4,1399794095886.7152a7fe13b6befa889799ef9c5742d0.
because compaction request was cancelled
2014-05-12 13:33:26,540 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1725ms
GC pool 'ParNew' had collection(s): count=1 time=276ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1462ms
2014-05-12 13:34:03,916 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1487ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1557ms
2014-05-12 13:34:30,846 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1216ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1565ms
2014-05-12 13:34:47,284 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1204ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1467ms
2014-05-12 13:34:54,791 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1005ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1457ms
2014-05-12 13:35:00,206 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1413ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1445ms
2014-05-12 13:35:03,369 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1662ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1818ms
2014-05-12 13:35:05,489 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1119ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1246ms
2014-05-12 13:35:12,693 INFO  [JvmPauseMonitor] util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately
1057ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=3 time=1558ms
2014-05-12 13:35:14,655 INFO  [main] zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52
GMT
2014-05-12 13:35:14,656 INFO  [main] zookeeper.ZooKeeper: Client
environment:host.name=app-hbase-2
2014-05-12 13:35:14,656 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_03
2014-05-12 13:35:14,656 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2014-05-12 13:35:14,656 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.7.0_03/jre
2014-05-12 13:35:14,656 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.class.path=/home/hadoop/hbase/bin/../conf:/usr/java/jdk1.7.0_03//lib/tools.jar:/home/hadoop/hbase/bin/..:/home/hadoop/hbase/bin/../lib/activation-1.1.jar:/home/hadoop/hbase/bin/../lib/asm-3.1.jar:/home/hadoop/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/hadoop/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hbase/bin/../lib/commons-cli-1.2.jar:/home/hadoop/hbase/bin/../lib/commons-codec-1.7.jar:/home/hadoop/hbase/bin/../lib/commons-collections-3.2.1.jar:/home/hadoop/hbase/bin/../lib/commons-configuration-1.6.jar:/home/hadoop/hbase/bin/../lib/commons-digester-1.8.jar:/home/hadoop/hbase/bin/../lib/commons-el-1.0.jar:/home/hadoop/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/hbase/bin/../lib/commons-io-2.4.jar:/home/hadoop/hbase/bin/../lib/commons-lang-2.6.jar:/home/hadoop/hbase/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/hbase/bin/../lib/commons-math-2.1.jar:/home/hadoop/hbase/bin/../lib/commons-net-1.4.1.jar:/home/hadoop/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/hadoop/hbase/bin/../lib/guava-12.0.1.jar:/home/hadoop/hbase/bin/../lib/hadoop-core-1.0.0.jar:/home/hadoop/hbase/bin/../lib/hamcrest-core-1.3.jar:/home/hadoop/hbase/bin/../lib/hbase-client-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-common-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-common-0.96.2-hadoop1-tests.jar:/home/hadoop/hbase/bin/../lib/hbase-examples-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-hadoop1-compat-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-hadoop-compat-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-it-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-it-0.96.2-hadoop1-tests.jar:/home/hadoop/hbase/bin/../lib/hbase-prefix-tree-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-protocol-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-server-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-server-0.96.2-hadoop1-tests.jar:/home/hadoop/hbase/bin/../lib/hbase-shell-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-testing-util-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/hbase-thrift-0.96.2-hadoop1.jar:/home/hadoop/hbase/bin/../lib/htrace-core-2.04.jar:/home/hadoop/hbase/bin/../lib/httpclient-4.1.3.jar:/home/hadoop/hbase/bin/../lib/httpcore-4.1.3.jar:/home/hadoop/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hbase/bin/../lib/jackson-xc-1.8.8.jar:/home/hadoop/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/home/hadoop/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/hadoop/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/hadoop/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/hadoop/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hbase/bin/../lib/jersey-core-1.8.jar:/home/hadoop/hbase/bin/../lib/jersey-json-1.8.jar:/home/hadoop/hbase/bin/../lib/jersey-server-1.8.jar:/home/hadoop/hbase/bin/../lib/jettison-1.3.1.jar:/home/hadoop/hbase/bin/../lib/jetty-6.1.26.jar:/home/hadoop/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/hadoop/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/hadoop/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/hadoop/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/hadoop/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/hadoop/hbase/bin/../lib/jsr305-1.3.9.jar:/home/hadoop/hbase/bin/../lib/junit-4.11.jar:/home/hadoop/hbase/bin/../lib/libthrift-0.9.0.jar:/home/hadoop/hbase/bin/../lib/log4j-1.2.17.jar:/home/hadoop/hbase/bin/../lib/metrics-core-2.1.2.jar:/home/hadoop/hbase/bin/../lib/netty-3.6.6.Final.jar:/home/hadoop/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/hadoop/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hbase/bin/../lib/slf4j-api-1.6.4.jar:/home/hadoop/hbase/bin/../lib/slf4j-log4j12-1.6.4.jar:/home/hadoop/hbase/bin/../lib/xmlenc-0.52.jar:/home/hadoop/hbase/bin/../lib/zookeeper-3.4.5.jar:
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-279.el6.x86_64
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.name=hadoop
2014-05-12 13:35:14,657 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.home=/home/hadoop
2014-05-12 13:35:14,658 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.dir=/home/hadoop/hbase
2014-05-12 13:35:14,661 INFO  [main] zookeeper.ZooKeeper: Initiating
client connection,
connectString=192.168.10.48:2181,192.168.10.47:2181,192.168.10.50:2181
sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@6735a4c2
2014-05-12 13:35:14,717 INFO  [main-SendThread(app-hbase-master:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
app-hbase-master/192.168.10.48:2181. Will not attempt to authenticate
using SASL (unknown error)
2014-05-12 13:35:14,728 INFO  [main-SendThread(app-hbase-master:2181)]
zookeeper.ClientCnxn: Socket connection established to
app-hbase-master/192.168.10.48:2181, initiating session
2014-05-12 13:35:14,740 INFO  [main-SendThread(app-hbase-master:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
app-hbase-master/192.168.10.48:2181, sessionid = 0x145de87b0da2b9f,
negotiated timeout = 30000

Mime
View raw message