hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Max Lehn <...@vomber.de>
Subject HBase breaks with NotReplicatedYetException after inserting some 100,000 rows
Date Mon, 22 Dec 2008 22:07:05 GMT
Hi.

I'm running HBase in a small distributed setup (3 machines). When I
import "larger" sets of data (around 700,000 rows within 20 minutes),
HBase eventually breaks and the logs show messages like

2008-12-17 13:56:20,235 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
could only be replicated to 0 nodes, instead of 1
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1270)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
(...)

and

2008-12-17 13:56:20,235 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
retries left 4

The Hadoop namenode logs show similar things, like

2008-12-17 13:56:20,098 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hbase,users,dialout,video	ip=/10.49.21.176	cmd=create	src=/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
dst=null	perm=hbase:supergroup:rw-r--r--
2008-12-17 13:56:20,230 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
enough replicas, still in need of 3
2008-12-17 13:56:20,232 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54310, call
addBlock(/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090,
DFSClient_-1955949534) from 10.49.21.176:39622: error:
java.io.IOException: File
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
could only be replicated to 0 nodes, instead of 1
java.io.IOException: File
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
could only be replicated to 0 nodes, instead of 1
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1270)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
(...)

but the other Hbase logs do not have any suspicious entries.

Does anyone know what this could be? I'm still new to Hadoop/HBase so
I don't really have an idea about what could be wrong.

Thanks,
Max



Mime
View raw message