hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From steven zhuang <steven.zhuang.1...@gmail.com>
Subject Re: get the impact hbase brings to HDFS, datanode log exploded after we started HBase.
Date Wed, 14 Apr 2010 04:41:01 GMT
hi, Andrew,
       Sorry, these days I have been working on something else.
       I know this progress, but I don't think this is my case.

      On the datanode where META table is stored, the huge number of
HDFS_READ seems are not caused by data importing too.  the DFS Client read
from the datanode and return the content to the same node.
      Seems there is a loop on the access port, I don't know what action in
HBase can cause this kind of read operation.

      I can paste some of the log I got today below, you can see the
destination ports are sequential, from 51586 to 51608, and they are reading
the same block(The META region?). There are actually millions lines of this
kind of record in some history HDFS log:
      I wonder if these records are generated when Master scans the META
table, but seems master will not scan using tens of thousands of successive
ports.
      I wonder if there is something wrong with HBase, out cluster uses
Hbase 0.20.3. and  Hadoop 0.20.1.

      Logs I got today:
2010-04-14 04:22:38,631 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51586*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,635 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51587*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,638 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51588*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,642 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51589*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,645 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51590*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,650 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51591*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,653 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51592*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,658 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:51593, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,662 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51594*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,665 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:51595, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,669 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51596*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,672 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:51597, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,676 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51598*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,679 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:51599, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,682 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51600*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,686 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:51601, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,690 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51602*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,694 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:51603, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,698 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51604*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,702 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51605*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,705 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51606*, bytes: 132096, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,709 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51607*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,713 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
10.76.112.212:50010, dest: /10.76.112.212:*51608*, bytes: 66048, op:
HDFS_READ, cliID: DFSClient_-430967455, srvID:
DS-2005911079-10.76.112.212-50010-1256672506882, blockid:
blk_4675655519784218461_1709237
2010-04-14 04:22:38,716 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.


On Sat, Apr 10, 2010 at 12:36 AM, Andrew Purtell <apurtell@apache.org>wrote:

> Steven,
>
> When loading data:
>
> - HBase caches data in RAM (memstore).
>
> - HBase periodically flushes the memstore to files in HDFS.
>
> - HBase periodically compacts the flush files. Compaction is READ of
> several flush files and WRITE of a new single file that is the merge.
>
> So during writes, especially heavy periods of writes, you can expect READs
> related to compaction. And typically they are nodeA -> nodeA.
>
> Hope this helps,
>
>   - Andy
>
> > From: steven zhuang
> > Subject: Re: get the impact hbase brings to HDFS, datanode log exploded
> after  we started HBase.
> > yeah, I see what you mean, and I know that Hbase will read
> > a lot when there
> > is random access.
> > but what I found is a little confusing to me, when we
> > import data into HBase
> > table, there shouldnot be many other reads, but on the
> > datanode "node-A"
> > where a regionserver serves the META table as well, the log
> > is 3.3GB,  there
> > were 10M HDFS_READ records in hdfs log, almost all of them
> > are from
> >  "node-A" to "node-A".
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message