hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: get the impact hbase brings to HDFS, datanode log exploded after we started HBase.
Date Fri, 09 Apr 2010 16:36:48 GMT
Steven,

When loading data:

- HBase caches data in RAM (memstore).

- HBase periodically flushes the memstore to files in HDFS.

- HBase periodically compacts the flush files. Compaction is READ of several flush files and
WRITE of a new single file that is the merge.

So during writes, especially heavy periods of writes, you can expect READs related to compaction.
And typically they are nodeA -> nodeA. 

Hope this helps,

   - Andy

> From: steven zhuang
> Subject: Re: get the impact hbase brings to HDFS, datanode log exploded after  we started
HBase.
> yeah, I see what you mean, and I know that Hbase will read
> a lot when there
> is random access.
> but what I found is a little confusing to me, when we
> import data into HBase
> table, there shouldnot be many other reads, but on the
> datanode "node-A"
> where a regionserver serves the META table as well, the log
> is 3.3GB,  there
> were 10M HDFS_READ records in hdfs log, almost all of them
> are from
>  "node-A" to "node-A".



      


Mime
View raw message