hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: hadoop fs -du & hbase table size
Date Mon, 14 Mar 2011 19:27:46 GMT
On Mon, Mar 14, 2011 at 7:34 AM, Alex Baranau <alex.baranov.v@gmail.com> wrote:
> As far as I understand, since "hadoop fs -du" command uses Linux' "du"
> internally this mean that the number of replicas (at the moment of command
> run) affect the result. Is that correct?
>

Yes.  I believe so.  You'll have more blocks if you have more
replication so du will report larger sizes.


> At some point I decided to reconfigure one of the slaves and shut it down.
> After reconfiguration (HBase already marked it as dead one) I brought it up
> again. Things went smoothly. However on the table size graph (I drew from
> data fetched with "hadoop fs -du" command) I noticed a little spike up on
> data size and then it went down to the normal/expected values. Can it be so
> that at some point of the taking out/reconfiguring/adding back node
> procedure at some point blocks were over-replicated? I'd expect them to be
> under-replicated for some time (as DN is down) and I'd expect to see the
> inverted spike: small decrease in data amount and then back to "expected"
> rate (after all blocks got replicated again). Any ideas?
>

Could it be that the du was counting the downed DNs blocks for a
while.  The spike was while re-replicating the downed blocks (counting
the downed DNs blocks).   When you brought back the old DN, NN told it
clean up blocks it had replicated elsewhere?

St.Ack

Mime
View raw message