hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Adrien <a...@jeanjean.ch>
Subject Re: Dataloss HBASE-728
Date Fri, 17 Oct 2008 10:09:43 GMT


stack-3 wrote:
> 
> 
> Yes.  Could have been deadlocked.  Next time send the "-QUIT" signal and 
> you'll get a threaddump in the regionserver .out file.
> 
> 

Ok. I'll try this if the case happens again. 


stack-3 wrote:
> 
> 
> Disks full on your HDFS?
> 
> 

Yes. One of the disk is full. Disk size are not heterogeneous. 
Moreover I tried to keep a part of space free for other usage, using the
following hadoop directives:

<property>
  <name>dfs.datanode.du.reserved</name>
  <value>4000000000</value>
  <description>Reserved space in bytes per volume. Always leave this much
space free for non dfs use.
  </description>
</property>

But free space goes under 1 Gb, as if I kept the default settings
Then I added

<property>
  <name>dfs.datanode.du.pct</name>
  <value>0.9f</value>
  <description>When calculating remaining space, only use this percentage of
the real available space
  </description>
</property>

I removed some stuff, but no changes. It falls under 1Gb (I expected to have
3Gb free at least)

Now df -h gives me

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              36G   34G  602M  99% /

I saw in hadoop jira that the implementation of these directives was subject
to discussions. Or is that possible that hbase client overrides the hadoop
configuration ?

Anyway, I'll post a messge in their board.


stack-3 wrote:
> 
> 
> I see above too on occasion on our clusters.  Was going to spend some 
> time on it today.  In particular, see if HADOOP-3831 helps.
> 
> 1GB of RAM as Jon Gray has suggested is not going to cut it in my 
> experience.   With so little memory, would suggest you run datanodes and 
> regionservers on their own machines.
> 
> While I'm giving out advice, smile, be sure to have upped your ulimit so 
> hbase can use > 1024 file descriptors and run with DEBUG enabled so you 
> can see more about why hbase is doing (See FAQ for how to do both).
> 
> Thanks J-A,
> St.Ack
> 
> 

File descriptors limit is set as FAQ said.

I'll fix the HD free space problem, and distribute process and memory
differently on my cluster, following you and Jonathan's advise  It should
help.

Thanks a lot.
Have a nice day
-- 
View this message in context: http://www.nabble.com/Dataloss-HBASE-728-tp20013732p20030496.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message