hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: major hdfs issues
Date Thu, 31 Mar 2011 06:21:18 GMT
I'm sorry we came late to the party. Just yesterday I noticed the latest Amazon Linux AMI beta
had this issue, with nproc set way too low. The HDFS balancer was exceeding the native threads
limit and bailing out and this left me wondering why. Would have affected our datanodes too
at sufficient load.

Added item #20 to http://wiki.apache.org/hadoop/Hbase/Troubleshooting

Sorry I didn't get to this sooner, Jack.

   - Andy

--- On Wed, 3/30/11, Jack Levin <magnito@gmail.com> wrote:

> From: Jack Levin <magnito@gmail.com>
> Subject: Re: major hdfs issues
> To: user@hbase.apache.org
> Cc: "Suraj Varma" <svarma.ng@gmail.com>
> Date: Wednesday, March 30, 2011, 7:31 PM
> Thanks to everyone chiming in to help me fix this issue... It has now
> been resolved,  JD and I spend some time looking at thread limits and
> apparently, our userid 'hadoop' had nproc limit (default) set to 1024,
> this of course caused the issue of running out of threads every time
> we were under load, (like compaction, or just high number of queries,
> or RS restart), now, this was addressed in /etc/security/limits.conf,
> where we set "hadoop soft/hard nproc 32000".  Please note that this is
> not same as ulimit -n, and neither it is xcievers, nor its handlers,
> or anything like that.  The user "root" does not run into this
> problem, but anyone installing stock HADOOP/HDFS from cloudera, likely
> would be running datanodes as user hadoop, and will hit that problem
> unless the above settings are adjusted.


View raw message