hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: one of our datanodes stops working after few hours
Date Mon, 02 May 2011 20:00:39 GMT
On Mon, May 2, 2011 at 12:56 PM, Jack Levin <magnito@gmail.com> wrote:

> Tried removing yourkit and run on javasun, same thing.  We have some
> threads blocked, does anyone know what they block on?
>

Which threads are blocked? Can you get some jstacks without yourkit?

-Todd


>
> -Jack
>
> On Mon, May 2, 2011 at 7:53 AM, Todd Lipcon <todd@cloudera.com> wrote:
> > Hi Jack,
> >
> > Does this happen even if you aren't running Yourkit on the DN?
> >
> > Can you try using a Sun JDK instead of OpenJDK?
> >
> > -Todd
> >
> > On Sun, May 1, 2011 at 7:34 PM, Jack Levin <magnito@gmail.com> wrote:
> >
> >> Version:         0.20.2+320 hdfs
> >> .89 HBASE
> >>
> >> ulimit is 32k
> >> xcievers is 5k
> >>
> >> Note from the jstack, I am not exceeding xcievers.
> >>
> >> -Jack
> >>
> >> On Sun, May 1, 2011 at 6:19 PM, Michael Segel <
> michael_segel@hotmail.com>
> >> wrote:
> >> >
> >> >
> >> > What's your xceivers set to?
> >> > What's the ulimit -n  set for hdfs/hadoop user... (You didn't say
> which
> >> release/version you were using.)
> >> >
> >> >> Date: Sun, 1 May 2011 17:47:18 -0700
> >> >> Subject: one of our datanodes stops working after few hours
> >> >> From: magnito@gmail.com
> >> >> To: user@hbase.apache.org
> >> >>
> >> >> I took a jstack (http://pastebin.com/5v6mHg3t).   After few hours,
> its
> >> >> literally staggers to a halt and gets very very slow... Any ideas
> >> >> whats its blocking on?
> >> >> (main issue is that fsreads for RS get really slow when that
> happens).
> >> >>
> >> >> -Jack
> >> >
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message