hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: Struggling with Region Servers Running out of Memory
Date Thu, 01 Nov 2012 17:01:56 GMT
We don't have GC logging enabled (we did but gc.log would begin filling up the hdd and there
was no 
way to clear it out without restarting the region server).  Anyway to en gc.log and keep it
to a 
reasonable size?

I have two separate jmap dumps of the a region server before it dies.  I haven't really looked
into 
those yet.  I'll try to do that today.  I've typically used eclipse memory analyzer tool or

netbeans.  Is there a profiler you'd recommend?

~Jeff

On 10/30/2012 11:39 PM, Stack wrote:
> On Mon, Oct 29, 2012 at 3:55 PM, Jeff Whiting <jeffw@qualtrics.com> wrote:
>> However what we are seeing is that our memory usage goes up slowly until the
>> region server starts sputtering due to gc collection issues and it will
>> eventually get timed out by zookeeper and be killed.
>>
> Hey Jeff.  You have GC logging enabled?  Might not tell you more than
> you already know, that something is retaining more and more objects
> over time.   You have a dumped heap?  What have you used to poke at
> it?  You generally want to find the objects that have the deepest size
> (Not all profilers let you do this though).  This is usually enough to
> give you a clue.
>
> Anything particular about the character of your load?  Ram asks if any
> big cells in the mix?
>
> St.Ack
>
>
>
>> At this point I feel somewhat lost as to how to debug the problem. I'm not
>> sure what to do next to figure out what is going on.  Any suggestions as to
>> what to look for or debug where the memory is being used? I can generate
>> heap dumps via jmap (although it effectively kills the region server) but I
>> don't really know what to look for to see where the memory is going. I also
>> have jmx setup on each region server and can connect to it that way.
>>
>> Thanks,
>> ~Jeff
>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com


Mime
View raw message