hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: Struggling with Region Servers Running out of Memory
Date Thu, 01 Nov 2012 19:25:21 GMT
Good to know.  Its nice they finally got that in.  We aren't on u36 right now in production
but I'm 
going to push on getting us there.


On 11/1/2012 11:07 AM, Jeremy Carroll wrote:
> Java 6 update 34 can rotate GC Logs. -XX:+UseGCLogFileRotation
> http://stackoverflow.com/questions/3822097/rolling-garbage-collector-logs-in-java
> As for profiling memory dumps, jprofiler7, yourrkit, etc.. YMMV.
> On Thu, Nov 1, 2012 at 10:01 AM, Jeff Whiting <jeffw@qualtrics.com> wrote:
>> We don't have GC logging enabled (we did but gc.log would begin filling up
>> the hdd and there was no way to clear it out without restarting the region
>> server).  Anyway to en gc.log and keep it to a reasonable size?
>> I have two separate jmap dumps of the a region server before it dies.  I
>> haven't really looked into those yet.  I'll try to do that today.  I've
>> typically used eclipse memory analyzer tool or netbeans.  Is there a
>> profiler you'd recommend?
>> ~Jeff
>> On 10/30/2012 11:39 PM, Stack wrote:
>>> On Mon, Oct 29, 2012 at 3:55 PM, Jeff Whiting <jeffw@qualtrics.com>
>>> wrote:
>>>> However what we are seeing is that our memory usage goes up slowly until
>>>> the
>>>> region server starts sputtering due to gc collection issues and it will
>>>> eventually get timed out by zookeeper and be killed.
>>>>   Hey Jeff.  You have GC logging enabled?  Might not tell you more than
>>> you already know, that something is retaining more and more objects
>>> over time.   You have a dumped heap?  What have you used to poke at
>>> it?  You generally want to find the objects that have the deepest size
>>> (Not all profilers let you do this though).  This is usually enough to
>>> give you a clue.
>>> Anything particular about the character of your load?  Ram asks if any
>>> big cells in the mix?
>>> St.Ack
>>>   At this point I feel somewhat lost as to how to debug the problem. I'm
>>>> not
>>>> sure what to do next to figure out what is going on.  Any suggestions as
>>>> to
>>>> what to look for or debug where the memory is being used? I can generate
>>>> heap dumps via jmap (although it effectively kills the region server)
>>>> but I
>>>> don't really know what to look for to see where the memory is going. I
>>>> also
>>>> have jmx setup on each region server and can connect to it that way.
>>>> Thanks,
>>>> ~Jeff
>>>> --
>>>> Jeff Whiting
>>>> Qualtrics Senior Software Engineer
>>>> jeffw@qualtrics.com
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com

Jeff Whiting
Qualtrics Senior Software Engineer

View raw message