hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Tuttle <...@mentacapital.com>
Subject RE: master consumes large amount of CPU for days
Date Thu, 26 Mar 2015 23:22:34 GMT
We have no regions in transition.

As far as who's requesting the table descriptors:  We are calling HTableIterface.getTableDescriptor
frequently.  We are going to cache the result to avoid the unnecessary network calls.

Thanks for the hint!

From: lars hofhansl [mailto:larsh@apache.org]
Sent: Thursday, March 26, 2015 3:43 PM
To: user@hbase.apache.org
Cc: Development
Subject: Re: master consumes large amount of CPU for days

Hi Ted,

Yes, it is safe to bounce the HMaster without taking the region servers down.

Are there any regions in transition (would be shown on the master's page)?

All of the threads involved with
- org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path) @bci=14,
line=309 (Compiled frame)
- org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List) @bci=50, line=1932
(Compiled frame)

are a bit suspect. Who's requesting all the table descriptors from the master?
-- Lars

From: Ted Tuttle <ted@mentacapital.com<mailto:ted@mentacapital.com>>
To: "user@hbase.apache.org<mailto:user@hbase.apache.org>" <user@hbase.apache.org<mailto:user@hbase.apache.org>>
Cc: Development <Development@mentacapital.com<mailto:Development@mentacapital.com>>
Sent: Thursday, March 26, 2015 12:18 PM
Subject: master consumes large amount of CPU for days


Our master process started consuming a large amount of CPU (75% of box) several days back
and hasn't stopped.  I have 2 questions:

                1) what is it doing (stack dump and log below)
                2) is it safe to restart the master without taking the whole cluster down?

Master stack dump:


Master log from last 15 mins or so:



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message