hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <la...@apache.org>
Subject Re: Very High CPU usage in RegionServer
Date Fri, 08 Jul 2016 15:08:27 GMT
Hi Sandeep,

Can you take a jstack of the RegionServer process when this happens the next time?(DebugDump
on the RegionServer UI page, or log into the box and run jstack yourself)
Otherwise we'll be guessing :)

-- Lars

      From: Sandeep Reddy <sandeepvreddy@outlook.com>
 To: "user@hbase.apache.org" <user@hbase.apache.org> 
 Sent: Thursday, July 7, 2016 11:06 PM
 Subject: Very High CPU usage in RegionServer

We are observing very high CPU load(400 to 600%) in one of our RegionServer to the point where
the machine is becoming unresponsive.

At this point the whole cluster of 20+ RegionServers becoming unresponsive.

Before cluster becomes unresponsive we observed following symptoms:

  *  Huge bandwidth spike
  *  CPU spikes vertically form normal load to very high usage only in one RegionServer
  *  Few times even though machine is unresponsive, it sending heartbeats to master
  *  There is no spike in number of requests to HBase
  *  We are observed this pattern at least twice is last week
  *  We don't have any co-processors in any of the region servers

What could be the possible reasons for this kind of behaviour.

We are using hbase-0.98.7, hadoop-2.5.1 versions.
Its production cluster so upgrading to latest version will not be possible right away.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message