hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Strange issue when DataNode goes down
Date Mon, 16 Mar 2015 20:32:16 GMT
Is there a particular reason why you are using HBase 0.98.0? The latest
0.98 release is 0.98.11. There's a known performance issue with 0.98.0
pertaining to RPC that was fixed in later releases, you should move up from
0.98.0. In addition hundreds of improvements and bug fixes have gone into
the ten releases since 0.98.0.

On Mon, Mar 16, 2015 at 6:40 AM, Dejan Menges <dejan.menges@gmail.com>

> Hi All,
> We have a strange issue with HBase performance (overall cluster
> performance) in case one of datanodes in the cluster unexpectedly goes
> down.
> So scenario is like follows:
> - Cluster works fine, it's stable.
> - One DataNode unexpectedly goes down (PSU issue, network issue, anything)
> - Whole HBase cluster goes down (performance becomes so bad that we have to
> restart all RegionServers to get it back to life).
> Most funny and latest issue that happened was that we added new node to the
> cluster (having 8 x 4T SATA disks) and we left just DataNode running on it
> to give it couple of days to get some data. At some point in time, due to
> hardware issue, server rebooted (twice during three hours) in moment when
> it had maybe 5% of data it would have in a couple of days. Nothing else
> beside DataNode was running, and once it went down, it affected literary
> everything, and restarting RegionServers in the end fixed it.
> We are using HBase 0.98.0 with Hadoop 2.4.0

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message