hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: hbase row locking
Date Sun, 28 Sep 2014 21:01:48 GMT
> When HDFS gets tiered storage, we can revive this and put HBase's WAL on SSD storage.

I think the archival storage work just committed (and ported into branch-2?) might be sufficient
for a pilot of HBASE-6572. A few important changes were filed late, like APIs for apps for
setting policy and persistence for changes in the edit log, so we should survey what is in
there exactly. 

Related: HDFS-5682



On Sep 28, 2014, at 1:32 PM, lars hofhansl <larsh@apache.org> wrote:

>> Are anyone aware of any company who does not
> use the hdfs default policy and flush every WAL sync.
> 
> 
> It's a trade-off. You'll only lose data when and the wrong three machines die around
the same time (you'd have an outage that any block that exists only on these three boxes).
Not also that time of the data not being on disk is bounded, eventually the OS the flush dirty
pages, Linux does it every 15s by default.
> 
> So you'd have all machines die before a single of them manages to flush the dirty pages
to disk. Of course that can happen, for example during a data center power outage.
> 
> 
> A while ago I added HDFS-744 to HDFS, but never finished the parts in HBase as nobody
(including myself in the end) was interested in it. Reminds to maybe take this up again in
HBase 2.0 since now we support fewer versions of Hadoop.
> 
> 
> When HDFS gets tiered storage, we can revive this and put HBase's WAL on SSD storage.
> 
> 
> -- Lars
> 
> 
> 
> ----- Original Message -----
> From: abhishek1015 <abhishek1015@gmail.com>
> To: user@hbase.apache.org
> Cc: 
> Sent: Sunday, September 28, 2014 9:13 AM
> Subject: Re: hbase row locking
> 
> Sorry for confusion. I meant that I am getting 6000 ops/sec throughput
> overall using 4 machine. That is 1500 ops/sec/regionserver on average.
> 
> I checked the ping response time between machines. It is approximately .09
> ms.
> 
> Assuming that WAL sync thread tries to sync with two other hdfs node
> sequentially, the row lock will be held for at least 0.18 ms, which will
> still give a very high throughput per regionserver even if only one thread
> is working and all other threads are blocked because of locking. 
> 
> It appears that bottleneck is then the hdfs disk flush.  And, consequently,
> above mentioned schema are equivalent w.r.t. performance.
> 
> However, I have a question regarding the default hdfs policy of not flushing
> every WAL sync. Are not people in industry afraid of data loss however small
> probability of this happening. Are anyone aware of any company who does not
> use the hdfs default policy and flush every WAL sync.
> 
> Thanks
> Abhishek
> 
> 
> 
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-row-locking-tp4064432p4064458.html
> 
> 
> 
> Sent from the HBase User mailing list archive at Nabble.com.
> 

Mime
View raw message