hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tsuna <tsuna...@gmail.com>
Subject Re: HTable client help
Date Sun, 27 Feb 2011 03:39:01 GMT
On Sat, Feb 26, 2011 at 10:28 AM, benchmark <zslist@gmail.com> wrote:
> On Fri, Feb 25, 2011 at 3:09 PM, benchmark <zslist@gmail.com> wrote:
>> We are using 0.20x hbase. We are seeing some big delays on the client's
>> read. The server side stats shows about 2-3 ms, but on the client side we
>> are seeing about 20ms. The network delay is negligible. We have a about 250
>> threads per JVM each having its own HTable client handle. Reducing the
>> number of client threads to 32 helps a tiny bit, but nothing significant.
>
> on the server side, the average RPC time is about 2-3 ms, the cache hit rate
> around 60%.
>
> on the client side, around 50% returns within 0-1 ms, around 30% returns in
> 16-32ms, and the remaining 20% in 64-128ms range.
>
> The client is doing random reads (HTable.get(Get)), the average data read is
> about 1k bytes.

How many hardware threads does your server have?  If you have 250
threads and just 4-8 hardware threads, and all the cores are actively
asking for CPU time, you could be seeing extra latency due do context
switching overhead and time spent waiting in the run queue.  If you
run "vmstat 1", do you see a lot of threads waiting in the run queue?
(first column, "r").  What about the number of context switches?
(column "cs")

If you don't do any writes, you could share the clients across
threads, I think HTable is only thread-unsafe when you use mutative
operations.

Alternatively, you could look at asynchbase
(https://github.com/stumbleupon/asynchbase), which is thread-safe and
uses a fixed number of threads (2 * N where N is the number of
hardware threads).  That's how I solved my scalability problems in
OpenTSDB.

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

Mime
View raw message