hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tsuna <tsuna...@gmail.com>
Subject Re: asynch api mailing list?
Date Mon, 28 Feb 2011 02:24:53 GMT
On Sun, Feb 27, 2011 at 1:15 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> Sorry to hijack the thread but I noticed thrift 5.0 generates both
> synchronous and asyn stubs. Why would someone pick async-hbase vs
> thrift-async?

If you don't wanna use Thrift.  Or if you want a real HBase client
that supports the HBase API, and not only whatever API is exposed
through Thrift (IIRC, you can't do everything through the Thrift API,
although I'm sure this could easily be fixed).

If you use the Thrift gateway, you still end up using the normal HBase
client on the Thrift gateway anyway.  The reason I wrote asynchase is
so I could have a thread-safe and highly scalable client.  asynchbase
uses far fewer locks and produces less garbage for the GC to collect,
which makes it more suitable for low-latency high-throughput servers
that want to use HBase as backend.

Here's the result of a micro-benchmark I ran in late November at
StumbleUpon.  It shows that asynchbase is 40% faster on those super
straightforward microbenchmarks:

Simple edit
HBase:       real=1005.1; user=539.5; sys=36.0; csw=230.8; icsw=55.7;
syscalls=7915; futex=754; rss=38488;
HBase Async: real=606.7; user=428.0; sys=40.5; csw=165.6; icsw=51.8;
syscalls=8074; futex=546; rss=36825;

Simple get
HBase:       real=1046.0; user=532.0; sys=38.0; csw=232.7; icsw=47.2;
syscalls=7920; futex=700; rss=38511;
HBase Async: real=614.0; user=429.0; sys=35.5; csw=166.2; icsw=51.5;
syscalls=8067; futex=570; rss=36730;

Multiple edits
HBase:       real=1193.5; user=587.5; sys=42.5; csw=731.5; icsw=60.8;
syscalls=10190; futex=2229; rss=39107;
HBase Async: real=626.3; user=439.5; sys=40.5; csw=181.7; icsw=49.9;
syscalls=8187; futex=580; rss=36811;

The tests are done on a clean, truncated table.  The first test simply
creates a Put and stores it.  The second reads it back.  The third
creates 100 Put, each for a different row key, but they all fall in
the same region (since the table was truncated, it has only a single
region).  The timings are end-to-end, including JVM startup time.
Nothing fancy going on, just a trivial main function of a few lines,
no other threads, no JVM tuning parameters.  Not a single GC cycle
happens during those short tests.  Values are averages over 40 runs.

As you can see, asynchbase spends only slightly less user+system time
than HBase's traditional client, but the wall clock time (real) is
significantly higher when using HTable.  I think this is entirely
attributed to context switches (csw = number of context switches, icsw
= number of involuntary context switches).  The number of calls to
futex is consistently higher with HTable, which indicates higher lock
contention (and causes more context switches).  Other fields are:
syscalls = number of system calls executed by the process, rss =
actual memory used ("resident set size") in KB before exiting main.

Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

View raw message