hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: random read/write performance
Date Thu, 08 Oct 2009 05:14:31 GMT
On Tue, Oct 6, 2009 at 10:52 PM, Adam Silberstein <silberst@yahoo-inc.com>wrote:

> Hey,
> Thanks for all the info...
> First, a few details to clarify my use case:
> -I have 6 region servers.
> -I loaded a total of 120GB in 1K records into my table, so 20GB per
> server.  I'm not sure how many regions that has created.

You could run the rowcounter mapreduce job to see:

./bin/hadoop jar hbase.jar rowcounter

That'll dump usage.  You pass a tablename, column and a tmpdir IIRC.

> -My reported numbers are on workloads taking place once the 120GB is in
> place, rather than while loading the 120GB.
> -I've run with combinations of 50,100,200 clients hitting the REST
> server.  So that's e.g. 200 clients across all region servers, not per
> region server.  Each client just repeatedly a) generates a random record
> known to exist, and b) reads or updates it.

Our client can be a bottleneck.  At its core is hadoop RPC with its single
connection to each server over which request/response are multiplexed.  As
per J-D's suggestion, you might be able to get more throughput by upping the
REST server count (or, should be non-issue when you move to java api).

REST server base64's everything too so this'll add a bit of friction.

> -I'm interested in both throughput and latency.  First, at medium
> throughputs (i.e. not at maximum capacity) what are average read/write
> latencies.  And then, what is the maximum possible throughput, even as
> that causes latencies to be very high.  What is the throughput wall?
> Plotting throughput vs. latency for different target throughputs reveals
> both of these.

Good stuff.  Let us know how else we can help out.

When I have 50 clients across 6 region server, this is fairly close to
> your read throughput experiment with 8 clients on 1 region server.  Your
> 2.4 k/sec throughput is obviously a lot better than what I'm seeing at
> 300/sec.  Since you had 10GB loaded, is it reasonable to assume that
> ~50% of the reads were from memory?

I think I had 3G per RS with 40% given over to cache. I had 1RS so not too
much coming from hbase cache (OS cache probably played a big factor).

>  In my case, with 20GB loaded and
> 6GB heapspace, I assume ~30% was served from memory.   I haven't run
> enough tests on different size tables to estimate the impact of having
> data in memory, though intuitively, in the time it takes to read a
> record from disk, you could read several from memory.  And the more the
> data is disk resident, the more the disk contention.
> Yes.

> Finally, I haven't tried LZO or increasing the logroll multiplier yet,

LZO would be good.  Logroll multiplier is more about writing which you are
doing little of so maybe its ok at default?

> and I'm hoping to move to the java client soon.  As you might recall,
> we're working toward a benchmark for cloud serving stores.  We're
> testing the newest version of our tool now.  Since it's in java, we'll
> be able to use it with HBase.

Tell us more?  You are comparing HBase to others with a tool of your

I'll report back when I find out how much these changes close the
> performance gap, and how much seems inherent when much of the data is
> disk resident.
Thanks Adam.

> -Adam
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> stack
> Sent: Tuesday, October 06, 2009 1:08 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: random read/write performance
> Hey Adam:
> Thanks for checking in.
> I just did some rough loadings on a small (old hardware) cluster using
> less
> memory per regionserver than you.  Its described on this page:
> http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation.  Random
> writing
> 1k records with the PerformanceEvaluation script to a single
> regionserver, I
> can do about 8-10k/writes/second on average using the 0.20.1 release
> candidate 1 with a single client.  Sequential writes are about the same
> speed usually.  Random reads are about 650/second on average with single
> client and about 2.4k/second on average with 8 concurrent clients.
> So it seems like you should be able to do better than
> 300ops/persecond/permachine -- especially if you can do the java api.
> This single regionserver was carrying about 50 regions.  Thats about
> 10GB.
> How many regions loaded in your case?
> If throughput is important to you, lzo should help (as per J-D).
> Turning
> off WAL will also help with write throughput but that might not be what
> you
> want.  Random-read-wise, the best thing you can do is give it RAM (6G
> should
> be good).
> Is that 50-200 clients per regionserver or for the overall cluster?  If
> per
> regionserver, I can try that over here.   I can try with bigger regions
> if
> you'd like -- 1G regions -- to see if that'd help your use case (if you
> enable lzo, this should up your throughput and shrink the number of
> regions
> any one server is hosting).
> St.Ack
> On Tue, Oct 6, 2009 at 8:59 AM, Adam Silberstein
> <silberst@yahoo-inc.com>wrote:
> > Hi,
> >
> > Just wanted to give a quick update on our HBase benchmarking efforts
> at
> > Yahoo.  The basic use case we're looking at is:
> >
> > 1K records
> >
> > 20GB of records per node (and 6GB of memory per node, so data is not
> > memory resident)
> >
> > Workloads that do random reads/writes (e.g. 95% reads, 5% writes).
> >
> > Multiple clients doing the reads/writes (i.e. 50-200)
> >
> > Measure throughput vs. latency, and see how high we can push the
> > throughput.
> >
> > Note that although we want to see where throughput maxes out, the
> > workload is random, rather than scan-oriented.
> >
> >
> >
> > I've been tweaking our HBase installation based on advice I've
> > read/gotten from a few people.  Currently, I'm running 0.20.0, have
> heap
> > size set to 6GB per server, and have iCMS off.  I'm still using the
> > server instead of the java client.  We're about to move our
> benchmarking
> > tool to java, so at that point we can use the java API.  At that
> point,
> > I want to turn off WAL as well.  If anyone has more suggestions for
> this
> > workload (either things to try while still using REST, or things to
> try
> > once I have a java client), please let me know.
> >
> >
> >
> > Given all that, I'm currently seeing maximal throughput of about 300
> > ops/sec/server.  Has anyone with a similar disk-resident and random
> > workload seen drastically different numbers, or guesses for what I can
> > expect with the java client?
> >
> >
> >
> > Thanks!
> >
> > Adam
> >
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message