hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: random read/write performance
Date Wed, 07 Oct 2009 11:46:00 GMT
Adam,

You could also try to run 1 REST server per client machine or, if the
clients are on the region servers, 1 REST server per RS. The Thrift
and REST servers are basically wrappers around the Java client API.

J-D

On Wed, Oct 7, 2009 at 1:52 AM, Adam Silberstein <silberst@yahoo-inc.com> wrote:
> Hey,
> Thanks for all the info...
>
> First, a few details to clarify my use case:
> -I have 6 region servers.
> -I loaded a total of 120GB in 1K records into my table, so 20GB per
> server.  I'm not sure how many regions that has created.
> -My reported numbers are on workloads taking place once the 120GB is in
> place, rather than while loading the 120GB.
> -I've run with combinations of 50,100,200 clients hitting the REST
> server.  So that's e.g. 200 clients across all region servers, not per
> region server.  Each client just repeatedly a) generates a random record
> known to exist, and b) reads or updates it.
> -I'm interested in both throughput and latency.  First, at medium
> throughputs (i.e. not at maximum capacity) what are average read/write
> latencies.  And then, what is the maximum possible throughput, even as
> that causes latencies to be very high.  What is the throughput wall?
> Plotting throughput vs. latency for different target throughputs reveals
> both of these.
>
> When I have 50 clients across 6 region server, this is fairly close to
> your read throughput experiment with 8 clients on 1 region server.  Your
> 2.4 k/sec throughput is obviously a lot better than what I'm seeing at
> 300/sec.  Since you had 10GB loaded, is it reasonable to assume that
> ~50% of the reads were from memory?  In my case, with 20GB loaded and
> 6GB heapspace, I assume ~30% was served from memory.   I haven't run
> enough tests on different size tables to estimate the impact of having
> data in memory, though intuitively, in the time it takes to read a
> record from disk, you could read several from memory.  And the more the
> data is disk resident, the more the disk contention.
>
> Finally, I haven't tried LZO or increasing the logroll multiplier yet,
> and I'm hoping to move to the java client soon.  As you might recall,
> we're working toward a benchmark for cloud serving stores.  We're
> testing the newest version of our tool now.  Since it's in java, we'll
> be able to use it with HBase.
>
> I'll report back when I find out how much these changes close the
> performance gap, and how much seems inherent when much of the data is
> disk resident.
>
> -Adam
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> stack
> Sent: Tuesday, October 06, 2009 1:08 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: random read/write performance
>
> Hey Adam:
>
> Thanks for checking in.
>
> I just did some rough loadings on a small (old hardware) cluster using
> less
> memory per regionserver than you.  Its described on this page:
> http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation.  Random
> writing
> 1k records with the PerformanceEvaluation script to a single
> regionserver, I
> can do about 8-10k/writes/second on average using the 0.20.1 release
> candidate 1 with a single client.  Sequential writes are about the same
> speed usually.  Random reads are about 650/second on average with single
> client and about 2.4k/second on average with 8 concurrent clients.
>
> So it seems like you should be able to do better than
> 300ops/persecond/permachine -- especially if you can do the java api.
>
> This single regionserver was carrying about 50 regions.  Thats about
> 10GB.
> How many regions loaded in your case?
>
> If throughput is important to you, lzo should help (as per J-D).
> Turning
> off WAL will also help with write throughput but that might not be what
> you
> want.  Random-read-wise, the best thing you can do is give it RAM (6G
> should
> be good).
>
> Is that 50-200 clients per regionserver or for the overall cluster?  If
> per
> regionserver, I can try that over here.   I can try with bigger regions
> if
> you'd like -- 1G regions -- to see if that'd help your use case (if you
> enable lzo, this should up your throughput and shrink the number of
> regions
> any one server is hosting).
>
> St.Ack
>
>
>
>
>
> On Tue, Oct 6, 2009 at 8:59 AM, Adam Silberstein
> <silberst@yahoo-inc.com>wrote:
>
>> Hi,
>>
>> Just wanted to give a quick update on our HBase benchmarking efforts
> at
>> Yahoo.  The basic use case we're looking at is:
>>
>> 1K records
>>
>> 20GB of records per node (and 6GB of memory per node, so data is not
>> memory resident)
>>
>> Workloads that do random reads/writes (e.g. 95% reads, 5% writes).
>>
>> Multiple clients doing the reads/writes (i.e. 50-200)
>>
>> Measure throughput vs. latency, and see how high we can push the
>> throughput.
>>
>> Note that although we want to see where throughput maxes out, the
>> workload is random, rather than scan-oriented.
>>
>>
>>
>> I've been tweaking our HBase installation based on advice I've
>> read/gotten from a few people.  Currently, I'm running 0.20.0, have
> heap
>> size set to 6GB per server, and have iCMS off.  I'm still using the
> REST
>> server instead of the java client.  We're about to move our
> benchmarking
>> tool to java, so at that point we can use the java API.  At that
> point,
>> I want to turn off WAL as well.  If anyone has more suggestions for
> this
>> workload (either things to try while still using REST, or things to
> try
>> once I have a java client), please let me know.
>>
>>
>>
>> Given all that, I'm currently seeing maximal throughput of about 300
>> ops/sec/server.  Has anyone with a similar disk-resident and random
>> workload seen drastically different numbers, or guesses for what I can
>> expect with the java client?
>>
>>
>>
>> Thanks!
>>
>> Adam
>>
>>
>

Mime
View raw message