hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: HBase 0.92rc3 rest performance
Date Wed, 18 Jan 2012 22:24:36 GMT
Thanks Ben.

See https://issues.apache.org/jira/browse/HBASE-5228

Best regards,


  - Andy


Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)


>________________________________
> From: Ben West <bwsithspawn00@yahoo.com>
>To: "user@hbase.apache.org" <user@hbase.apache.org>; Andrew Purtell <apurtell@apache.org>

>Sent: Wednesday, January 18, 2012 7:18 AM
>Subject: Re: HBase 0.92rc3 rest performance
> 
>Thanks for the quick responses!
>
>No higher CPU or memory usage as far as I can tell. No WARNs. We're using the default
Jersey (1.4) and Jetty (6.1.26).
>
>Yes, you only need to start up some number of concurrent GETs. Here is my test script,
if that helps. It is quite simple (in ksh):
>
>maxId=274894038
>for i in {1..1000}
>do
>                # get a random number
>                hex=`dd if=/dev/urandom bs=1 count=8 2>/dev/null |
>                        od -tx1 | head -1 | cut -d' ' -f2- |
>                        tr -d ' ' | tr '[a-f]' '[A-F]'`
>                # convert from hexadecimal to decimal:
>                dec=`echo "ibase=16; $hex" | bc`
>                # echo >&2 "DEBUG: hex=<$hex>; dec=<$dec>"
>                dec=$(( dec % maxId))
>                #echo "$dec"   
>                start=$(date +%s%N)
>                curl -silent http://server.com:8080/table/$dec > /dev/null
>                elapsed=$(($(date +%s%N) - $start))
>                echo $elapsed
>done
>
>I have another script like
>
>for i in {1..100}
>do
>   ksh myScript >> logfile &
>done
>
>
>I will try jstack to see if I can find anything, thanks for the suggestion.
>
>
>----- Original Message -----
>From: Andrew Purtell <apurtell@apache.org>
>To: "user@hbase.apache.org" <user@hbase.apache.org>
>Cc: 
>Sent: Tuesday, January 17, 2012 5:48 PM
>Subject: Re: HBase 0.92rc3 rest performance
>
>jstacks could help point out if the REST server has some internal lock or monitor contention.
>
>Internal to the REST server is just the HBase Java client. REST uses a HTablePool to manage
a small pool of HTable instances that interact with the cluster according to the request at
hand. Client changes could show up in REST but it would be odd (and possibly a REST bug) to
see them only there.
>
>Other things to check:
>
>  - What version of Jetty?
>
>  - What version of Jersey?
>
>  - Any WARNs in logs from the REST server?
>
>Reproducing this would be as simple as starting up 50 or so concurrent fetches?
>
>Best regards,
>
>
>      - Andy
>
>Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>----- Original Message -----
>> From: Jean-Daniel Cryans <jdcryans@apache.org>
>> To: user@hbase.apache.org
>> Cc: 
>> Sent: Tuesday, January 17, 2012 1:45 PM
>> Subject: Re: HBase 0.92rc3 rest performance
>> 
>>T his seems to single out the REST server since thrift and native
>> clients stayed the same.
>> 
>> Can you provide us your test so we can do testing on our side too?
>> 
>> Maybe doing a few jstacks on the REST server could point out the
>> obvious bottlenecks.
>> 
>> J-D
>> 
>> On Tue, Jan 17, 2012 at 11:25 AM, Ben West <bwsithspawn00@yahoo.com> 
>> wrote:
>>>  Hi all
>>> 
>>>  We're trying out .92rc3 instead of .90.4, and for the most part 
>> everything seems fine. But we have a simple test of REST performance which is 
>> basically a large number of cURL jobs getting random rows, and this test is 
>> running *a lot* slower under .92.
>>> 
>>>  When we run just a single client doing REST GETs, the performance is fine.

>> But once I have dozens or hundreds of clients, performance is ~20x worse than 
>> under .90.4 (response time is 7-800ms instead of 40-50ms).
>>> 
>>>  YCSB has pretty much the same performance under both versions, as do other

>> internal tools measuring Thrift and native performance, so I don't feel like 
>> this is a problem with HBase coresetup (although it could be). I don't see 
>> anything suspicious in any logs, IO and CPU utilization are both low. Has anyone

>> run into this or have thoughts on how to troubleshoot?
>>> 
>>>  Thanks!
>>>  Ben
>>
>
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message