From user-return-35949-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org Thu May 2 01:01:40 2013 Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3F8971095D for ; Thu, 2 May 2013 01:01:40 +0000 (UTC) Received: (qmail 7137 invoked by uid 500); 2 May 2013 01:01:38 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 7035 invoked by uid 500); 2 May 2013 01:01:38 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 7024 invoked by uid 99); 2 May 2013 01:01:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 May 2013 01:01:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bryanck@gmail.com designates 209.85.220.48 as permitted sender) Received: from [209.85.220.48] (HELO mail-pa0-f48.google.com) (209.85.220.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 May 2013 01:01:30 +0000 Received: by mail-pa0-f48.google.com with SMTP id kp6so48564pab.35 for ; Wed, 01 May 2013 18:01:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:from:content-type:message-id:mime-version:subject:date :references:to:in-reply-to:x-mailer; bh=1wVQ6m6GE7W7p7PLyFr346iRiESegBdJXfpI8TJ9x68=; b=CIvI8hYGKNpkX03Ye0tPuB03vfvjkq1+fDKi80loRUGqIdNkyd75i/3288d2LMLmqd CJ9bWf2vWFMWXIhCd6YoA+dsw+yl+/lYkEoWeICJcXxBeT1Qrgb04Lz6nV3hIDONTnY1 aHARm3l7gWDxjfU1wpMaeChwT18fkCqYZgYYgoXtDGXljx4BnMBeAO/AlIJiOYADxNHQ qEq/lMuetgqd+6LWAKGJqcjKlyxLGHm1Cph6A6PRDmPhDaOfGV1HzID01XZfoPiI4TOp quM6HKFV8kwPlVPBg9hpYw3hxj4J69a3Xco3pai/4qbJknTS/G/i3E5YlbUzdYd01qgL wE3g== X-Received: by 10.66.233.167 with SMTP id tx7mr7325513pac.110.1367456468819; Wed, 01 May 2013 18:01:08 -0700 (PDT) Received: from [172.16.194.131] (c-69-181-100-95.hsd1.ca.comcast.net. [69.181.100.95]) by mx.google.com with ESMTPSA id qi1sm5735930pac.21.2013.05.01.18.01.06 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 01 May 2013 18:01:07 -0700 (PDT) From: Bryan Keller Content-Type: multipart/alternative; boundary="Apple-Mail=_98DD32E7-3CFF-420D-A2F9-D9617440B080" Message-Id: <5F425C64-5065-4800-88BA-BA22B567C939@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Poor HBase map-reduce scan performance Date: Wed, 1 May 2013 18:01:04 -0700 References: <992ED057-7C3F-4759-B1F4-5F166D549F18@gmail.com> <1367384494.5120.YahooMailNeo@web140601.mail.bf1.yahoo.com> <20E5E82A-4A5F-4696-864B-E30C3B7B97CB@gmail.com> <1367389307.94182.YahooMailNeo@web140602.mail.bf1.yahoo.com> To: "user@hbase.apache.org" In-Reply-To: X-Mailer: Apple Mail (2.1503) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_98DD32E7-3CFF-420D-A2F9-D9617440B080 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 I tried running my test with 0.94.4, unfortunately performance was about = the same. I'm planning on profiling the regionserver and trying some = other things tonight and tomorrow and will report back. On May 1, 2013, at 8:00 AM, Bryan Keller wrote: > Yes I would like to try this, if you can point me to the pom.xml patch = that would save me some time. >=20 > On Tuesday, April 30, 2013, lars hofhansl wrote: > If you can, try 0.94.4+; it should significantly reduce the amount of = bytes copied around in RAM during scanning, especially if you have wide = rows and/or large key portions. That in turns makes scans scale better = across cores, since RAM is shared resource between cores (much like = disk). >=20 >=20 > It's not hard to build the latest HBase against Cloudera's version of = Hadoop. I can send along a simple patch to pom.xml to do that. >=20 > -- Lars >=20 >=20 >=20 > ________________________________ > From: Bryan Keller > To: user@hbase.apache.org > Sent: Tuesday, April 30, 2013 11:02 PM > Subject: Re: Poor HBase map-reduce scan performance >=20 >=20 > The table has hashed keys so rows are evenly distributed amongst the = regionservers, and load on each regionserver is pretty much the same. I = also have per-table balancing turned on. I get mostly data local mappers = with only a few rack local (maybe 10 of the 250 mappers). >=20 > Currently the table is a wide table schema, with lists of data = structures stored as columns with column prefixes grouping the data = structures (e.g. 1_name, 1_address, 1_city, 2_name, 2_address, 2_city). = I was thinking of moving those data structures to protobuf which would = cut down on the number of columns. The downside is I can't filter on one = value with that, but it is a tradeoff I would make for performance. I = was also considering restructuring the table into a tall table. >=20 > Something interesting is that my old regionserver machines had five = 15k SCSI drives instead of 2 SSDs, and performance was about the same. = Also, my old network was 1gbit, now it is 10gbit. So neither network nor = disk I/O appear to be the bottleneck. The CPU is rather high for the = regionserver so it seems like the best candidate to investigate. I will = try profiling it tomorrow and will report back. I may revisit = compression on vs off since that is adding load to the CPU. >=20 > I'll also come up with a sample program that generates data similar to = my table. >=20 >=20 > On Apr 30, 2013, at 10:01 PM, lars hofhansl wrote: >=20 > > Your average row is 35k so scanner caching would not make a huge = difference, although I would have expected some improvements by setting = it to 10 or 50 since you have a wide 10ge pipe. > > > > I assume your table is split sufficiently to touch all = RegionServer... Do you see the same load/IO on all region servers? > > > > A bunch of scan improvements went into HBase since 0.94.2. > > I blogged about some of these changes here: = http://hadoop-hbase.blogspot.com/2012/12/hbase-profiling.html > > > > In your case - since you have many columns, each of which carry the = rowkey - you might benefit a lot from HBASE-7279. > > > > In the end HBase *is* slower than straight HDFS for full scans. How = could it not be? > > So I would start by looking at HDFS first. Make sure Nagle's is = disbaled in both HBase and HDFS. > > > > And lastly SSDs are somewhat new territory for HBase. Maybe Andy = Purtell is listening, I think he did some tests with HBase on SSDs. > > With rotating media you typically see an improvement with = compression. With SSDs the added CPU needed for decompression might = outweigh the benefits. > > > > At the risk of starting a larger discussion here, I would posit that = HBase's LSM based design, which trades random IO with sequential IO, = might be a bit more questionable on SSDs. > > > > If you can, it would be nice to run a profiler against one of the = RegionServers (or maybe do it with the single RS setup) and see where it = is bottlenecked. > > (And if you send me a sample program to generate some data - not = 700g, though :) - I'll try to do a bit of profiling during the next days = as my day job permits, but I do not have any machines with SSDs). > > > > -- Lars > > > > > > > > > > ________________________________ > > From: Bryan Keller > > To: user@hbase.apache.org > > Sent: Tuesday, April 30, 2013 9:31 PM > > Subject: Re: Poor HBase map-reduce scan performance > > > > > > Yes, I have tried various settings for setCaching() and I have = setCacheBlocks(false) > > > > On Apr 30, 2013, at 9:17 PM, Ted Yu wrote: > > > >> =46rom http://hbase.apache.org/book.html#mapreduce.example : > >> > >> scan.setCaching(500); // 1 is the default in Scan, which = will > >> be bad for MapReduce jobs > >> scan.setCacheBlocks(false); // don't set to true for MR jobs > >> > >> I guess you have used the above setting. > >> > >> 0.94.x releases are compatible. Have you considered upgrading to, = say > >> 0.94.7 which was recently released ? > >> > >> Cheers > >> > >> On Tue, Apr 30, 2013 at 9:01 PM, Bryan Keller