From user-return-35911-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org Wed May 1 04:32:16 2013 Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 858ED108E4 for ; Wed, 1 May 2013 04:32:16 +0000 (UTC) Received: (qmail 28727 invoked by uid 500); 1 May 2013 04:32:14 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 28662 invoked by uid 500); 1 May 2013 04:32:14 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 28646 invoked by uid 99); 1 May 2013 04:32:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 May 2013 04:32:13 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bryanck@gmail.com designates 209.85.220.44 as permitted sender) Received: from [209.85.220.44] (HELO mail-pa0-f44.google.com) (209.85.220.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 May 2013 04:32:07 +0000 Received: by mail-pa0-f44.google.com with SMTP id rl6so709661pac.17 for ; Tue, 30 Apr 2013 21:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; bh=cGJkTEwpFXyYBerdmh8CdU3hi31Lx7d41KYfiegen7A=; b=CII6nSKFUFo+P/HbZiD2m7ssafaridXOZk1s5OidANbnNwydzfIyP68jSTDs0ZKCkd mIKrN8cltUOSl6CVTOoi6pEy/RN50p2sjs43756AXIMXwk2BAmm0A8WUvtcyTUbaj6wq qfAepnv+CuV6z6y/HqyKc6C/rCbIvVLZXZtTcPwtRkvDGWXdpJb//S+TDVXULYpQ2TLN EQPcTQUwSlPtHN+/3s9ZzbHQKoJQBqWZ1LHrF/lxlX6r2RvsEcqUAL5Jeh0M6C1T/8Je 6bgvvGNyVeyO0gq7JQr88wgklHxsCPzeOVHTIXXYnKxEMcY9gySimIa/f6VOtoac5cDT e9dA== X-Received: by 10.66.190.104 with SMTP id gp8mr3156913pac.84.1367382707368; Tue, 30 Apr 2013 21:31:47 -0700 (PDT) Received: from [172.16.194.131] (c-69-181-100-95.hsd1.ca.comcast.net. [69.181.100.95]) by mx.google.com with ESMTPSA id dg5sm1385024pbc.29.2013.04.30.21.31.45 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 30 Apr 2013 21:31:46 -0700 (PDT) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Poor HBase map-reduce scan performance From: Bryan Keller In-Reply-To: Date: Tue, 30 Apr 2013 21:31:44 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <992ED057-7C3F-4759-B1F4-5F166D549F18@gmail.com> To: user@hbase.apache.org X-Mailer: Apple Mail (2.1503) X-Virus-Checked: Checked by ClamAV on apache.org Yes, I have tried various settings for setCaching() and I have = setCacheBlocks(false) On Apr 30, 2013, at 9:17 PM, Ted Yu wrote: > =46rom http://hbase.apache.org/book.html#mapreduce.example : >=20 > scan.setCaching(500); // 1 is the default in Scan, which will > be bad for MapReduce jobs > scan.setCacheBlocks(false); // don't set to true for MR jobs >=20 > I guess you have used the above setting. >=20 > 0.94.x releases are compatible. Have you considered upgrading to, say > 0.94.7 which was recently released ? >=20 > Cheers >=20 > On Tue, Apr 30, 2013 at 9:01 PM, Bryan Keller = wrote: >=20 >> I have been attempting to speed up my HBase map-reduce scans for a = while >> now. I have tried just about everything without much luck. I'm = running out >> of ideas and was hoping for some suggestions. This is HBase 0.94.2 = and >> Hadoop 2.0.0 (CDH4.2.1). >>=20 >> The table I'm scanning: >> 20 mil rows >> Hundreds of columns/row >> Column keys can be 30-40 bytes >> Column values are generally not large, 1k would be on the large side >> 250 regions >> Snappy compression >> 8gb region size >> 512mb memstore flush >> 128k block size >> 700gb of data on HDFS >>=20 >> My cluster has 8 datanodes which are also regionservers. Each has 8 = cores >> (16 HT), 64gb RAM, and 2 SSDs. The network is 10gbit. I have a = separate >> machine acting as namenode, HMaster, and zookeeper (single instance). = I >> have disk local reads turned on. >>=20 >> I'm seeing around 5 gbit/sec on average network IO. Each disk is = getting >> 400mb/sec read IO. Theoretically I could get 400mb/sec * 16 =3D = 6.4gb/sec. >>=20 >> Using Hadoop's TestDFSIO tool, I'm seeing around 1.4gb/sec read = speed. Not >> really that great compared to the theoretical I/O. However this is = far >> better than I am seeing with HBase map-reduce scans of my table. >>=20 >> I have a simple no-op map-only job (using TableInputFormat) that = scans the >> table and does nothing with data. This takes 45 minutes. That's about >> 260mb/sec read speed. This is over 5x slower than straight HDFS. >> Basically, with HBase I'm seeing read performance of my 16 SSD = cluster >> performing nearly 35% slower than a single SSD. >>=20 >> Here are some things I have changed to no avail: >> Scan caching values >> HDFS block sizes >> HBase block sizes >> Region file sizes >> Memory settings >> GC settings >> Number of mappers/node >> Compressed vs not compressed >>=20 >> One thing I notice is that the regionserver is using quite a bit of = CPU >> during the map reduce job. When dumping the jstack of the process, it = seems >> like it is usually in some type of memory allocation or decompression >> routine which didn't seem abnormal. >>=20 >> I can't seem to pinpoint the bottleneck. CPU use by the regionserver = is >> high but not maxed out. Disk I/O and network I/O are low, IO wait is = low. >> I'm on the verge of just writing the dataset out to sequence files = once a >> day for scan purposes. Is that what others are doing?