From user-return-34051-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org Tue Feb 12 09:23:26 2013 Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9A558ED10 for ; Tue, 12 Feb 2013 09:23:26 +0000 (UTC) Received: (qmail 8953 invoked by uid 500); 12 Feb 2013 09:23:24 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 8793 invoked by uid 500); 12 Feb 2013 09:23:24 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 8773 invoked by uid 99); 12 Feb 2013 09:23:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Feb 2013 09:23:23 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of anoopsj@huawei.com designates 119.145.14.65 as permitted sender) Received: from [119.145.14.65] (HELO szxga02-in.huawei.com) (119.145.14.65) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Feb 2013 09:23:14 +0000 Received: from 172.24.2.119 (EHLO szxeml214-edg.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.3.4-GA FastPath queued) with ESMTP id AXA92898; Tue, 12 Feb 2013 17:22:51 +0800 (CST) Received: from SZXEML405-HUB.china.huawei.com (10.82.67.60) by szxeml214-edg.china.huawei.com (172.24.2.29) with Microsoft SMTP Server (TLS) id 14.1.323.7; Tue, 12 Feb 2013 17:22:49 +0800 Received: from SZXEML553-MBS.china.huawei.com ([169.254.6.181]) by szxeml405-hub.china.huawei.com ([::1]) with mapi id 14.01.0323.007; Tue, 12 Feb 2013 17:22:50 +0800 From: Anoop Sam John To: "user@hbase.apache.org" Subject: RE: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space Thread-Topic: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space Thread-Index: AQHOCOR1s51AQ77Xj0eIzH9eN5tiaph1xcgr//+kroCAAIitsw== Date: Tue, 12 Feb 2013 09:22:49 +0000 Message-ID: <0CE69E9126D0344088798A3B7F7F80863AECC596@szxeml553-mbs.china.huawei.com> References: <51F68F1C-6C3A-4B29-A97C-C269387FC69E@gmail.com> <0CE69E9126D0344088798A3B7F7F80863AECC345@szxeml553-mbs.china.huawei.com>,<98A8F664-6AFB-44EB-970D-71ABC8D2E34E@gmail.com> In-Reply-To: <98A8F664-6AFB-44EB-970D-71ABC8D2E34E@gmail.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.96.95] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Virus-Checked: Checked by ClamAV on apache.org >The question is: is it "legal" to change a KV I received from the Internal= Scanner before adding it the Result - i..e returning it from my own Interna= lScanner? You can change as per your need IMO -Anoop- ________________________________________ From: Mesika, Asaf [asaf.mesika@gmail.com] Sent: Tuesday, February 12, 2013 2:43 PM To: user@hbase.apache.org Subject: Re: Custom preCompact RegionObserver crashes entire cluster on OOM= E: Heap Space I am trying to reduce the amount of KeyValue generated during the preCompac= t, but I'm getting some weird behaviors. Let me describe what I am doing in short: We have a counters table, with the following structure: RowKey =3D A combination of field values representing group by key. CF =3D time span aggregate (Hour, Day, Month). Currently we have only for H= our. CQ =3D Round-to-Hour timestamp (long). Value =3D The count We collect raw data, and updates the counters table for the matched group b= y key, hour. We tried using Increment, but discovered its very very slow. Instead we've decided to update the counters upon compaction. We write the = deltas into the same row-key, but a longer column qualifier: . is: Delta or Aggregate. Delta stands for a delta column qualifier we send from our client. in the preCompact, I create an InternalScanner which aggregates the delta c= olumn qualifier values and generates a new key value with Type Aggregate: <= TS> The problem with this implementation that it consumes more memory. Now, I've tried avoiding the creation of the Aggregate type KV, by simply r= e-using the 1st delta column qualifier: simply changing its value in the Ke= yValue. But from some reason, after a couple of minor / major compactions, I see da= ta loss, when I count the values and compare them to the expected. The question is: is it "legal" to change a KV I received from the InternalS= canner before adding it the Result - i..e returning it from my own Internal= Scanner? On Feb 12, 2013, at 8:44 AM, Anoop Sam John wrote: > Asaf, > You have created a wrapper around the original InternalScanner = instance created by the compaction flow? > >> Where do the KV generated during the compaction process queue up before = being written to the disk? Is this buffer configurable? > When I wrote the Region Observer my assumption was the the compaction pro= cess works in Streaming fashion, thus even if I decide to generate a KV per= KV I see, it still shouldn't be a problem memory wise. > > There is no queuing. Your assumption is correct only. It is written to th= e writer as and when. (Just like how memstore flush doing the HFile write) = As Lars said a look at your code can tell if some thing is going wrong. Do= you have blooms being used? > > -Anoop- > ________________________________________ > From: Mesika, Asaf [asaf.mesika@gmail.com] > Sent: Tuesday, February 12, 2013 11:16 AM > To: user@hbase.apache.org > Subject: Custom preCompact RegionObserver crashes entire cluster on OOME:= Heap Space > > Hi, > > I wrote a RegionObserver which does preCompact. > I activated in pre-production, and then entire cluster dropped dead: One = RegionServer after another crashed on OutOfMemoryException: Heap Space. > > My preCompact method generates a KeyValue per each set of Column Qualifie= rs it sees. > When I remove the coprocessor and restart the cluster, cluster remains st= able. > I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific t= able I'm working on) per Region Server. > Running HBase 0.94.3 > > The crash occur when the major compaction fires up, apparently cluster wi= de. > > > My question is this: Where do the KV generated during the compaction proc= ess queue up before being written to the disk? Is this buffer configurable? > When I wrote the Region Observer my assumption was the the compaction pro= cess works in Streaming fashion, thus even if I decide to generate a KV per= KV I see, it still shouldn't be a problem memory wise. > > Of course I'm trying to improve my code so it will generate much less new= KV (by simply altering the existing KVs received from the InternalScanner)= . > > Thank you, > > Asaf=