From user-return-19437-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org Tue May 24 22:44:44 2011 Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6FE4B6A04 for ; Tue, 24 May 2011 22:44:44 +0000 (UTC) Received: (qmail 1578 invoked by uid 500); 24 May 2011 22:44:43 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 1552 invoked by uid 500); 24 May 2011 22:44:43 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 1544 invoked by uid 99); 24 May 2011 22:44:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:44:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ddlatham@gmail.com designates 74.125.83.41 as permitted sender) Received: from [74.125.83.41] (HELO mail-gw0-f41.google.com) (74.125.83.41) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:44:36 +0000 Received: by gwaa12 with SMTP id a12so3627222gwa.14 for ; Tue, 24 May 2011 15:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:from :date:x-google-sender-auth:message-id:subject:to:content-type; bh=y956Lpcm8idZ6oN3I5ZhnSNeRZ6ufUdjU4G7jKlCMes=; b=VvDOPqM7vcN1HQrB0Jyxg9uMDtFG48AXHo9mojoG+GBlhtkuRmugnV2KQIDNzeBmV5 9pi8T9tkGVq7ZhC+gt9DpAUQtmMINzf8WaYg681kGSJOtMTt5BShiOjCq9995gM2+u1A kdB9M9GgzJOFmwB/u0m+Vayn6Lzwe9xykrgd4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; b=LoUBkcv70+DicKsjxzYJvzMYWAhaId/tLGWmxUIv3C5fecxDb/jwvcF+74HenRoGHw u0g1tjrmIXkf+9/IfKmbujHa5jOyxgkqPKnmQzrmrwaUtxQWqvFJ+sPQyTwr0jJUYOVg qmuICpa++6JxdZ3srHYdATF0OG8srQVnJTzrg= Received: by 10.236.192.234 with SMTP id i70mr5075955yhn.526.1306277055114; Tue, 24 May 2011 15:44:15 -0700 (PDT) MIME-Version: 1.0 Sender: ddlatham@gmail.com Received: by 10.236.153.194 with HTTP; Tue, 24 May 2011 15:43:55 -0700 (PDT) In-Reply-To: References: From: Dave Latham Date: Tue, 24 May 2011 15:43:55 -0700 X-Google-Sender-Auth: 5sBskjjOmcb6xLRtOFWi29dX4RY Message-ID: Subject: Re: 0.90.3 To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=20cf30564319d92f5704a40d5356 X-Virus-Checked: Checked by ClamAV on apache.org --20cf30564319d92f5704a40d5356 Content-Type: text/plain; charset=ISO-8859-1 Are you using the graceful_stop script? In 0.90.3 the bin/graceful_stop.sh script was updated to disable the master's balancer. However, it doesn't seem that anything re-enables it, so if you're using it you need to re-enable it on your own. See the book for more details: http://hbase.apache.org/book.html#decommission Dave On Tue, May 24, 2011 at 3:33 PM, Jack Levin wrote: > just put new hbase version on our test cluster. and been testing it... > so far if I shutdown an RS, master does not reassign its regions, and > we remain inconsistent forerver, likewise when new RS is up, it does > not get regions assigned to it, this is the master log: > > > 2011-05-24 15:30:57,724 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > master:60000-0x1302094818900a4-0x1302094818900a4 Received ZooKeeper > Event, type=NodeDeleted, state=SyncConnected, > path=/hbase/rs/img645.prod.imageshack.com,60020,1306276075768 > 2011-05-24 15:30:57,724 INFO > org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer > ephemeral node deleted, processing expiration > [img645.prod.imageshack.com,60020,1306276075768] > 2011-05-24 15:30:57,724 INFO > org.apache.hadoop.hbase.zookeeper.RegionServerTracker: No HServerInfo > found for img645.prod.imageshack.com,60020,1306276075768 > 2011-05-24 15:30:57,726 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > master:60000-0x1302094818900a4-0x1302094818900a4 Received ZooKeeper > Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs > 2011-05-24 15:31:03,330 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > master:60000-0x1302094818900a4-0x1302094818900a4 Received ZooKeeper > Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs > 2011-05-24 15:31:03,338 DEBUG > org.apache.hadoop.hbase.zookeeper.ZKUtil: > master:60000-0x1302094818900a4-0x1302094818900a4 Retrieved 32 byte(s) > of data from znode > /hbase/rs/img645.prod.imageshack.com,60020,1306276262774 and set > watcher; img645.prod.imageshack.com:60020 > 2011-05-24 15:31:03,350 INFO > org.apache.hadoop.hbase.master.ServerManager: Server start rejected; > we already have img645.imageshack.us:60020 registered; > existingServer=serverName=img645.imageshack.us,60020,1306276075768, > load=(requests=0, regions=0, usedHeap=40, maxHeap=3995), > newServer=serverName=img645.imageshack.us,60020,1306276262774, > load=(requests=0, regions=0, usedHeap=23, maxHeap=3995) > 2011-05-24 15:31:03,350 INFO > org.apache.hadoop.hbase.master.ServerManager: Triggering server > recovery; existingServer img645.imageshack.us,60020,1306276075768 > looks stale > 2011-05-24 15:31:03,353 DEBUG > org.apache.hadoop.hbase.master.ServerManager: > Added=img645.imageshack.us,60020,1306276075768 to dead servers, > submitted shutdown handler to be executed, root=false, meta=false > 2011-05-24 15:31:03,353 INFO > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: > Splitting logs for img645.imageshack.us,60020,1306276075768 > 2011-05-24 15:31:04,348 INFO > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: > Reassigning 0 region(s) that img645.imageshack.us,60020,1306276075768 > was carrying (skipping 0 regions(s) that are already in transition) > 2011-05-24 15:31:04,348 INFO > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished > processing of shutdown of img645.imageshack.us,60020,1306276075768 > 2011-05-24 15:31:06,333 DEBUG > org.apache.hadoop.hbase.master.ServerManager: Server > img645.imageshack.us,60020,1306276262774 came back up, removed it from > the dead servers list > 2011-05-24 15:31:06,333 INFO > org.apache.hadoop.hbase.master.ServerManager: Registering > server=img645.imageshack.us,60020,1306276262774, regionCount=0, > userLoad=false > 2011-05-24 15:31:49,890 DEBUG > org.apache.hadoop.hbase.zookeeper.ZKUtil: hconnection opening > connection to ZooKeeper with ensemble (img648:2181) > 2011-05-24 15:31:49,890 INFO org.apache.zookeeper.ZooKeeper: > Initiating client connection, connectString=img648:2181 > sessionTimeout=180000 watcher=hconnection > 2011-05-24 15:31:49,891 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server img648/38.99.76.205:2181 > 2011-05-24 15:31:49,892 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to img648/38.99.76.205:2181, initiating session > 2011-05-24 15:31:49,893 INFO org.apache.zookeeper.ClientCnxn: Session > establishment complete on server img648/38.99.76.205:2181, sessionid = > 0x13024216e690004, negotiated timeout = 180000 > 2011-05-24 15:31:49,894 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: hconnection > Received ZooKeeper Event, type=None, state=SyncConnected, path=null > 2011-05-24 15:31:49,895 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > hconnection-0x13024216e690004 connected > 2011-05-24 15:31:49,896 DEBUG > org.apache.hadoop.hbase.zookeeper.ZKUtil: > hconnection-0x13024216e690004 Set watcher on existing znode > /hbase/master > 2011-05-24 15:31:49,896 DEBUG > org.apache.hadoop.hbase.zookeeper.ZKUtil: > hconnection-0x13024216e690004 Retrieved 32 byte(s) of data from znode > /hbase/master and set watcher; img648.prod.imageshack.com:60000 > 2011-05-24 15:31:49,897 DEBUG > org.apache.hadoop.hbase.zookeeper.ZKUtil: > hconnection-0x13024216e690004 Set watcher on existing znode > /hbase/root-region-server > 2011-05-24 15:31:49,897 DEBUG > org.apache.hadoop.hbase.zookeeper.ZKUtil: > hconnection-0x13024216e690004 Retrieved 26 byte(s) of data from znode > /hbase/root-region-server and set watcher; img731.imageshack.us:60020 > 2011-05-24 15:31:49,900 DEBUG > org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting > at row= for max=2147483647 rows > 2011-05-24 15:31:49,900 DEBUG > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Lookedup root region location, > > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@26f50154 > ; > hsa=img731.imageshack.us:60020 > 2011-05-24 15:31:49,913 DEBUG > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Cached location for .META.,,1.1028785192 is img654.imageshack.us:60020 > 2011-05-24 15:31:50,061 INFO > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Closed zookeeper sessionid=0x13024216e690004 > 2011-05-24 15:31:50,063 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x13024216e690004 closed > 2011-05-24 15:31:50,063 INFO org.apache.zookeeper.ClientCnxn: > EventThread shut down > > Please help :) > > -Jack > --20cf30564319d92f5704a40d5356--