hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: how to check whether zookeeper quorum is working fine or not ?
Date Fri, 15 Jul 2011 17:20:59 GMT
hmmm... I just took a look and I don't see zk dumping it out anywhere.
 Could run a script that puts up 2k -1 clients and see how far you
get? (Maybe someone else has better idea).

St.Ack

On Fri, Jul 15, 2011 at 3:22 AM, praveenesh kumar <praveenesh@gmail.com> wrote:
> I am not able to see the total number of connections as 2k.
> Here is my zookeeper output :
>
> HBase is rooted at /hbase
> Master address: ub13:60000
> Region server holding ROOT: ub5:60020
> Region servers:
>  ub8:60020
>  ub12:60020
>  ub11:60020
>  ub4:60020
>  ub2:60020
>  ub13:60020
>  ub7:60020
>  ub3:60020
>  ub10:60020
>  ub1:60020
>  ub9:60020
>  ub5:60020
> Quorum Server Statistics:
>  ub11:2181
>  Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
>  Clients:
>   /162.192.100.53:51589[1](queued=0,recved=24,sent=24)
>   /162.192.100.53:51591[0](queued=0,recved=1,sent=0)
>
>  Latency min/avg/max: 0/1/22
>  Received: 25
>  Sent: 24
>  Outstanding: 0
>  Zxid: 0x60000026e
>  Mode: leader
>  Node count: 23
>  ub12:2181
>  Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
>  Clients:
>   /162.192.100.53:34213[0](queued=0,recved=1,sent=0)
>
>  Latency min/avg/max: 0/0/0
>  Received: 1
>  Sent: 0
>  Outstanding: 0
>  Zxid: 0x60000026e
>  Mode: follower
>  Node count: 23
>  ub13:2181
>  Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
>  Clients:
>   /162.192.100.41:49999[1](queued=0,recved=141,sent=180)
>   /162.192.100.53:57877[1](queued=0,recved=38,sent=41)
>   /162.192.100.45:48814[1](queued=0,recved=148,sent=185)
>   /162.192.100.53:57884[1](queued=0,recved=38,sent=40)
>   /162.192.100.42:60452[1](queued=0,recved=38,sent=40)
>   /162.192.100.44:57546[1](queued=0,recved=146,sent=186)
>   /162.192.100.50:55070[1](queued=0,recved=33,sent=37)
>   /162.192.100.45:48813[1](queued=0,recved=38,sent=40)
>   /162.192.100.47:51598[1](queued=0,recved=127,sent=161)
>   /162.192.100.53:57878[1](queued=0,recved=127,sent=163)
>   /162.192.100.49:42398[1](queued=0,recved=38,sent=40)
>   /162.192.100.51:52655[1](queued=0,recved=38,sent=40)
>   /162.192.100.50:55068[1](queued=0,recved=137,sent=177)
>   /162.192.100.52:60928[1](queued=0,recved=38,sent=40)
>   /162.192.100.44:57544[1](queued=0,recved=38,sent=40)
>   /162.192.100.53:57879[1](queued=0,recved=868,sent=1202)
>   /162.192.100.51:52656[1](queued=0,recved=127,sent=162)
>   /162.192.100.47:51596[1](queued=0,recved=38,sent=40)
>   /162.192.100.53:48015[0](queued=0,recved=1,sent=0)
>   /162.192.100.42:60455[1](queued=0,recved=146,sent=185)
>   /162.192.100.48:60603[1](queued=0,recved=118,sent=150)
>   /162.192.100.41:49997[1](queued=0,recved=38,sent=41)
>   /162.192.100.43:49647[1](queued=0,recved=33,sent=37)
>   /162.192.100.52:60930[1](queued=0,recved=118,sent=150)
>   /162.192.100.43:49645[1](queued=0,recved=123,sent=159)
>   /162.192.100.49:42399[1](queued=0,recved=127,sent=162)
>   /162.192.100.48:60602[1](queued=0,recved=38,sent=40)
>
>  Latency min/avg/max: 0/24/208
>  Received: 2967
>  Sent: 3767
>  Outstanding: 0
>  Zxid: 0x60000026e
>  Mode: follower
>  Node count: 23
>
>
>
> On Fri, Jul 8, 2011 at 9:02 PM, Stack <stack@duboce.net> wrote:
>
>> When you looked at zk with zkdump or via the zkcli did it show max
>> connections as 2k?
>> St.Ack
>>
>> On Fri, Jul 8, 2011 at 8:14 AM, praveenesh kumar <praveenesh@gmail.com>
>> wrote:
>> > yeah..!!
>> >
>> > On Fri, Jul 8, 2011 at 8:43 PM, Stack <stack@duboce.net> wrote:
>> >
>> >> Is hbase managing zk?  Did you see zk restart when you restarted hbase?
>> >> St.Ack
>> >>
>> >> On Fri, Jul 8, 2011 at 8:09 AM, praveenesh kumar <praveenesh@gmail.com>
>> >> wrote:
>> >> > I restarted Hbase.. Is there any specific way to do that ?
>> >> >
>> >> > On Fri, Jul 8, 2011 at 8:35 PM, Stack <stack@duboce.net> wrote:
>> >> >
>> >> >> Did you restart your zk ensemble after making the change?
>> >> >> St.Ack
>> >> >>
>> >> >> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <
>> praveenesh@gmail.com>
>> >> >> wrote:
>> >> >> > Guys.. I have increased the zookeeper connections to 2000,
still I
>> am
>> >> >> > getting the same zookeeper exception while running the pig
scripts.
>> >> >> > Can anyone tell me what can be the reason ?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Praveenesh
>> >> >> >
>> >> >> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <
>> >> barneyfranks1@gmail.com
>> >> >> >wrote:
>> >> >> >
>> >> >> >> Why not just set the maxClientCnxcns to 0?  That would
make it
>> >> >> unlimited,
>> >> >> >> right?
>> >> >> >>
>> >> >> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yuzhihong@gmail.com>
>> wrote:
>> >> >> >>
>> >> >> >> > To 1000.
>> >> >> >> >
>> >> >> >> > Click on 'zk dump' link from Master UI.
>> >> >> >> > You would know roughly how many connections were
established.
>> >> >> >> > If 1000 is exhausted, increase more.
>> >> >> >> >
>> >> >> >> > See
>> >> >> >> >
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>> >> >> >> >
>> >> >> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar
<
>> >> >> praveenesh@gmail.com
>> >> >> >> > >wrote:
>> >> >> >> >
>> >> >> >> > > I did that.. I increased it to 200.. still facing
the same
>> >> error..
>> >> >> what
>> >> >> >> > > should be the ideal value ?
>> >> >> >> > >
>> >> >> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yuzhihong@gmail.com>
>> >> >> wrote:
>> >> >> >> > >
>> >> >> >> > > > Increase hbase.zookeeper.property.maxClientCnxns
to much
>> bigger
>> >> >> >> value.
>> >> >> >> > > >
>> >> >> >> > > >
>> >> >> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh
kumar <
>> >> >> >> > praveenesh@gmail.com
>> >> >> >> > > > >wrote:
>> >> >> >> > > >
>> >> >> >> > > > > My HBase and pig both are running
fine for me..
>> >> >> >> > > > > Only problem is coming whenever I
am trying to access big
>> >> hbase
>> >> >> >> table
>> >> >> >> > > > > containing too many columns and rows..
>> >> >> >> > > > > whenever I am trying to access hbase
table from pig... I
>> am
>> >> >> getting
>> >> >> >> > > this
>> >> >> >> > > > > following error in the middle of the
map process:
>> >> >> >> > > > >
>> >> >> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats
- ERROR 2997:
>> >> >> Unable
>> >> >> >> to
>> >> >> >> > > > > recreate exception from backed error:
>> >> >> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> HBase
>> >> is
>> >> >> able
>> >> >> >> > to
>> >> >> >> > > > > connect to ZooKeeper but the connection
closes
>> immediately.
>> >> This
>> >> >> >> > could
>> >> >> >> > > be
>> >> >> >> > > > a
>> >> >> >> > > > > sign that the server has too many
connections (30 is the
>> >> >> default).
>> >> >> >> > > > Consider
>> >> >> >> > > > > inspecting your ZK server logs for
that error and then
>> make
>> >> sure
>> >> >> >> you
>> >> >> >> > > are
>> >> >> >> > > > > reusing HBaseConfiguration as often
as you can. See
>> HTable's
>> >> >> >> javadoc
>> >> >> >> > > for
>> >> >> >> > > > > more information.
>> >> >> >> > > > > But if I access a small HBase table
from pig.. its working
>> >> >> fine..
>> >> >> >> no
>> >> >> >> > > > > zookeeper exception is coming. I am
able to do all other
>> >> things
>> >> >> >> > > perfectly
>> >> >> >> > > > > fine.
>> >> >> >> > > > > This error made me think that there
is some problem in my
>> >> >> zookeeper
>> >> >> >> > > that
>> >> >> >> > > > is
>> >> >> >> > > > > why I was asking is there anyway to
check whether the
>> >> zookeeper
>> >> >> are
>> >> >> >> > > > working
>> >> >> >> > > > > fine or not ?
>> >> >> >> > > > >
>> >> >> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
>> >> >> >> > > > > My hbase-site.xml
>> >> >> >> > > > >
>> >> >> >> > > > > <configuration>
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.rootdir</name>
>> >> >> >> > > > > <value>hdfs://ub13:54310/hbase</value>
>> >> >> >> > > > > <final>true</final>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > >
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.zookeeper.quorum</name>
>> >> >> >> > > > > <value>ub13,ub12,ub11</value>
>> >> >> >> > > > > <final>true</final>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
>> >> >> >> > > > > <value>200</value>
>> >> >> >> > > > > <final>true</final>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.cluster.distributed</name>
>> >> >> >> > > > > <value>true</value>
>> >> >> >> > > > > <description>The mode the cluster
will be in. Possible
>> values
>> >> >> are
>> >> >> >> > > > > false: standalone and pseudo-distributed
setups with
>> managed
>> >> >> >> > Zookeeper
>> >> >> >> > > > > true: fully-distributed with unmanaged
Zookeeper Quorum
>> >> >> >> > > > > </description>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > >
>> >> >> >> > > > >
>> >> >> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack
<stack@duboce.net>
>> >> >> wrote:
>> >> >> >> > > > >
>> >> >> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM,
praveenesh kumar <
>> >> >> >> > > praveenesh@gmail.com
>> >> >> >> > > > >
>> >> >> >> > > > > > wrote:
>> >> >> >> > > > > > > Can you please go through
this and tell me what can be
>> >> the
>> >> >> >> issue
>> >> >> >> > ?
>> >> >> >> > > > > > >
>> >> >> >> > > > > >
>> >> >> >> > > > > > Can you summarize?  (Life is
short)
>> >> >> >> > > > > >
>> >> >> >> > > > > > If stuff works generally -- shell?
 And can you do the
>> >> hbase
>> >> >> >> > > > > > rowcounter mapreduce job successfully
-- but then when
>> you
>> >> put
>> >> >> >> PIG
>> >> >> >> > in
>> >> >> >> > > > > > the picture it does not, it would
seem to be an issue
>> with
>> >> >> your
>> >> >> >> PIG
>> >> >> >> > > > > > setup?
>> >> >> >> > > > > >
>> >> >> >> > > > > > St.Ack
>> >> >> >> > > > > >
>> >> >> >> > > > >
>> >> >> >> > > >
>> >> >> >> > >
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>

Mime
View raw message