hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Wilson <registrat...@circle-cross-jn.com>
Subject Re: HBASE -- RS expire?
Date Fri, 06 Jul 2012 04:03:51 GMT
I don't see that in the RS logs.  Would I see that in the ZK logs?

At this point there is no network.  Just a switch.  I reduced the number
of nodes to 40 and had all of them placed on the same switch with a
single vlan.  I even had the network techs use a completely different
switch just to be safe.

Is there some heatbeat timer I can tweak?

---
Jay Wilson

On 7/5/2012 8:34 PM, Amandeep Khurana wrote:
> 
> 
> On Thursday, July 5, 2012 at 8:25 PM, Jay Wilson wrote:
> 
>> Finally my HMaster has stabilized and been running for 7 hours. I
>> believe my networking issues are behind me now. Thank you everyone for
>> the help.
>>
>>
> 
> Awesome.
> 
> Looks like the same issue is biting you with the RS too. The RS isn't heartbeating to
ZK and the ZK session expires, causing the RS to die.
> Do you see a YouAreDeadException in the logs? 
>>
>> New issue is my RSes continue to die after about 20 minutes. Again the
>> cluster is idle. No jobs are running and I get this on all of my RSes
>> at almost the same time:
>>
>> 2012-07-05 19:34:05,283 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server devrackA-04/172.18.0.5:2181
>> 2012-07-05 19:34:05,288 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection established to devrackA-04/172.18.0.5:2181, initiating session
>> 2012-07-05 19:34:05,301 INFO org.apache.zookeeper.ClientCnxn: Session
>> establishment complete on server devrackA-04/172.18.0.5:2181, sessionid
>> = 0x13858fc240f0003, negotiated timeout = 180000
>> 2012-07-05 19:34:05,399 INFO
>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Installed shutdown
>> hook thread: Shutdownhook:regionserver60020
>> 2012-07-05 20:06:40,279 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> read additional data from server sessionid 0x13858fc240f0003, likely
>> server has closed socket, closing socket connection and attempting reconnect
>> 2012-07-05 20:06:40,573 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server devrackA-03/172.18.0.4:2181
>> 2012-07-05 20:06:40,574 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection established to devrackA-03/172.18.0.4:2181, initiating session
>> 2012-07-05 20:06:40,578 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> reconnect to ZooKeeper service, session 0x13858fc240f0003 has expired,
>> closing socket connection
>> 2012-07-05 20:06:40,586 FATAL
>> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> server serverName=devrackB-07,60020,1341542045088, load=(requests=0,
>> regions=0, usedHeap=0, maxHeap=0): regionserver:60020-0x13858fc240f0003
>> regionserver:60020-0x13858fc240f0003 received expired from ZooKeeper,
>> aborting
>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired
>>
>> Could the fact that the cluster is idle cause the sessions to expire?
>> It's almost like a timing trigger pops, the sessions expire, and then
>> can reconnect. Is there a timer I need to adjust?
>>
>> Could this be related to a TCP or IP timer that needs to be adjusted?
>> The session goes into a Fin/Wait state and then closes?
>>
>> Thank you
>> ---
>> Jay Wilson
>>
>>
> 
> 
> 



Mime
View raw message