hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: ZooKeeperConnectionException whenever table is flushed or majorcompacted
Date Tue, 05 Apr 2011 05:04:41 GMT
Nothing comes to mind as to why it would "fix" it, maybe I don't
understand what you did instead.

BTW I created https://issues.apache.org/jira/browse/HBASE-3734 to
track the issue.

J-D

On Mon, Apr 4, 2011 at 9:54 PM, Hari Sreekumar <hsreekumar@clickable.com> wrote:
> Ah, I didn't notice it was happening for tableExists() in this instance. But
> it was always happening for flush() and majorCompact() methods earlier so I
> didn't check the log when copying it. So I thought it might have something
> to do with these methods. Yes, I see "Too many connections" error in the
> zookeeper logs just about the time I go the exception.
>
> Basically, I was using the same HBaseAdmin instance to create all the tables
> (There are about 15 tables). When I split it into groups of 5 tables with
> each HBaseAdmin instance, and then recreating an HBaseAdmin before going for
> the other tables, then I don't get this error. When I had done this, I was
> assuming something related to the asynchronous nature of flush(), but that
> doesn't make sense at all now. So why is it getting fixed (seems to) when I
> do this?
>
> Thanks,
> Hari
>
> On Mon, Apr 4, 2011 at 10:40 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> As far as I can tell the async nature of those operations has nothing
>> to do with what you see since it's not even able to get a session from
>> ZooKeeper (so it's not even talking to the region servers). If you
>> look at the stack trace:
>>
>>
>> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
>> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>>
>> It's not even happening on the call you said it did ;)
>>
>> But going through the code I see a major issue,
>> HBaseAdmin.getCatalogTracker creates new Configuration like if it was
>> free (it's not). Calling this method more than 30 times will probably
>> give you a "too many connections" in the zookeeper log (if you look at
>> the correct, what you pasted is 8 seconds later).
>>
>> I'll open a jira.
>>
>> J-D
>>
>> On Mon, Apr 4, 2011 at 3:43 AM, Hari Sreekumar <hsreekumar@clickable.com>
>> wrote:
>> > I have this in the zookeeper logs, which might be helpful:
>> >
>> > 2011-04-04 21:26:40,356 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51467 which had sessionid 0x12f20e3713b02ee
>> > 2011-04-04 21:26:40,357 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> > EndOfStreamException: Unable to read additiona
>> > l data from client sessionid 0x12f20e3713b02d9, likely client has closed
>> > socket
>> > 2011-04-04 21:26:40,357 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51435 which had sessionid 0x12f20e3713b02d9
>> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> > EndOfStreamException: Unable to read additiona
>> > l data from client sessionid 0x12f20e3713b02e5, likely client has closed
>> > socket
>> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51453 which had sessionid 0x12f20e3713b02e5
>> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> > EndOfStreamException: Unable to read additiona
>> > l data from client sessionid 0x12f20e3713b02fd, likely client has closed
>> > socket
>> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51490 which had sessionid 0x12f20e3713b02fd
>> >
>> >
>> >
>> > On Mon, Apr 4, 2011 at 4:03 PM, Hari Sreekumar <hsreekumar@clickable.com
>> >wrote:
>> >
>> >> Hi,
>> >>
>> >> I get this exception when I try to flush META using
>> >> HbaseAdmin.flush(".META."). I get the same exception when I do major
>> >> compact:
>> >>
>> >> 11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection
>> >> established to hadoopqa2/192.168.1.50:2181, initiating session
>> >> 11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
>> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
>> >> connection and attempting reconnect
>> >> java.io.IOException: Connection reset by peer
>> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
>> >> connection to server hadoopqa2/192.168.1.50:2181
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection
>> >> established to hadoopqa2/192.168.1.50:2181, initiating session
>> >> 11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
>> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
>> >> connection and attempting reconnect
>> >> java.io.IOException: Connection reset by peer
>> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >> *************************************************************
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >> KeeperErrorCode = ConnectionLoss for /hbase
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
>> >>
>> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>> >>
>> >>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>> >>
>> >>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
>> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> java.lang.reflect.Method.invoke(Method.java:597)
>> >> org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> >>
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >> *************************************************************
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
>> >> connection to server hadoopqa2/192.168.1.50:2181
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection
>> >> established to hadoopqa2/192.168.1.50:2181, initiating session
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
>> >> additional data from server sessionid 0x0, likely server has closed
>> socket,
>> >> closing socket connection and attempting reconnect
>> >>
>> >> This doesn't happen always. I think it is happening sometimes only. I
>> >> suspect this has something to do with the asynchronous nature of these
>> >> commands. Any ideas?
>> >>
>> >> Thanks,
>> >> Hari
>> >>
>> >
>>
>

Mime
View raw message