hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Sreekumar <hsreeku...@clickable.com>
Subject Re: ZooKeeperConnectionException whenever table is flushed or majorcompacted
Date Tue, 05 Apr 2011 04:54:48 GMT
Ah, I didn't notice it was happening for tableExists() in this instance. But
it was always happening for flush() and majorCompact() methods earlier so I
didn't check the log when copying it. So I thought it might have something
to do with these methods. Yes, I see "Too many connections" error in the
zookeeper logs just about the time I go the exception.

Basically, I was using the same HBaseAdmin instance to create all the tables
(There are about 15 tables). When I split it into groups of 5 tables with
each HBaseAdmin instance, and then recreating an HBaseAdmin before going for
the other tables, then I don't get this error. When I had done this, I was
assuming something related to the asynchronous nature of flush(), but that
doesn't make sense at all now. So why is it getting fixed (seems to) when I
do this?

Thanks,
Hari

On Mon, Apr 4, 2011 at 10:40 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> As far as I can tell the async nature of those operations has nothing
> to do with what you see since it's not even able to get a session from
> ZooKeeper (so it's not even talking to the region servers). If you
> look at the stack trace:
>
>
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>
> It's not even happening on the call you said it did ;)
>
> But going through the code I see a major issue,
> HBaseAdmin.getCatalogTracker creates new Configuration like if it was
> free (it's not). Calling this method more than 30 times will probably
> give you a "too many connections" in the zookeeper log (if you look at
> the correct, what you pasted is 8 seconds later).
>
> I'll open a jira.
>
> J-D
>
> On Mon, Apr 4, 2011 at 3:43 AM, Hari Sreekumar <hsreekumar@clickable.com>
> wrote:
> > I have this in the zookeeper logs, which might be helpful:
> >
> > 2011-04-04 21:26:40,356 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51467 which had sessionid 0x12f20e3713b02ee
> > 2011-04-04 21:26:40,357 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > EndOfStreamException: Unable to read additiona
> > l data from client sessionid 0x12f20e3713b02d9, likely client has closed
> > socket
> > 2011-04-04 21:26:40,357 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51435 which had sessionid 0x12f20e3713b02d9
> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > EndOfStreamException: Unable to read additiona
> > l data from client sessionid 0x12f20e3713b02e5, likely client has closed
> > socket
> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51453 which had sessionid 0x12f20e3713b02e5
> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > EndOfStreamException: Unable to read additiona
> > l data from client sessionid 0x12f20e3713b02fd, likely client has closed
> > socket
> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51490 which had sessionid 0x12f20e3713b02fd
> >
> >
> >
> > On Mon, Apr 4, 2011 at 4:03 PM, Hari Sreekumar <hsreekumar@clickable.com
> >wrote:
> >
> >> Hi,
> >>
> >> I get this exception when I try to flush META using
> >> HbaseAdmin.flush(".META."). I get the same exception when I do major
> >> compact:
> >>
> >> 11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection
> >> established to hadoopqa2/192.168.1.50:2181, initiating session
> >> 11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
> >> connection and attempting reconnect
> >> java.io.IOException: Connection reset by peer
> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> >> connection to server hadoopqa2/192.168.1.50:2181
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection
> >> established to hadoopqa2/192.168.1.50:2181, initiating session
> >> 11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
> >> connection and attempting reconnect
> >> java.io.IOException: Connection reset by peer
> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >> *************************************************************
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss for /hbase
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
> >>
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
> >>
> >>
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
> >>
> >>
> com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> java.lang.reflect.Method.invoke(Method.java:597)
> >> org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >> *************************************************************
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> >> connection to server hadoopqa2/192.168.1.50:2181
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection
> >> established to hadoopqa2/192.168.1.50:2181, initiating session
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
> >> additional data from server sessionid 0x0, likely server has closed
> socket,
> >> closing socket connection and attempting reconnect
> >>
> >> This doesn't happen always. I think it is happening sometimes only. I
> >> suspect this has something to do with the asynchronous nature of these
> >> commands. Any ideas?
> >>
> >> Thanks,
> >> Hari
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message