hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Hbase thrift server issues
Date Sun, 10 Jan 2016 17:40:06 GMT
What I meant was to capture stack trace of the thrift server (using jstack
command) when your query is running.

This would give us some more clue as to the state thrift server was in when
the problem happened.

Cheers

On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <rajeshkumarit8292@gmail.com>
wrote:

> Hi Ted,
>
>    This is what I found in the Hbase thrift server logs
>
> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:java.library.path=;C:\Hadoop\bin
> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=C:\Windows\TEMP\
> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:os.name=Windows Server 2012
> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:os.version=6.2
> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:user.name=xxxx
> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:user.home=C:\
> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:user.dir=C:\HBase\bin
> 2016-01-09 06:27:43,142 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Initiating client connection,
> connectString=hostname:2181,hostname:2181,hostname:2181
> sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> 2016-01-09 06:27:43,339 INFO  [thrift-worker-0-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2016-01-09 06:27:43,342 INFO  [thrift-worker-0-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> hostname/xx.xx.x.xx:2181, initiating session
> 2016-01-09 06:27:43,365 INFO  [thrift-worker-0-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated timeout
> = 40000
> 2016-01-09 07:14:10,607 INFO
>  [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> Monitor missed its start time
> 2016-01-09 07:16:43,511 INFO
>  [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> Client=Administrator/null create 'chk2', {NAME => 'fa', DATA_BLOCK_ENCODING
> => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}
> 2016-01-09 07:16:44,121 INFO  [ProcedureExecutorThread-1]
> procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> exec-time=462msec exception=org.apache.hadoop.hbase.TableExistsException:
> chk2
> 2016-01-09 07:49:55,574 INFO  [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x1522613893e0004
> 2016-01-09 07:49:55,629 INFO  [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> 2016-01-09 07:49:55,661 INFO  [thrift-worker-0-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2016-01-09 18:40:29,469 INFO  [thrift-worker-1]
> zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x591287f8
> connecting to ZooKeeper ensemble=hostname:2181,hostname:2181,hostname:2181
> 2016-01-09 18:40:29,482 INFO  [thrift-worker-1] zookeeper.ZooKeeper:
> Initiating client connection,
> connectString=hostname:2181,hostname:2181,hostname:2181
> sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> 2016-01-09 18:40:29,553 INFO  [thrift-worker-1-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2016-01-09 18:40:29,559 INFO  [thrift-worker-1-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> hostname/xx.xx.x.xx:2181, initiating session
> 2016-01-09 18:40:29,569 INFO  [thrift-worker-1-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated timeout
> = 40000
> 2016-01-09 18:54:25,558 INFO  [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x35226137e300008
> 2016-01-09 18:54:25,566 INFO  [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> 2016-01-09 18:54:25,578 INFO  [thrift-worker-1-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2016-01-09 20:05:42,469 INFO  [thrift-worker-13]
> zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6305dbeb
> connecting to ZooKeeper ensemble=hostname:2181,hostname:2181,hostname:2181
> 2016-01-09 20:05:42,476 INFO  [thrift-worker-13] zookeeper.ZooKeeper:
> Initiating client connection,
> connectString=hostname:2181,hostname:2181,hostname:2181
> sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> 2016-01-09 20:05:42,507 INFO  [thrift-worker-13-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2016-01-09 20:05:42,510 INFO  [thrift-worker-13-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> hostname/xx.xx.x.xx:2181, initiating session
> 2016-01-09 20:05:42,708 INFO  [thrift-worker-13-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout
> = 40000
> 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x1522613893e0005
> 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
>
>
> Thanks,
> Rajeshkumar J
>
> On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Can you take a stack trace of the thrift server and pastebin the trace ?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > I am connecting via Hbase thrift server to access records in Hbase and
> I
> > am
> > > doing this in C# and i am using range scan to get records from the
> table.
> > > This seems to be working for few hours but after few hours it didn't
> > > returned any records and then i searched hbase logs i found this
> > >
> > > 2016-01-04 00:44:54,880 INFO  [ConnectionCleaner]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x3520b1721650006
> > > 2016-01-04 00:44:54,891 INFO  [ConnectionCleaner] zookeeper.ZooKeeper:
> > > Session: 0x3520b1721650006 closed
> > > 2016-01-04 00:44:54,891 INFO  [thrift-worker-0-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > >
> > > And I am using hbase 0.98.13
> > >
> > > I have asked this question before and some user suggested me to upgrade
> > to
> > > new version so i have upgraded HBase to 1.1.2. But now it also it
> didn't
> > > return any records but different log generated as shown below
> > >
> > > 2016-01-09 20:05:42,708 INFO  [thrift-worker-13-SendThread(xxxx:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> timeout =
> > > 40000
> > > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x1522613893e0005
> > > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > 2016-01-09 20:15:45,567 INFO  [thrift-worker-13-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > >
> > > Can any one help me in this?
> > >
> > > Regards,
> > > Rajeshkumar J
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message