hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhijit Pol <a...@rocketfuel.com>
Subject increasing hbase get latencies
Date Wed, 08 Jun 2011 23:39:30 GMT
We are on hbase 0.90 and using hbase for a while to perform high volume data
lookup using hbase client (no map-reduce involved).

Recently we observed that our "get" latencies keep increasing over the
period (and eventually flatten out at higher value) and if we restart hbase
server, latencies go back to good state (low values) and start increasing
again.

We do 60k qps reads and about 3k qps writes using 10 nodes cluster each with
large RAM and heavy use of block cache.

- checked GC logs, no full gc, small CMS pauses
- no abnormal logging on master or client side
- on region server once in a while we see following WARN dump

2011-06-08 19:xx:xx,188 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call get([B@16d0ea6f, row=adx:1CAESEInUuWF-DYppoEm8dJh5pFQ,
maxVersions=1, cacheBlocks=true, timeRange=[0,9223372036854775807),
families={(family=active_columns, columns=ALL}) from xx.xx.xx.xxx:34406:
output error
2011-06-08 19:xx:xx,189 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 311 on 60020 caught: java.nio.channels.ClosedChannelException
        at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
        at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1336)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:724)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:789)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1080)


since server restart make things look good, is this might be related to
minor compaction & block cache?


*
*

*Abhijit Pol | Senior Rocket Scientist | apol@rocketfuel.com | 408.892.3377
p |*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message