hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Ilyin <alexan...@weborama.com>
Subject Spikes in HBase response time
Date Mon, 24 Oct 2016 12:44:45 GMT
Hi,

I'm experiencing some spikes in HBase response time. The problem appeared
in one moment: before all the requests used to take 4-30ms to complete and
now some of them (several percent) take as long as 800ms decreasing
significantly an average processing time.

This happens with the requests to all the tables I have so it doesn't seem
to be related to some problem in data schema. And it didn't change anyway.

I don't see any problems with the lack of resources: there is enough CPU,
memory on datanodes.

Automatic splitting is off, no cluster balancing is taking place so it
shouldn't be the client retries as well.

There are some stacktraces in datanode log but as far as I understand these
are normal and anyway they had existed before the problem appeared:
2016-10-24 05:42:19,186 INFO
[B.defaultRpcServer.handler=17,queue=2,port=16020]
shortcircuit.ShortCircuitCache: ShortCircuitCache(0x3a130ae2): could not
load 1075792477_BP-415105157-10.190.240.206-1468423545420 due to
InvalidToken exception.
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control
error while attempting to set up short-circuit access to
/apps/hbase/data/data/profile/DailyUserScores/84105becaefcf071e43ebc4972d0249f/d/c7653cc98860451cb79e678e9c3f0e85Block
token with block_token_identifier (expiryDate=1477279957879,
keyId=2039256459, userId=hbase,
blockPoolId=BP-415105157-10.190.240.206-1468423545420, blockId=1075792477,
access modes=[READ]) is expired.

At the moment I'm a bit stuck so any advices on what to check next are
welcome.

PS. HBase version is 1.1.2

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message