phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Tarnas <...@biotiquesystems.com>
Subject Re: Regionserver burns CPU and stops responding to RPC calls on HDP 2.1
Date Tue, 13 May 2014 21:56:28 GMT
Hello,

We set the HBase RegionServer Handler to 10 (it appears to have been set to 60 by Ambari during
install process). Now we have narrowed down what causes the CPU to increase and have some
detailed logs:

If we connect using sqlline.py and execute a select that selects one row using the primary_key,
no increate in CPU is observed and the number of RPC threads in a RUNNABLE state remains the
same. 

If we execute a select that scans the table such as "select count(*) from TABLE" or where
the "where" clause only limits on non-primary key attributes, then the number of RUNNABLE
RpcServer.handler threads increases and the CPU utilization of the regionserver increases
by ~105%. 

Disconnecting the client does not have an effect and the RpcServer.handler thread is left
RUNNABLE and the CPU stays at the higher usage. 

Checking the Web Console for the Regionserver just shows 10 RpcServer.reader tasks, all in
a WAITING state, no other monitored tasks are happening. The regionserver has a Max Heap of
10G and a Used heap of 445.2M.

I've attached the regionserver log with IPC debug logging turned on right when one of the
Phoenix statements is executed (this statement actually used up the last available handler).

thanks,
-chris


Mime
View raw message