hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wellington Chevreuil <wellington.chevre...@gmail.com>
Subject Re: coprocessor and hbase.regionserver.handler.count
Date Tue, 22 Jan 2019 04:49:34 GMT
>
>
I am not familiar with thread-dumping or profiler. Is there any reference
> that I can study about how to use those tools to analyze this kind of issue?

You can start collecting jstack [1] frames of the RS process while your
test is running. Jstack command dumps state of all threads from the given
java process, you can then check which threads are blocked/running most of
the time. Below sample script would collect 20 frames with 1 second
interval, notice you would need to replace RSPID by the actual RS pid, and
it assumes RS process is running as hbase user (jstack needs to be ran as
same user as the java process).

for i in $(seq 1 20); do sudo -u hbase /usr/java/default/bin/jstack $RSPID
> /tmp/rs-jstack-$i.txt; sleep 1s; done



>

I just observed that all resources CPU, memory,. I/O are still idle, but
> the avg response time for a single request (put/get) become slower when the
> concurrency increasing.
>
Interesting, is the CP code using any type of locks? How is the response
time for single requests being measured? Is it possible to run this same
client with the CP disabled, and see if the performance issue still
manifests?

[1] https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstack.html



Em qui, 17 de jan de 2019 às 16:14, Ankit Singhal <ankitsinghal59@gmail.com>
escreveu:

> I hope you have configured 200+ handlers in server config and not creating
> a new connection in every thread. Sharing your java code, hbase version,
> machine configuration and hbase-site.xml may help us comment more.
>
> Regards,
> Ankit Singhal
>
> On Wed, Jan 16, 2019 at 11:35 PM ming.liu <ming.liu@esgyn.cn> wrote:
>
> > Thanks, Stack,
> >
> > I still cannot figure out the root cause. I am not familiar with
> > thread-dumping or profiler. Is there any reference that I can study about
> > how to use those tools to analyze this kind of issue?
> > Or is there some similar performance test that I can refer to? Maybe my
> > test is wrong.
> > I just write a simple java program to get a row from a table in a loop.
> > And by spawning this program from 1 to 200, and check the response time.
> > I just observed that all resources CPU, memory,. I/O are still idle, but
> > the avg response time for a single request (put/get) become slower when
> the
> > concurrency increasing. All the request goes to the same region in my
> test.
> > So I am trying to understand those parameters.
> >
> > Thanks,
> > Ming
> >
> > -----Original Message-----
> > From: Stack <stack@duboce.net>
> > Sent: Thursday, January 17, 2019 3:17 AM
> > To: Hbase-User <user@hbase.apache.org>
> > Subject: Re: coprocessor and hbase.regionserver.handler.count
> >
> > On Mon, Jan 14, 2019 at 5:37 PM ming.liu <ming.liu@esgyn.cn> wrote:
> >
> > > Hi, all,
> > >
> > >
> > >
> > > Our application is using coprocessor to communicate with HBase, not
> using
> > > the get()/put() client API. When there are large concurrency, the
> > > performance is degrading very fast.
> > >
> > >
> > Can you figure why? Thread-dumping, profiler?
> >
> >
> > > I checked there is hbase.regionserver.handler.count which control how
> > many
> > > thread in RS to handle client request. Will coprocessor use those same
> > > threads?
> > >
> > >
> > Yes. CPs decorate the existing read/write paths serviced by handlers.
> >
> > S
> >
> >
> >
> > >
> > >
> > > thanks,
> > >
> > > Ming
> > >
> > >
> > >
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message