hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajeshkumar J <rajeshkumarit8...@gmail.com>
Subject Re: RPC calls in hbase
Date Mon, 09 Jan 2017 09:50:14 GMT
we have 7000 regions in which we only have data in 20 regions. Whether this
may be the reason for this?

On Thu, Jan 5, 2017 at 10:58 PM, Stack <stack@duboce.net> wrote:

> On Thu, Jan 5, 2017 at 6:09 AM, Rajeshkumar J <rajeshkumarit8292@gmail.com
> >
> wrote:
>
> > Adding this
> > Here Clauster have three machines each have 125 GB as RAM in which 70 GB
> is
> > free, One Machine acts as HMaster and also Region Server. Another machine
> > acts as Secondary Hmaster and also Region Server. Third Machine is a
> > dedicated Hregion Server. We have about 7000 Regions.
> > Maybe is this the problem?? Containing more regions
> >
> >
> Hard to diagnose with so little info. Given the above, you have only a few
> (if beefy) machines with more than the usual allotment of regions.
>
> Suggest you do some background reading and study your cluster while under
> load looking at resource usage, metrics, logs and the usual suspects.
>
> St.Ack
>
>
>
> > On Thu, Jan 5, 2017 at 3:27 AM, Stack <stack@duboce.net> wrote:
> >
> > > (Thanks for jumping in Billy)
> > >
> > > Smaller Scans? (This is good on Scans in 1.1.+ which you seem to be on:
> > > https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1)
> > >
> > > Otherwise look at other factors. Are the servers loaded? GC on
> > > client/server or resources being taken up by adjacent processes; etc.
> > >
> > > St.Ack
> > >
> > >
> > >
> > > On Wed, Jan 4, 2017 at 11:00 AM, Billy Watson <
> williamrwatson@gmail.com>
> > > wrote:
> > >
> > > > Disclaimer: this is what I think I know and I'm not an HBase
> > > contributor...
> > > >
> > > > Anytime we get scanner timeouts like that, we first expand the
> timeout
> > > > itself like you're doing to fix the short-term problem. But then we
> > look
> > > at
> > > > our rowkey design and what we're doing. Timeouts like that are
> usually
> > > > symptomatic of bad rowkey utilization, too few servers or too many
> > > regions
> > > > per server (and therefore too little RAM to serve the regions).
> > > >
> > > > Make sure you're using good rowkey design and actually taking
> advantage
> > > of
> > > > your rowkey. HBase wasn't really intended to do random scans of
> columns
> > > > without rowkey filters which is why I think that timeout is set so
> low
> > by
> > > > default.
> > > >
> > > > This page helps:
> > > >
> > > > http://hbase.apache.org/0.94/book/rowkey.design.html
> > > >
> > > > William Watson
> > > > Lead Software Engineer
> > > >
> > > > On Wed, Jan 4, 2017 at 1:41 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Stack,
> > > > >
> > > > >    Yes it worked. I had a serious doubt regarding lease does not
> > exist
> > > > > exception. We have 30 tables which have 45 million records each.
We
> > are
> > > > > scanning the table initially process is successful but after it
> gets
> > > > failed
> > > > > with lease does not exist exception. By changing hbase.rpc.timeout
> > and
> > > > > hbase.client.scanner.timeout.period it got succeeded. It runs for
> > > > sometime
> > > > > but again it fails. Then again increasing the value it gets
> > succeeded.
> > > > > Initially I have default values for these two property 60000 and
> then
> > > > > changed to 70000 and then changed to 80000.
> > > > >
> > > > >   Can you tell me how to resolve this?
> > > > >
> > > > > On Wed, Jan 4, 2017 at 11:42 AM, Stack <stack@duboce.net> wrote:
> > > > >
> > > > > > Yeah. Try it.
> > > > > > S
> > > > > >
> > > > > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J <
> > > > > rajeshkumarit8292@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > So I need to add property in log4j.properties as below
> > > > > > >
> > > > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE
> > > > > > >
> > > > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack <stack@duboce.net>
> wrote:
> > > > > > >
> > > > > > > > See http://hbase.apache.org/book.html#log4j
> > > > > > > >
> > > > > > > > Or to set it temporarily via UI, see
> > > > > > > > http://hbase.apache.org/book.html#trouble.log.levels
> > > > > > > >
> > > > > > > > St.Ack
> > > > > > > >
> > > > > > > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J <
> > > > > > > rajeshkumarit8292@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Do we have to include this property in hbase-site.xml?
> > > > > > > > >
> > > > > > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack <stack@duboce.net>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar
J <
> > > > > > > > > > rajeshkumarit8292@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Do all rpc calls are logged in hbase
region server log?
> > > Also
> > > > I
> > > > > > need
> > > > > > > > to
> > > > > > > > > > find
> > > > > > > > > > > the time taken for each scan calls
is this possible
> > > > > > > > > > >
> > > > > > > > > > > If you enable trace level logging for
the rpcserver
> > class,
> > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcServer, then
you can see
> a
> > > > bunch
> > > > > of
> > > > > > > > > detail
> > > > > > > > > > on each rpc invocation (including timings,
size, etc):
> > > > > > > > > >
> > > > > > > > > > St.Ack
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message