hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Very high read rate in a single RegionServer
Date Mon, 26 Jan 2015 04:43:16 GMT
Please try to determine how much percent of the reqs/sec metric is due to
read.

At this level, do your queries / jobs get slower ?

Cheers

On Sun, Jan 25, 2015 at 6:48 PM, Pun Intended <punintendedyo@gmail.com>
wrote:

> Sorry for the confusion.  I should have used the same place to grab the
> numbers from.
>
> In my original email, I grabbed the reqs/sec metric  from the Hbase UI
> directly. It was showing 1000 Reqs/sec. Right now, when I look at that, the
> number is 891 Reqs/sec, but it fluctuates.
>
> The 55K Reqs/sec in my last email, I got from Cloudera Manager. I am not
> sure why it is reporting a different number. It was 55K Reqs/sec, but now
> it jumped back up to 60K Reqs/sec.
>
>
>
> On Sun, Jan 25, 2015 at 9:37 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > From your first email:
> >
> > bq. 1000reqs/sec vs 4-5 reqs/sec elsewhere
> >
> > Now the request count becomes 55K requests/sec ?
> > Just want to clarify whether the request count increased or decreased
> when
> > compaction is not in progress.
> >
> > Cheers
> >
> > On Sun, Jan 25, 2015 at 6:23 PM, Pun Intended <punintendedyo@gmail.com>
> > wrote:
> >
> > > Thanks!
> > > I ran hbase hbck and all seems healthy.
> > >
> > > 0 inconsistencies detected.
> > > Status: OK
> > >
> > > Also the meta region shows to be 100% compacted, but I still see 55K
> > > requests/sec to that RegionServer:
> > >
> > >                                            Region
> > > Name                                          Num. Compacting KVs Num.
> > > Compacted KVs Compaction Progress
> > >
> > > hbase:meta,,1.1588230740
> > > 195349              195349             100.00%
> > >
> > > best
> > >
> > >
> > > On Sun, Jan 25, 2015 at 8:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > After the adjustment, hbase:meta would still be accessed for region
> > > > locations. But the request count should come down noticeably.
> > > >
> > > > bq. Is there a way to inspect whether I have corruptions in the META
> > > table?
> > > >
> > > > Please take a look at:
> > > > http://hbase.apache.org/book.html#hbck.in.depth
> > > >
> > > > Cheers
> > > >
> > > > On Sun, Jan 25, 2015 at 5:20 PM, Pun Intended <
> punintendedyo@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Will do! Thanks you for your suggestion. I will keep an eye on the
> > read
> > > > > rate to that RegionServer, but in case it doesn't drop, do you
> think
> > > > there
> > > > > should be something else I need to look at?
> > > > >
> > > > > * Is there a way to inspect whether I have corruptions in the META
> > > table?
> > > > > * Any other possible misconfiguration that comes to mind that I
> > should
> > > > keep
> > > > > an eye for?
> > > > >
> > > > > best
> > > > >
> > > > > On Sun, Jan 25, 2015 at 8:05 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > > >
> > > > > > Suggest you lower the frequency of compacting hbase:meta
> > > > > >
> > > > > > Move the second command to another script.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Jan 25, 2015, at 4:46 PM, Pun Intended <
> > punintendedyo@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > I have a small script:  major_compact_hbase.hbsh
> > > > > > >
> > > > > > > major_compact 'users'
> > > > > > > major_compact 'hbase:meta'
> > > > > > > exit
> > > > > > >
> > > > > > >
> > > > > > > and a cron calls it once a day with:
> > > > > > > /usr/bin/hbase shell major_compact_hbase.hbsh
> > > > > > >
> > > > > > > Thanks again!
> > > > > > >
> > > > > > >
> > > > > > >> On Sun, Jan 25, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com>
> > > > wrote:
> > > > > > >>
> > > > > > >> How do you trigger major compaction manually ?
> > > > > > >>
> > > > > > >> Cheers
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>>> On Jan 25, 2015, at 3:40 PM, Pun Intended <
> > > > punintendedyo@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> Thank you so much for your help!
> > > > > > >>> You are right, the slow region server hosts hbase:meta!!!
 I
> am
> > > > using
> > > > > > >> Hbase
> > > > > > >>> 0.96.1.1
> > > > > > >>>
> > > > > > >>> I have major compactions disabled. I run major
compaction
> > > manually
> > > > > > once a
> > > > > > >>> day and when it dpes that besides requesting major
compaction
> > on
> > > my
> > > > > > >> tables,
> > > > > > >>> it also requests a major_compact 'hbase:meta'.
Is that what
> > could
> > > > be
> > > > > > >>> causing the issue? I did a scan on 'hbase:meta'
and I see
> that
> > it
> > > > > > >> contains
> > > > > > >>> about the same number of regions as it should -
I have two
> > tables
> > > > > with
> > > > > > >>> ~3000 regions each.
> > > > > > >>>
> > > > > > >>> Do you think I have some misconfiguration? Do I
need to do
> some
> > > > > cleanup
> > > > > > >> on
> > > > > > >>> the meta region? Am I hitting some Hbase bug? Or
am I being
> > dumb
> > > > > > >> requesting
> > > > > > >>> major compaction for the meta region daily?
> > > > > > >>>
> > > > > > >>> I believe on the client side I am doing things
right - using
> > > > scanner
> > > > > > >>> caching, etc.
> > > > > > >>>
> > > > > > >>> Thank you so much for your help!!
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message