hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: When compactions become major ones
Date Tue, 05 Jan 2016 17:37:04 GMT
For #1,
bq. would this minor which becomes major take care of deleted rows

Yes.

For #2, please consider the following guide:

dfs.blocksize (value: ${propdata["dfs.blocksize"]}) * 0.95 *
hbase.regionserver.maxlogs (value:
${propdata["hbase.regionserver.maxlogs"]}) should be greater than
hbase.regionserver.global.memstore.upperLimit * HBASE_HEAPSIZE (the value
for -Xmx)

Cheers

On Tue, Jan 5, 2016 at 8:39 AM, Mehdi Ben Haj Abbes <mehdi.abbes@gmail.com>
wrote:

> Thanks Ted for the clarification about the major compactions. So if I did
> understand well when a minor compaction is triggered and the policy selects
> all the store files, this compaction becomes a major one. But would this
> minor which becomes major take care of deleted rows as a major one would do
> or at the end it is just a minor that happened and selected all the store
> files ?
>
> About disabling splitting I have already hbase.hregion.max.filesize set to
> 10GB besides I pre splitted my table.
>
> And I still dont understand how the store files resulting after memstore
> flushes are having a size of 40MB. Does it hove smth to do with memstore
> upper limit and these 42MB are the result of forcing the memstore to be
> flushed? The problem is that all the newly store files added to HDFS are
> starting with this size (42MB) I did not mention that my CF is in-memory.
>
> Best regards,
>
> On Tue, Jan 5, 2016 at 4:04 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > For #1, when all store files are selected for compaction, the compaction
> > becomes major
> >
> > see 'Determine the Optimal Number of Pre-Split Regions' under:
> > http://hbase.apache.org/book.html#disable.splitting
> >
> > See also http://hbase.apache.org/book.html#managed.compactions
> >
> > Cheers
> >
> > On Tue, Jan 5, 2016 at 6:52 AM, Mehdi Ben Haj Abbes <
> mehdi.abbes@gmail.com
> > >
> > wrote:
> >
> > > Hi folks,
> > >
> > > I'm using hbase 0.98. I have heavy writes workload. I'm writing to one
> > > table with one CF compressed with GZ. My table is pre splitted to 27
> > > regions. As I start writing to this table I start seeing HFiles of the
> > size
> > > of 2-4 MB across the regions. I have the default hbase configuration
> for
> > > compaction properties. The compactions start as soon as I start writing
> > to
> > > HBase but many of these compaction are major ones. I can see this
> through
> > > HBase master UI on the table details view. So I wanted to understand
> > when a
> > > compaction becomes major.
> > >
> > > Another question, If I'm not wrong we have a memstore per region, so
> > when a
> > > memstore is flushed I will have a HFile with 128MB but I only see files
> > > with 42MB (without compression and 2.5MB when compressed with GZ).
> > >
> > > Any explanation ?
> > >
> > > Thanks in advance.
> > > --
> > > Mehdi BEN HAJ ABBES
> > >
> >
>
>
>
> --
> Mehdi BEN HAJ ABBES
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message