hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: MOB integration
Date Sat, 19 May 2018 18:08:23 GMT
Mob store file is renamed from /apps/hbase/data/mobdir to the final
location under region directory for the table.

This explains why you don't see much data under mobdir since data ingestion
has finished.

Cheers

On Sat, May 19, 2018 at 9:58 AM, Nicolas Paris <niparisco@gmail.com> wrote:

> Not having access cluster for few days, but I will be looking
> to logs.
>
> However, when looking at your logs, it seems that I mispell
> my mlob dir in the first post. It was "mobdir".
> The /apps/hbase/data/mobdir/ is nearly empty, sizing 4 or 10 kb
>
> Would this confirm the mob flushing process wouldn't be activated ?
>
>
>
> 2018-05-19 18:38 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:
>
> > If you have a chance to look at region server log, you would see some
> line
> > such as the following:
> >
> > 2018-05-19 16:31:23,548 INFO  [MemStoreFlusher.0] regionserver.HMobStore:
> > Renaming flushed file from
> > hdfs://mycluster/apps/hbase/data/mobdir/.tmp/
> > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377
> > to
> > hdfs://mycluster/apps/hbase/data/mobdir/data/default/
> > IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/
> > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377
> >
> > Meaning Mob store file is first saved under
> /apps/hbase/data/mobdir/.tmp/ ,
> > then renamed to under the usual location under region directory for the
> > table.
> >
> > From high level, as long as you can query what you ingested, you can be
> > assured that Mob data is persisted.
> >
> > Cheers
> >
> > On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <niparisco@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > ​Yes the data comes back as expected.
> > > My table is not called "mlob" however since I found such folder
> > > I thought it was storing mob objects.
> > >
> > > I do have 500 folder hashed as you mentionned. They contains the
> > > whole dataset (2TO)
> > > However, how beeing sure the data is actually stored as MOB (and not
> > > as traditional binary)
> > >
> > > Thanks
> > >
> > >
> > > 2018-05-19 15:59 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:
> > >
> > > > bq. look into hdfs hbase/data/mlob
> > > >
> > > > Is 'mlob' name of your table ?
> > > >
> > > > bq. nearly empty folder
> > > >
> > > > Here is listing under a one region table:
> > > >
> > > > drwxr-xr-x   - hbase hdfs          0 2018-05-16 23:51
> > > > /apps/hbase/data/data/default/atlas_janus/.tabledesc
> > > > drwxr-xr-x   - hbase hdfs          0 2018-05-16 23:51
> > > > /apps/hbase/data/data/default/atlas_janus/.tmp
> > > > drwxr-xr-x   - hbase hdfs          0 2018-05-17 00:55
> > > > /apps/hbase/data/data/default/atlas_janus/
> > 8033ea259cb7272d43bc137ca0ab29
> > > 06
> > > >
> > > > Not sure if the above matches your description of being nearly empty.
> > > > Here data is stored under 8033ea259cb7272d43bc137ca0ab2906
> > > >
> > > > If you query the table, does the data come back as expected ?
> > > >
> > > > Thanks
> > > >
> > > > On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris <niparisco@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > >
> > > > > I am using hbase 1.1 and hive 1.2
> > > > >
> > > > > I created an hbase table with a mob column with the default
> > > > > threshold (100K)
> > > > > I mapped the table into hive with a binary format, and loaded
> > > > > 20M of pdf of size between 50k and 20mb
> > > > >
> > > > > Apparently the mob is not populated because when I look into
> > > > > hdfs hbase/data/mlob, it is a nearly empty folder.
> > > > >
> > > > > Does it mean hive cannot populate hbase mob columns  ?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message