hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manjeet Singh <manjeet.chand...@gmail.com>
Subject Re: Query for OldWals and use of WAl for Hbase indexer
Date Wed, 11 Jul 2018 17:49:47 GMT
Thanks Sean for your reply

I still have some question un answered like
Q1: How Hbase syncronized with Hbase indexer.
Q2 What optimization I can apply.
Q3 As it's clear from my stats, data in OldWals is quite huge so it's not
getting clear my HMaster., how can I improve my HDFS space issue due to
this?

Thanks
Manjeet Singh

On Wed, Jul 11, 2018 at 9:33 PM, Sean Busbey <busbey@apache.org> wrote:

> Presuming you're using the Lily indexer[1], yes it relies on hbase's
> built in cross-cluster replication.
>
> The replication system stores WALs until it can successfully send them
> for replication. If you look in ZK you should be able to see which
> regionserver(s) are waiting to send those WALs over. The easiest way
> to do this is probably to look at the "zk dump" web page on the
> Master's web ui[2].
>
> Once you have the particular region server(s), take a look at their
> logs for messages about difficulty sending edits to the replication
> peer you have set up for the destination solr collection.
>
> If you remove the WALs then the solr collection will have a hole in
> it. Depending on how far behind you are, it might be quicker to 1)
> remove the replication peer, 2) wait for old wals to clear, 3)
> reenable replication, 4) use a batch indexing tool to index data
> already in the table.
>
> [1]:
>
> http://ngdata.github.io/hbase-indexer/
>
> [2]:
>
> The specifics will vary depending on your installation, but the page
> is essentially at a URL like
> https://active-master-host.example.com:22002/zk.jsp
>
> the link is on the master UI landing page, near the bottom, in the
> description of the "ZooKeeper Quorum" row. it's the end of "Addresses
> of all registered ZK servers. For more, see zk dump."
>
> On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh
> <manjeet.chandhok@gmail.com> wrote:
> > Hi All
> >
> > I have a query regarding Hbase replication and OldWals
> >
> > Hbase version 1.2.1
> >
> > To enable Hbase indexing we use below command on table
> >
> > alter '<NameOfTable>', {NAME => 'CF1', REPLICATION_SCOPE => 1}
> >
> > By Doing this actually replication get enabled as hbase-indexer required
> > it, as per my understanding indexer use hbase WAL (Please correct me if I
> > am wrong).
> >
> > so question is How Hbase syncronize with Solr Indexer? What is the role
> of
> > replication? what optimization we can apply in order to reduce data size?
> >
> >
> > I can see that our OldWals are getting filled , if Hmaster it self taking
> > care why it's reached to 7.2 TB? what if I delete it, does it impact solr
> > indexing?
> >
> > 7.2 K   21.5 K  /hbase/.hbase-snapshot
> > 0       0       /hbase/.tmp
> > 0       0       /hbase/MasterProcWALs
> > 18.3 G  60.2 G  /hbase/WALs
> > 28.7 G  86.1 G  /hbase/archive
> > 0       0       /hbase/corrupt
> > 1.7 T   5.2 T   /hbase/data
> > 42      126     /hbase/hbase.id
> > 7       21      /hbase/hbase.version
> > 7.2 T   21.6 T  /hbase/oldWALs
> >
> >
> >
> >
> > Thanks
> > Manjeet Singh
>



-- 
luv all

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message