hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@apache.org>
Subject Re: Query for OldWals and use of WAl for Hbase indexer
Date Wed, 11 Jul 2018 16:03:21 GMT
Presuming you're using the Lily indexer[1], yes it relies on hbase's
built in cross-cluster replication.

The replication system stores WALs until it can successfully send them
for replication. If you look in ZK you should be able to see which
regionserver(s) are waiting to send those WALs over. The easiest way
to do this is probably to look at the "zk dump" web page on the
Master's web ui[2].

Once you have the particular region server(s), take a look at their
logs for messages about difficulty sending edits to the replication
peer you have set up for the destination solr collection.

If you remove the WALs then the solr collection will have a hole in
it. Depending on how far behind you are, it might be quicker to 1)
remove the replication peer, 2) wait for old wals to clear, 3)
reenable replication, 4) use a batch indexing tool to index data
already in the table.




The specifics will vary depending on your installation, but the page
is essentially at a URL like

the link is on the master UI landing page, near the bottom, in the
description of the "ZooKeeper Quorum" row. it's the end of "Addresses
of all registered ZK servers. For more, see zk dump."

On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh
<manjeet.chandhok@gmail.com> wrote:
> Hi All
> I have a query regarding Hbase replication and OldWals
> Hbase version 1.2.1
> To enable Hbase indexing we use below command on table
> alter '<NameOfTable>', {NAME => 'CF1', REPLICATION_SCOPE => 1}
> By Doing this actually replication get enabled as hbase-indexer required
> it, as per my understanding indexer use hbase WAL (Please correct me if I
> am wrong).
> so question is How Hbase syncronize with Solr Indexer? What is the role of
> replication? what optimization we can apply in order to reduce data size?
> I can see that our OldWals are getting filled , if Hmaster it self taking
> care why it's reached to 7.2 TB? what if I delete it, does it impact solr
> indexing?
> 7.2 K   21.5 K  /hbase/.hbase-snapshot
> 0       0       /hbase/.tmp
> 0       0       /hbase/MasterProcWALs
> 18.3 G  60.2 G  /hbase/WALs
> 28.7 G  86.1 G  /hbase/archive
> 0       0       /hbase/corrupt
> 1.7 T   5.2 T   /hbase/data
> 42      126     /hbase/hbase.id
> 7       21      /hbase/hbase.version
> 7.2 T   21.6 T  /hbase/oldWALs
> Thanks
> Manjeet Singh

View raw message