hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wellington Chevreuil <wellington.chevre...@gmail.com>
Subject Re: Removal of table rows.
Date Wed, 16 Dec 2020 09:08:27 GMT
>
> Hbase:meta has info for one table region which is NOT on a master. Is that
> correct? I would have expected all meta info to be stored on a master.
>
meta table is an "hbase system table" that has info about which regions are
assigned to which region servers in your cluster. As any user table, it
also has a region, and this region must be hosted on a region server in the
cluster. Master is just responsible for the coordination of regions and
some other housekeeping actions.

As I state below, a simple method to remove everything connected to
> alfa:rfilenameext would seem the easiest of tasks as the namespace:name
> consistently appears on every relevant row.

 Looks like hbck2 *extraRegionsinMeta *is what you need here, and it should
work with hbase 2.1. It's not available on hbck2 1.0 release, but you can
build a new hbck2 jar out of current master branch, and that would give you
an hbck2 with extraRegionsinMeta. You can do it by cloning the repo below,
then do a *mvn install* from the main module:

https://github.com/apache/hbase-operator-tools.git

Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins <marc.hoppins@eset.sk>
escreveu:

> Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
>
> The UI page for the master shows NO report links/tabs.
>
> Hbase:meta has info for one table region which is NOT on a master. Is that
> correct? I would have expected all meta info to be stored on a master.
>
> Name                                                Region Server
>                                     Read Requests    Write Requests
> Num.Storefiles          MemSize         Locality
> hbase:meta,,1.1588230740        ba-wtmp05.asgardalfa.hq.com:16030
>  219,168,603     35,419  1 MB      4                             0 B
>              1.0
>
> -----Original Message-----
> From: Wellington Chevreuil <wellington.chevreuil@gmail.com>
> Sent: Tuesday, December 15, 2020 7:10 PM
> To: Hbase-User <user@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this may
> > be more suited to 'dev' list.
> >
> Welcome, this is the right channel for this kind of questions.
>
>
> > The solution offered by hbase-operator-tools  - extraRegionsinMeta -
> > offered hope.  Once again, however, another problem has surfaced: this
> > tools command for extra regions is incompatible with the hbase version
> > we are running.
> >
>  This command does not rely on any Master/RegionServer interface, so it
> should not have any incompatibility issues. It only uses public client API
> to cleanup meta table, so maybe you just need to build latest hbck2 master
> branch version? It would work for any hbase 2, in theory, it could even
> work with hbase 1, but that was never tested. Luckly, you got Stack's
> attention, so yeah, if you confirm on his previous questions we might be
> able to help with further directions.
>
> Em ter., 15 de dez. de 2020 às 17:23, Stack <stack@duboce.net> escreveu:
>
> > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <marc.hoppins@eset.sk>
> wrote:
> >
> > > Hi all,
> > >
> > > I am an HBASE newbie so I apologise if I am being repetitious.
> > >
> > > Apologies also if this is not the right group. Am not sure if this
> > > may be more suited to 'dev' list.  However,
> > >
> > > A problem question and a technical/wishlist question.
> > >
> > > Problem:
> > >
> > > I have inherited a problem with an HBASE table. The original issue
> > > may have erupted due to a network outage.  A table has 48 region in
> > transition
> > > operations, stuck that way for several weeks.  A previous attempt to
> > > fix things with hbck failed. An attempt to DISABLE then DROP the
> > > table also failed. The four or five attempts to work the table ALSO
> > > now had stuck procedures.  Subsequent DFS and ZOO operations left
> > > the situation where there was no data and no real table: just a 6K
> > > file in an empty
> > structure.
> > >
> > >
> > Which version of hbase, do you know Marc? (Look at the base of the
> > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> >
> > What is the name of the 6k file? (I am trying to understand what the
> > file you are referencing is).
> >
> > The table directory was removed from hdfs and zookeeper?
> >
> Yes.   The data had already been removed from the previous tech attempts
> to fix the issue.
>
> Thus, the HDFS 6K file structure contained no actual information, just
> empty files and directories.  Zkshell had been used to remove znodes.
>
>
> >
> > > When I got to the problem my knowledge of HBASE was nil. It is
> > > little better than that now but anyway...
> > >
> > > Fortunately for me this is a testing/dev cluster. The 'owner' was
> > > content that the table can be removed - and appeared to have already
> > > been
> > done...of
> > > a kind.
> > >
> > > Reading and reading of others' similar issues lead me to the point I
> > > also was going to clean the HDFS data and ZK data for this table.  I
> > > shut down HBASE, cleaned HDFS and ZK node data, deleted the
> > > masterprocwals and restarted HBASE.
> > >
> > > When all came up I was happy to see that the affected table appeared
> > > nowhere and that the procedures had all disappeared.
> > >
> > > However, when I hopped to hbase master, even though no table of that
> > > name existed, 48 regions were still in transition.  Further research
> > > steered
> > me
> > > toward hbase:meta and sure enough, the references to the RITs lived
> > happily
> > > among other data for other tables.
> > >
> > >
> > Do you have a 'Procedures and Locks' tab on your master home page? If
> > so, what does it report? Is there an "HBCK Report" tab? If so, what
> > does it show?
> >
>
> When I got to the problem procedures & locks showed one item for each
> problem on that table. After removal of masterprocwals they all
> disappeared. The only entry now is one normal-looking WAL log file entry
> whose timestamp changes according to operation time (currently 8.25am
> today).
>
> >
> > > The solution offered by hbase-operator-tools  - extraRegionsinMeta -
> > > offered hope.  Once again, however, another problem has surfaced:
> > > this tools command for extra regions is incompatible with the hbase
> > > version we are running.
> > >
> > > So...
> > >
> > > How can I remove the references to namespace:kaput_table from
> hbase:meta?
> > >
> > >
> > > Sounds like an hbase-2.1.x or a 2.0.x.
> >
> > Will wait on your answers to the above... It might be a crass delete
> > of each row from the hbase:meta table then restart (even then, if
> > procedures in the procedure store, you may have to clear it again as
> > you did above before the restart to purge the procedures as you don't
> > have tooling to do it from cmdline... do you have an 'hbck2 bypass
> --override'?).
> >
> A crass delete of each row would work for me if I could Identify how to
> remove the rows.  Sample data:-
>
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> 35925292c25898671e5a894ce387e167, NAME =>
> 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.',
> STARTKEY => '', ENDKEY => '0'}
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:seqnumDuringOpen, timestamp=1601269814633,
> value=\x00\x00\x00\x00\x00\x00\x008
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:server, timestamp=1601269814633, value=
> ba-wtmp04.asgardalfa.hq.com:16020
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:serverstartcode, timestamp=1601269814633, value=1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:sn, timestamp=1604388258050, value=ba-wtmp04.asgardalfa.hq.com
> ,16020,1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:state, timestamp=1604388258225, value=CLOSED
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> 787d1455b84f2d846ce6089392f01fd2, NAME =>
> 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.',
> STARTKEY => '0', ENDKEY => '1'}
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:seqnumDuringOpen, timestamp=1600780187722,
> value=\x00\x00\x00\x00\x00\x02^\xBB
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:server, timestamp=1600780187722, value=
> ba-wtmp08.asgardalfa.hq.com:16020
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:serverstartcode, timestamp=1600780187722, value=1600780162556
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:sn, timestamp=1600969938610, value=ba-wtmp07.asgardalfa.hq.com
> ,16020,1600936054386
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:state, timestamp=1600969938610, value=OPENING
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.',
> STARTKEY => '1', ENDKEY => '2'}
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:seqnumDuringOpen, timestamp=1600780186976,
> value=\x00\x00\x00\x00\x00\x02^\xA9
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:server, timestamp=1600780186976, value=
> dr1-wtmp02.asgardalfa.hq.com:16020
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:serverstartcode, timestamp=1600780186976, value=1600780163021
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:sn, timestamp=1601060563980, value=ba-wtmp05.asgardalfa.hq.com
> ,16020,1601049145467
>
>
> As I state below, a simple method to remove everything connected to
> alfa:rfilenameext would seem the easiest of tasks as the namespace:name
> consistently appears on every relevant row.
>
> >
> >
> > > Technical:
> > >
> > > Is there to be any implementation of such a fix within HBASe itself
> > > where table manipulation can be performed by Eg.,
> > >
> > > delete hbase:meta namespace
> > > delete hbase:meta  namespace:table
> > >
> > > or even
> > >
> > > scan hbase:meta filter = namespace:table | deleterow
> > >
> > > or some such?
> > >
> > >
> > Scan doesn't return result (unfortunately) so you can assign to a
> > shell variable; it dumps scan output2 on stdout/stderr.
> >
> > I'm sure there is a better way but something like below (don't laugh!
> > and I've not really tried it so be careful):
> >
> > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan
> > "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell >
> > /tmp/out.txt # Need to get the row only. Above got rows and columns.
> > We don't want to filter on column # because we do not know which
> > columns are still in hbase:meta. Need to filter on table # again to
> > cut the shell preamble and footer out. Unique the rows.
> > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> > /tmp/unique_rows.txt # Now you have rows to delete. For each in shell
> > do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt and per
> > for add .... deleteall " prefix and a "
> > suffix.... to make
> > # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> > hbase shell /tmp/delete_rows.txt.
> >
> > But before messing w/ the above, lets get the answers to above questions.
> > Thanks,
> > S
> >
> >
> >
> >
> >
> >
> >
> >
> > > Thanks
> > >
> > > M
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message