hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wellington Chevreuil <wellington.chevre...@gmail.com>
Subject Re: Removal of table rows.
Date Thu, 17 Dec 2020 12:33:42 GMT
Yes, depending on hbade version, you would need a master restart to
effectively clean master cache.

On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <marc.hoppins@eset.sk> wrote:

> OK. I had a meandering circuit to get a version of operator-tools built
> and running.
>
> After running it on the meta table
>
> sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar extraRegionsInMeta
> alfa:rfilenameext --fix
>
> I have ended up with:-
>
> alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> column=info:state, timestamp=1608197544904, value=OPEN
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfiles column=table:state, timestamp=1602600776355, value=\x08\x00
>
> I still have one row referring to the missing table.  Do I need to restart
> hbase service to remove this or will it vanish at some time?
>
> HBASE on the master still shows the 48 regions in transition when I open
> the interface.  I assume this is because the service has not been restarted.
>
> M
>
> -----Original Message-----
> From: Wellington Chevreuil <wellington.chevreuil@gmail.com>
> Sent: Wednesday, December 16, 2020 3:39 PM
> To: Hbase-User <user@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > Do I build hbck2-tools on similar OS & java version?  I have been
> > informed we have 'maven' installed on one host on a cluster, which is
> > centos and
> > (probably) a different java, and the build itself is needed for Ubuntu16.
>
> I would stick to the same java major version. OS wise, this module doesn't
> rely on any OS native call, AFAIK, so should work fine between these
> different versions.
>
> Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins <marc.hoppins@eset.sk>
> escreveu:
>
> > Thanks. Once again, a newbie in this regard but,
> >
> > Do I build hbck2-tools on similar OS & java version?  I have been
> > informed we have 'maven' installed on one host on a cluster, which is
> > centos and
> > (probably) a different java, and the build itself is needed for Ubuntu16.
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <wellington.chevreuil@gmail.com>
> > Sent: Wednesday, December 16, 2020 10:08 AM
> > To: Hbase-User <user@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > >
> > > Hbase:meta has info for one table region which is NOT on a master.
> > > Is that correct? I would have expected all meta info to be stored on
> > > a
> > master.
> > >
> > meta table is an "hbase system table" that has info about which
> > regions are assigned to which region servers in your cluster. As any
> > user table, it also has a region, and this region must be hosted on a
> > region server in the cluster. Master is just responsible for the
> > coordination of regions and some other housekeeping actions.
> >
> > As I state below, a simple method to remove everything connected to
> > > alfa:rfilenameext would seem the easiest of tasks as the
> > > namespace:name consistently appears on every relevant row.
> >
> >  Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> > should work with hbase 2.1. It's not available on hbck2 1.0 release,
> > but you can build a new hbck2 jar out of current master branch, and
> > that would give you an hbck2 with extraRegionsinMeta. You can do it by
> > cloning the repo below, then do a *mvn install* from the main module:
> >
> > https://github.com/apache/hbase-operator-tools.git
> >
> > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > <marc.hoppins@eset.sk>
> > escreveu:
> >
> > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > >
> > > The UI page for the master shows NO report links/tabs.
> > >
> > > Hbase:meta has info for one table region which is NOT on a master.
> > > Is that correct? I would have expected all meta info to be stored on
> > > a
> > master.
> > >
> > > Name                                                Region Server
> > >                                     Read Requests    Write Requests
> > > Num.Storefiles          MemSize         Locality
> > > hbase:meta,,1.1588230740        ba-wtmp05.asgardalfa.hq.com:16030
> > >  219,168,603     35,419  1 MB      4                             0 B
> > >              1.0
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <wellington.chevreuil@gmail.com>
> > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > To: Hbase-User <user@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > >
> > > > Apologies also if this is not the right group. Am not sure if this
> > > > may be more suited to 'dev' list.
> > > >
> > > Welcome, this is the right channel for this kind of questions.
> > >
> > >
> > > > The solution offered by hbase-operator-tools  - extraRegionsinMeta
> > > > - offered hope.  Once again, however, another problem has surfaced:
> > > > this tools command for extra regions is incompatible with the
> > > > hbase version we are running.
> > > >
> > >  This command does not rely on any Master/RegionServer interface, so
> > > it should not have any incompatibility issues. It only uses public
> > > client API to cleanup meta table, so maybe you just need to build
> > > latest hbck2 master branch version? It would work for any hbase 2,
> > > in theory, it could even work with hbase 1, but that was never tested.
> > > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > > previous questions we might be able to help with further directions.
> > >
> > > Em ter., 15 de dez. de 2020 às 17:23, Stack <stack@duboce.net>
> escreveu:
> > >
> > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > <marc.hoppins@eset.sk>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > >
> > > > > Apologies also if this is not the right group. Am not sure if
> > > > > this may be more suited to 'dev' list.  However,
> > > > >
> > > > > A problem question and a technical/wishlist question.
> > > > >
> > > > > Problem:
> > > > >
> > > > > I have inherited a problem with an HBASE table. The original
> > > > > issue may have erupted due to a network outage.  A table has 48
> > > > > region in
> > > > transition
> > > > > operations, stuck that way for several weeks.  A previous
> > > > > attempt to fix things with hbck failed. An attempt to DISABLE
> > > > > then DROP the table also failed. The four or five attempts to
> > > > > work the table ALSO now had stuck procedures.  Subsequent DFS
> > > > > and ZOO operations left the situation where there was no data
> > > > > and no real table: just a 6K file in an empty
> > > > structure.
> > > > >
> > > > >
> > > > Which version of hbase, do you know Marc? (Look at the base of the
> > > > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > >
> > > > What is the name of the 6k file? (I am trying to understand what
> > > > the file you are referencing is).
> > > >
> > > > The table directory was removed from hdfs and zookeeper?
> > > >
> > > Yes.   The data had already been removed from the previous tech
> attempts
> > > to fix the issue.
> > >
> > > Thus, the HDFS 6K file structure contained no actual information,
> > > just empty files and directories.  Zkshell had been used to remove
> znodes.
> > >
> > >
> > > >
> > > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > > little better than that now but anyway...
> > > > >
> > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > was content that the table can be removed - and appeared to have
> > > > > already been
> > > > done...of
> > > > > a kind.
> > > > >
> > > > > Reading and reading of others' similar issues lead me to the
> > > > > point I also was going to clean the HDFS data and ZK data for
> > > > > this table.  I shut down HBASE, cleaned HDFS and ZK node data,
> > > > > deleted the masterprocwals and restarted HBASE.
> > > > >
> > > > > When all came up I was happy to see that the affected table
> > > > > appeared nowhere and that the procedures had all disappeared.
> > > > >
> > > > > However, when I hopped to hbase master, even though no table of
> > > > > that name existed, 48 regions were still in transition.  Further
> > > > > research steered
> > > > me
> > > > > toward hbase:meta and sure enough, the references to the RITs
> > > > > lived
> > > > happily
> > > > > among other data for other tables.
> > > > >
> > > > >
> > > > Do you have a 'Procedures and Locks' tab on your master home page?
> > > > If so, what does it report? Is there an "HBCK Report" tab? If so,
> > > > what does it show?
> > > >
> > >
> > > When I got to the problem procedures & locks showed one item for
> > > each problem on that table. After removal of masterprocwals they all
> > > disappeared. The only entry now is one normal-looking WAL log file
> > > entry whose timestamp changes according to operation time (currently
> > > 8.25am today).
> > >
> > > >
> > > > > The solution offered by hbase-operator-tools  -
> > > > > extraRegionsinMeta
> > > > > - offered hope.  Once again, however, another problem has surfaced:
> > > > > this tools command for extra regions is incompatible with the
> > > > > hbase version we are running.
> > > > >
> > > > > So...
> > > > >
> > > > > How can I remove the references to namespace:kaput_table from
> > > hbase:meta?
> > > > >
> > > > >
> > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > >
> > > > Will wait on your answers to the above... It might be a crass
> > > > delete of each row from the hbase:meta table then restart (even
> > > > then, if procedures in the procedure store, you may have to clear
> > > > it again as you did above before the restart to purge the
> > > > procedures as you don't have tooling to do it from cmdline... do
> > > > you have an 'hbck2 bypass
> > > --override'?).
> > > >
> > > A crass delete of each row would work for me if I could Identify how
> > > to remove the rows.  Sample data:-
> > >
> > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > value=\x08\x00
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > > 35925292c25898671e5a894ce387e167, NAME =>
> > > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > ,
> > > STARTKEY => '', ENDKEY => '0'}
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > value=\x00\x00\x00\x00\x00\x00\x008
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:server, timestamp=1601269814633, value=
> > > ba-wtmp04.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:serverstartcode, timestamp=1601269814633,
> > > value=1601061167123
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:sn, timestamp=1604388258050,
> > > value=ba-wtmp04.asgardalfa.hq.com
> > > ,16020,1601061167123
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > ',
> > > STARTKEY => '0', ENDKEY => '1'}
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:server, timestamp=1600780187722, value=
> > > ba-wtmp08.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:serverstartcode, timestamp=1600780187722,
> > > value=1600780162556
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:sn, timestamp=1600969938610,
> > > value=ba-wtmp07.asgardalfa.hq.com
> > > ,16020,1600936054386
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:state, timestamp=1600969938610, value=OPENING
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > ',
> > > STARTKEY => '1', ENDKEY => '2'}
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:server, timestamp=1600780186976, value=
> > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:serverstartcode, timestamp=1600780186976,
> > > value=1600780163021
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:sn, timestamp=1601060563980,
> > > value=ba-wtmp05.asgardalfa.hq.com
> > > ,16020,1601049145467
> > >
> > >
> > > As I state below, a simple method to remove everything connected to
> > > alfa:rfilenameext would seem the easiest of tasks as the
> > > namespace:name consistently appears on every relevant row.
> > >
> > > >
> > > >
> > > > > Technical:
> > > > >
> > > > > Is there to be any implementation of such a fix within HBASe
> > > > > itself where table manipulation can be performed by Eg.,
> > > > >
> > > > > delete hbase:meta namespace
> > > > > delete hbase:meta  namespace:table
> > > > >
> > > > > or even
> > > > >
> > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > >
> > > > > or some such?
> > > > >
> > > > >
> > > > Scan doesn't return result (unfortunately) so you can assign to a
> > > > shell variable; it dumps scan output2 on stdout/stderr.
> > > >
> > > > I'm sure there is a better way but something like below (don't laugh!
> > > > and I've not really tried it so be careful):
> > > >
> > > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo
> > > > 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> > > > shell > /tmp/out.txt # Need to get the row only. Above got rows and
> columns.
> > > > We don't want to filter on column # because we do not know which
> > > > columns are still in hbase:meta. Need to filter on table # again
> > > > to cut the shell preamble and footer out. Unique the rows.
> > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u
> > > > > /tmp/unique_rows.txt # Now you have rows to delete. For each in
> > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> > > > suffix.... to make
> > > > # a file with lines of deleteall "ROW1", etc. then pass it to the
> > shell:
> > > > hbase shell /tmp/delete_rows.txt.
> > > >
> > > > But before messing w/ the above, lets get the answers to above
> > questions.
> > > > Thanks,
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > Thanks
> > > > >
> > > > > M
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message