hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amanda Moran <amanda.mo...@esgyn.com>
Subject Re: Issues with HBase Regions in Transition
Date Thu, 03 Dec 2015 18:26:18 GMT
Hi there Ted-

Just a quick updated.

SUCCESS! Actually just swapping out the files didn't work, BUT thanks to
this email my co-worker (Narendra) mentioned we would might need to remove
some of our custom settings as well. I changed the setting back to
default... and SUCCESS the upgrade was successful. I was then able to
reinstall Trafodion no issue.

Thanks alot for the help! You led us down the right path!

Amanda


On Wed, Dec 2, 2015 at 1:03 PM, Amanda Moran <amanda.moran@esgyn.com> wrote:

> Hi there Ted-
>
> Thanks for the info. Very interesting...I will be trying this for sure.
>
> Thanks alot, great information.
>
> Amanda
>
> On Wed, Dec 2, 2015 at 12:57 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> From log snippet:
>>
>> Class
>> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
>> not found
>>
>> I assume the above class is in the trafodion jar (just a guess)
>> This can explain why regions would be stuck in transition.
>>
>> For upgrade, consider replacing the trafodion jar instead of removing it.
>>
>> Cheers
>>
>> On Wed, Dec 2, 2015 at 12:51 PM, Amanda Moran <amanda.moran@esgyn.com>
>> wrote:
>>
>> > Hi there Ted-
>> >
>> > Thanks for response. Update: I copied my trafodion.jar files back to
>> > ../hbase/lib , did a restart and then hbase is back online... I then
>> > removed them again (just to make sure) did a restart .. and then its
>> back
>> > in the region in transition. Seems like we should be able to remove our
>> > files and do a restart and have hbase be back online... Is this
>> incorrect?
>> >
>> > Log snippet:
>> >
>> > 2015-12-02 19:48:41,614 ERROR
>> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
>> open
>> > of region=hbase:meta,,1.1588230740, starting to roll back the global
>> > memstore size.
>> > java.lang.IllegalStateException: Could not instantiate a region
>> instance.
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
>> >         at
>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>> >         at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >         at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >         at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> > Class
>> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
>> not
>> > found
>> >         at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
>> >         ... 10 more
>> > Caused by: java.lang.ClassNotFoundException: Class
>> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
>> not
>> > found
>> >         at
>> >
>> >
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
>> >         at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
>> >         ... 11 more
>> > 2015-12-02 19:48:41,627 INFO
>> > org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening
>> of
>> > region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY =>
'',
>> > ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
>> > expecting version 4
>> > 2015-12-02 19:48:41,673 INFO
>> > org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
>> > hbase:meta,,1.1588230740
>> > 2015-12-02 19:48:41,679 ERROR
>> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
>> open
>> > of region=hbase:meta,,1.1588230740, starting to roll back the global
>> > memstore size.
>> > java.lang.IllegalStateException: Could not instantiate a region
>> instance.
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
>> >         at
>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>> >         at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >         at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >         at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> > Class
>> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
>> not
>> > found
>> >         at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
>> >         ... 10 more
>> > Caused by: java.lang.ClassNotFoundException: Class
>> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
>> not
>> > found
>> >         at
>> >
>> >
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
>> >         at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
>> >         ... 11 more
>> >
>> >
>> > Let me know if you need more from the logs.
>> >
>> > THANKS!
>> >
>> > On Wed, Dec 2, 2015 at 12:37 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >
>> > > Do you mind pastebin snippet of region server log where the region
>> stuck
>> > in
>> > > transition was hosted ?
>> > >
>> > > This would give us some clue.
>> > >
>> > > Cheers
>> > >
>> > > On Wed, Dec 2, 2015 at 12:14 PM, Amanda Moran <amanda.moran@esgyn.com
>> >
>> > > wrote:
>> > >
>> > > > Hi there All-
>> > > >
>> > > > I apologize if this issue has been raised before... I have done
>> ALOT of
>> > > > googling on this issues... and finally decided to just write here
to
>> > the
>> > > > source :)
>> > > >
>> > > > Background:
>> > > >
>> > > > I work on the incubating Apache Trafodion project.
>> > > >
>> > > > I am currently using HBase 1.0 with Cloudera 5.4.5.
>> > > >
>> > > > I have tried many times to upgrade my cloudera cluster (to get the
>> > latest
>> > > > version of HBase) and have always failed due to my HBase master
>> failing
>> > > to
>> > > > come backup do to a Region suck in Transition. It is always the
>> > > hbase:meta.
>> > > >
>> > > > In the past I just assumed that the upgrade feature just didn't
>> > actually
>> > > > work... and so even though frustrating... I just let it go.
>> > > >
>> > > > But now I have come across this issue again, in a slightly different
>> > > way...
>> > > > and this makes me think it is an HBase problem.
>> > > >
>> > > > I am preparing for an upgrade, and because of this I decided to
>> remove
>> > > the
>> > > > "Trafodion".jar files that are located in the ../hbase/lib
>> directory. I
>> > > > thought this might be the cause of not being able to upgrade,
>> because I
>> > > was
>> > > > leaving the trafodion files in this directory. I removed them, then
>> > did a
>> > > > deploy client config (since some configs were stale), and then did
a
>> > > > restart. Now I have a region stuck in transition, even before the
>> > > upgrade!
>> > > >
>> > > > Everything I have seen on the web, has not worked in the past. How
>> do I
>> > > get
>> > > > pass this issue? Do I have no choice but to do a full
>> > uninstall/reinstall
>> > > > and lose my data?
>> > > >
>> > > > Thanks all for any help!
>> > > >
>> > > >
>> > > > --
>> > > > Thanks,
>> > > >
>> > > > Amanda Moran
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> >
>> > Amanda Moran
>> >
>>
>
>
>
> --
> Thanks,
>
> Amanda Moran
>



-- 
Thanks,

Amanda Moran

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message