hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Austin Heyne <ahe...@ccri.com>
Subject Re: EMR Read Replica Metadata Table Name
Date Wed, 11 Jul 2018 15:10:33 GMT
To expand on this, I'm also having the inverse issue. I had to take down 
our main HBase today and now when I try to run hbck it is trying to look 
for the hbase:meta,,1 table on a region server that is serving a read 
replica metadata table and failing.

It seems like something is messed up on HBase knowing which metadata 
table to use when and where that metadata table is located. I image this 
is all state that should be maintained in zookeeper but I don't know 
where things are going wrong.

-Austin


On 07/10/2018 07:47 PM, Austin Heyne wrote:
> I currently have an EMR cluster that's running a continuous ingest. 
> I'd like to spin up read-only clusters with Spark and Zeppelin to 
> query with. I've gotten the replica up and running with Spark but when 
> an executor tries to query HBase it's throwing 
> NotServingRegionExceptions.
>
> """
> 18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10, 
> retries=35, started=38532 ms ago, cancelled=false, 
> msg=org.apache.hadoop.hbase.NotServingRegionException: Region 
> hbase:meta,,1 is not online on 
> ip-10-0-24-63.ec2.internal,16020,1531253339025
>     at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
>     at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
>     at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
>     at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>     at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>     at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> """
>
> Which makes sense because on an EMR read replica the metadata table 
> should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not 
> hbase:meta,,1. Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on 
> ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or 
> how to fix it?
>
> Thanks,
>

-- 
Austin L. Heyne


Mime
View raw message