To expand on this, I'm also having the inverse issue. I had to take down
our main HBase today and now when I try to run hbck it is trying to look
for the hbase:meta,,1 table on a region server that is serving a read
replica metadata table and failing.
It seems like something is messed up on HBase knowing which metadata
table to use when and where that metadata table is located. I image this
is all state that should be maintained in zookeeper but I don't know
where things are going wrong.
-Austin
On 07/10/2018 07:47 PM, Austin Heyne wrote:
> I currently have an EMR cluster that's running a continuous ingest.
> I'd like to spin up read-only clusters with Spark and Zeppelin to
> query with. I've gotten the replica up and running with Spark but when
> an executor tries to query HBase it's throwing
> NotServingRegionExceptions.
>
> """
> 18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10,
> retries=35, started=38532 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.NotServingRegionException: Region
> hbase:meta,,1 is not online on
> ip-10-0-24-63.ec2.internal,16020,1531253339025
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> """
>
> Which makes sense because on an EMR read replica the metadata table
> should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not
> hbase:meta,,1. Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on
> ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or
> how to fix it?
>
> Thanks,
>
--
Austin L. Heyne
|