phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "William Yang (JIRA)" <>
Subject [jira] [Commented] (PHOENIX-3360) Secondary index configuration is wrong
Date Fri, 03 Feb 2017 04:00:53 GMT


William Yang commented on PHOENIX-3360:

I don't think that current patch has solved the problem because it is still setting the RPC
class in RegionServer level instead of region level. 

We've encountered this problem too, our resolution is simple. The only thing we need to do
is to set specified priority ONLY for index updates that originated from RS. We can do this
in IndexWriterUtils#getDefaultDelegateHTableFactory() and CoprocessorHTableFactory. Method
getDefaultDelegateHTableFactory() creates an htable factory to get all HTableInterface objects
to do index updates in current region server. If we can make these HTableInterface objects
have this RPC config, then all index update requests will have the right priority and will
not interfere with other RS-RS requests.

Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each data table
regions init, then each index region has its own CoprocessorHTableFactory object. We need
to do :
  * create a single HBase connection for current RS and this connection is configured with
this RPC class
  * all data table regions get HTableInterface from this connection instead of calling CoprocessorEnvironment#getTable()
before doing a batch mutations to their index regions.

You can create this connection in getDefaultDelegateHTableFactory() or just let all regions
share the same CoprocessorHTableFactory object. No matter which way you choose, the only target
here is to make sure all HTableInterface objects for updating index tables are created from
the specified connection. In this case, CoprocessorHTableFactory is no longer a suitable name
as we don't get table from coprocessor env, but from an explicit connection object. 

I've done this and it works fine.

> Secondary index configuration is wrong
> --------------------------------------
>                 Key: PHOENIX-3360
>                 URL:
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>            Assignee: Rajeshbabu Chintaguntla
>            Priority: Critical
>             Fix For: 4.10.0
>         Attachments: PHOENIX-3360.patch
> IndexRpcScheduler allocates some handler threads and uses a higher priority for RPCs.
The corresponding IndexRpcController is not used by default as it is, but used through ServerRpcControllerFactory
that we configure from Ambari by default which sets the priority of the outgoing RPCs to either
metadata priority, or the index priority.
> However, after reading code of IndexRpcController / ServerRpcController it seems that
the IndexRPCController DOES NOT look at whether the outgoing RPC is for an Index table or
not. It just sets ALL rpc priorities to be the index priority. The intention seems to be the
case that ONLY on servers, we configure ServerRpcControllerFactory, and with clients we NEVER
configure ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We configure
ServerRpcControllerFactory from Ambari, which in affect makes it so that ALL rpcs from Phoenix
are only handled by the index handlers by default. It means all deadlock cases are still there.

> The documentation in is also wrong
in this sense. It does not talk about server side / client side. Plus this way of configuring
different values is not how HBase configuration is deployed. We cannot have the configuration
show the ServerRpcControllerFactory even only for server nodes, because the clients running
on those nodes will also see the wrong values. 

This message was sent by Atlassian JIRA

View raw message