hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "BELUGA BEHR (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-20844) Cache Instances of LockManager in DummyTxnManager
Date Wed, 31 Oct 2018 14:37:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

BELUGA BEHR updated HIVE-20844:
-------------------------------
    Summary: Cache Instances of LockManager in DummyTxnManager  (was: Cache Instances of CacheManager
in DummyTxnManager)

> Cache Instances of LockManager in DummyTxnManager
> -------------------------------------------------
>
>                 Key: HIVE-20844
>                 URL: https://issues.apache.org/jira/browse/HIVE-20844
>             Project: Hive
>          Issue Type: Improvement
>          Components: HiveServer2, Locking
>    Affects Versions: 2.3.2, 3.1.0, 4.0.0
>            Reporter: BELUGA BEHR
>            Priority: Major
>
> I noticed that the {{DummyTxnManager}} class instantiates quite a few instances of {{ZooKeeperHiveLockManager}}.
The ZooKeeper LM creates a connection to ZK for each instance created.  It also does some
initialization steps that are almost always just noise and pressure on ZooKeeper because it
has already been initialized and the steps are therefore NOOPs.  {{ZooKeeperHiveLockManager}}
should be a singleton class with one long-lived connection to the ZooKeeper service. Perhaps
the {{HiveLockManager}} interface could have a {{isSingleton()}} method which indicates that
the LM should only be instantiated once and cached for subsequent sessions.
>  
> {code:java}
> 2018-05-14 22:45:30,574  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1252389]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:27,865  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1252671]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:37,552  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1252686]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:49,046  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1252736]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:50,664  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1252742]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:00:54,314  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1253479]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:17:26,867  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1254180]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:24:25,426  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool:
Thread-1255493]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> {code}
> {code:java|title=DummyTxnManager.java}
> @Override
>   public HiveLockManager getLockManager() throws LockException {
>     if (lockMgr == null) {
>       boolean supportConcurrency =
>           conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
>       if (supportConcurrency) {
>         String lockMgrName =
>             conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER);
>         if ((lockMgrName == null) || (lockMgrName.isEmpty())) {
>           throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg());
>         }
>         try {
>          // CACHE LM HERE
>           LOG.info("Creating lock manager of type " + lockMgrName);
>           lockMgr = (HiveLockManager)ReflectionUtils.newInstance(
>               conf.getClassByName(lockMgrName), conf);
>           lockManagerCtx = new HiveLockManagerCtx(conf);
>           lockMgr.setContext(lockManagerCtx);
>         } catch (Exception e) {
> ...
> {code}
> [https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java]
>  {code:java|title=ZooKeeperHiveLockManager Initialization}
>     try {
>       curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>       parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>       try{
>         curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" +  parent,
new byte[0]);
>       } catch (Exception e) {
>         // ignore if the parent already exists
>         if (!(e instanceof KeeperException) || ((KeeperException)e).code() != KeeperException.Code.NODEEXISTS)
{
>           LOG.warn("Unexpected ZK exception when creating parent node /" + parent, e);
>         }
>       }
> {code}
>  
> https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L96-L106



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message