hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
Date Mon, 11 Feb 2019 09:02:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21206?focusedWorklogId=196839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-196839
]

ASF GitHub Bot logged work on HIVE-21206:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 11/Feb/19 09:01
            Start Date: 11/Feb/19 09:01
    Worklog Time Spent: 10m 
      Work Description: sankarh commented on pull request #529: HIVE-21206: Bootstrap replication
is slow as it opens lot of metastore connections.
URL: https://github.com/apache/hive/pull/529
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 196839)
            Time Spent: 10m
    Remaining Estimate: 0h

> Bootstrap replication is slow as it opens lot of metastore connections.
> -----------------------------------------------------------------------
>
>                 Key: HIVE-21206
>                 URL: https://issues.apache.org/jira/browse/HIVE-21206
>             Project: Hive
>          Issue Type: Bug
>          Components: repl
>    Affects Versions: 4.0.0
>            Reporter: Sankar Hariappan
>            Assignee: Sankar Hariappan
>            Priority: Major
>              Labels: DR, pull-request-available, replication
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running slower compared
to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as below:
> ||    Hive2- Hive2    ||  Hive3 - Hive3   ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is causing slow
down.
> {code}
> 2019-02-08T12:28:30,174 INFO  [HiveServer2-Background-Pool: Thread-1134]: ql.Driver (:())
- Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO  [HiveServer2-Background-Pool: Thread-1134]: exec.Task (:())
- Loading data to table nondefault.nondefault_table1 from hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Trying to connect to metastore with URI thrift://ctr-e139-1542663976389-62755-01-000014.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - HMSC::open(): Could not find delegation token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Opened a connection to metastore, current connections: 4
> 2019-02-08T12:28:30,206 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.RetryingMetaStoreClient
(:()) - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
ugi=hive/ctr-e139-1542663976389-62755-01-000014.hwx.site@HWQE.HORTONWORKS.COM (auth:KERBEROS)
retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO  [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
(:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch, finalDestination=hiveServer2.async.multi_dest.batch.solr,
interval=01:00.002 minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO  [HiveServer2-Background-Pool: Thread-1134]: common.FileUtils
(FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_0000001
> 2019-02-08T12:28:31,245 INFO  [HiveServer2-Background-Pool: Thread-1134]: ql.Driver (:())
- Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Closed a connection to metastore, current connections: 3
> 2019-02-08T12:28:31,246 INFO  [HiveServer2-Background-Pool: Thread-1134]: exec.Task (:())
- Loading data to table nondefault.nondefault_table2 from hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Trying to connect to metastore with URI thrift://ctr-e139-1542663976389-62755-01-000014.hwx.site:9083
> 2019-02-08T12:28:31,327 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - HMSC::open(): Could not find delegation token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:31,336 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Opened a connection to metastore, current connections: 4
> 2019-02-08T12:28:31,337 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.HiveMetaStoreClient
(:()) - Connected to metastore.
> 2019-02-08T12:28:31,337 INFO  [HiveServer2-Background-Pool: Thread-1134]: metastore.RetryingMetaStoreClient
(:()) - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
ugi=hive/ctr-e139-1542663976389-62755-01-000014.hwx.site@HWQE.HORTONWORKS.COM (auth:KERBEROS)
retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:31,642 INFO  [HiveServer2-Background-Pool: Thread-1134]: common.FileUtils
(FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/base_0000001
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message