hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amine ZITOUNI (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20600) Metastore connection leak
Date Tue, 25 Aug 2020 16:30:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184177#comment-17184177
] 

Amine ZITOUNI commented on HIVE-20600:
--------------------------------------

Hi,

I reproduce the issue  with  HiveServer2 & HiveServer2 Interactive (LLAP).

The Hive stack version is 3.1.0 (used via Hortonworks HDP 3.1.4.0-315).

 

I run 8 queries per minute. The number of connections increases each time a query is launched.

Netstat shows a thousands of connection from HS2 to Hive metastore with CLOSE_WAIT status.

 

After time, HS2 is unresponsive and should be restarted.

HiveServer2 Heap Size is 16GB

Metastore Heap Size is 16GB

> Metastore connection leak
> -------------------------
>
>                 Key: HIVE-20600
>                 URL: https://issues.apache.org/jira/browse/HIVE-20600
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 2.3.3
>            Reporter: Damon Cortesi
>            Priority: Major
>         Attachments: HIVE-20600.patch, consume_threads.py
>
>
> Within the execute method of HiveServer2, there appears to be a connection leak. With
fairly straightforward series of INSERT statements, the connection count in the logs continues
to increase over time. Under certain loads, this can also consume all underlying threads of
the Hive metastore and result in HS2 becoming unresponsive to new connections.
> The log below is the result of some python code executing a single insert statement,
and then looping through a series of 10 more insert statements. We can see there's one dangling
connection left open after each execution leaving us with 12 open connections (11 from the
execute statements + 1 from HS2 startup).
> {code}
> 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore (HiveMetaStoreClient.java:open(481))
- Opened a connection to metastore, current connections: 1
>  2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b HiveServer2-Handler-Pool:
Thread-31([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection
to metastore, current connections: 2
>  2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
1
>  2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
2
>  2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
1
>  2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
2
>  2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
1
>  2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
2
> ...
>  2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b HiveServer2-Handler-Pool:
Thread-31([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection
to metastore, current connections: 12
>  2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
11
>  2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
12
>  2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
11
>  2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
12
>  2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
11
>  2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
12
>  2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections:
13
>  2018-09-19T17:15:21,542 INFO [HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections:
12
> {code}
> Attached is a simple [impyla|https://github.com/cloudera/impyla] script that triggers
the condition.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message