hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vaibhav Gumashta (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-11499) Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs
Date Fri, 07 Aug 2015 21:21:45 GMT

     [ https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vaibhav Gumashta updated HIVE-11499:
------------------------------------
    Description: When UDFs are used, we create a new classloader to add the UDF jar. Similar
to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161).
JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115).
Until we call  NucleusContext#close, the classloader cache is not cleared. In case of UDFs
this can lead to permgen leak, as shown in the attached screenshot, where NucleusContext holds
on to several URLClassloader objects.  (was: When UDFs are used, we create a new classloader
to add the UDF jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus
caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161).
JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115).
Until we call  NucleusContext#close, the classloader cache is not cleared. In case of UDFs
this can lead as shows in the attached screenshot, where NucleusContext holds on to several
URLClassloader objects.)

> Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with
UDFs
> --------------------------------------------------------------------------------------------
>
>                 Key: HIVE-11499
>                 URL: https://issues.apache.org/jira/browse/HIVE-11499
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, Metastore
>    Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>         Attachments: HS2-NucleusCache-Leak.tiff
>
>
> When UDFs are used, we create a new classloader to add the UDF jar. Similar to what hadoop's
reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161).
JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115).
Until we call  NucleusContext#close, the classloader cache is not cleared. In case of UDFs
this can lead to permgen leak, as shown in the attached screenshot, where NucleusContext holds
on to several URLClassloader objects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message