sentry-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Binglin Chang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SENTRY-705) Possible memory leak of HiveAuthzBinding.authzBindingMap
Date Sat, 09 May 2015 16:47:00 GMT

    [ https://issues.apache.org/jira/browse/SENTRY-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536756#comment-14536756
] 

Binglin Chang commented on SENTRY-705:
--------------------------------------

Add some logging and found the root cause. HiveAuthzBinding.set is called twice when compile(each
with different queryId), but in execute, clear can only remove the last added object in authzBindingMap.
so each successful query will end up add a HiveAuthzBinding object to authzBindingMap, never
got freed.
{noformat}
2015-05-10 00:13:49,414 WARN [HiveServer2-Handler-Pool: Thread-28]: authz.HiveAuthzBinding
(HiveAuthzBinding.java:set(157)) - authzBindingMap.set 3acea9e4-a715-4890-b051-b473f242a1ff_3
java.lang.Exception
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.set(HiveAuthzBinding.java:157)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.authorizeWithHiveBindings(HiveAuthzBindingHook.java:527)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:326)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
......
2015-05-10 00:13:49,415 WARN [HiveServer2-Handler-Pool: Thread-28]: authz.HiveAuthzBinding
(HiveAuthzBinding.java:set(157)) - authzBindingMap.set 3acea9e4-a715-4890-b051-b473f242a1ff_4
java.lang.Exception
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.set(HiveAuthzBinding.java:157)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:343)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
......
2015-05-10 00:13:49,424 INFO [HiveServer2-Background-Pool: Thread-50]: ql.Driver (Driver.java:execute(1206))
- Starting command: select 1+1
2015-05-10 00:13:49,424 WARN [HiveServer2-Background-Pool: Thread-50]: authz.HiveAuthzBinding
(HiveAuthzBinding.java:clear(169)) - authzBindingMap.remove 3acea9e4-a715-4890-b051-b473f242a1ff_4
{noformat}

> Possible memory leak of HiveAuthzBinding.authzBindingMap
> --------------------------------------------------------
>
>                 Key: SENTRY-705
>                 URL: https://issues.apache.org/jira/browse/SENTRY-705
>             Project: Sentry
>          Issue Type: Bug
>    Affects Versions: 1.4.0
>            Reporter: Binglin Chang
>
> We are using hive0.13 with sentry, version cdh5.2.0, noticed that hiveserver uses lot
of memory and doesn't get freed.
> Memory profiling showed that there are lots(millions) of HiveConf and HiveAuthzConf instances
hold by HiveAuthzBinding.authzBindingMap
> Look at the code, related call order:
> Driver.compile
>    HiveAuthzBindingHook.preAnalyze
>    HiveAuthzBindingHook.postAnalyze
>       hiveAuthzBinding.set
> Driver.execute
>    HiveAuthzBindingPreExecHook.run
>       hiveAuthzBinding.clear
> Between Driver.compile and Driver.execute, there are too many logic(which sentry can
not control), not sure hiveAuthzBinding.clear is guaranteed to be called



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message