[ https://issues.apache.org/jira/browse/RANGER-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153236#comment-17153236 ] Andrew Wong commented on RANGER-2899: ------------------------------------- [~abhayk] I rebuilt our test with a policy poll interval of 30s and it resulted in the same result, with seemingly the same [^jstack] {code:java} "policyDownloadTimer" #14 daemon prio=5 os_prio=0 tid=0x00007f5d7c992800 nid=0x4d7cf in Object.wait() [0x00007f5d28601000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006c000c288> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000006c000c288> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "PolicyRefresher(serviceName=kudu)-12" #12 daemon prio=5 os_prio=0 tid=0x00007f5d7c988000 nid=0x4d7ce waiting on condition [0x00007f5d28702000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c000c4a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:206) {code} > Ranger Kudu plugin seems to hang when authorizing requests in certain environments > ---------------------------------------------------------------------------------- > > Key: RANGER-2899 > URL: https://issues.apache.org/jira/browse/RANGER-2899 > Project: Ranger > Issue Type: Bug > Components: plugins > Affects Versions: 2.1.0 > Reporter: Andrew Wong > Priority: Major > Attachments: jstack, kudu-3154_jstacks.txt > > > Kudu's authorization provider is a Java process that communicates with the Kudu master processes with IPC going over a pipe. The Java process enqueues requests to be authorized by a ["kudu"-typed RangerBasePlugin|https://github.com/apache/kudu/blob/master/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ranger/authorization/RangerKuduAuthorizer.java]. > In some of our pre-commit tests that run on Ubuntu 14.04, the plugin appears to hang, periodically refreshing privileges (we've seen the issue with a refresh interval of 200ms and 500ms) until the request eventually times out on the Kudu side: > {code:java} > 2020-07-03 09:21:25.793 [DEBUG - main] (PolicyRefresher.java:154) Scheduled policyDownloadRefresher to download policies every 500 milliseconds > 2020-07-03 09:21:25.794 [INFO - main] (RangerKuduAuthorizer.java:92) Finished Ranger Kudu plugin initialization > 2020-07-03 09:21:25.814 [DEBUG - pool-3-thread-1] (QueueUtil.java:40) Message being taken from the queue > 2020-07-03 09:21:25.829 [DEBUG - pool-2-thread-1] (QueueUtil.java:58) Message being put onto the queue > 2020-07-03 09:21:25.830 [DEBUG - pool-3-thread-2] (QueueUtil.java:40) Message being taken from the queue > 2020-07-03 09:21:25.834 [DEBUG - pool-3-thread-3] (QueueUtil.java:40) Message being taken from the queue > 2020-07-03 09:21:25.838 [DEBUG - pool-3-thread-1] (QueueUtil.java:44) Message: org.apache.kudu.subprocess.InboundRequest@6a0874e7 has been taken from the queue > 2020-07-03 09:21:25.841 [DEBUG - pool-4-thread-1] (QueueUtil.java:40) Message being taken from the queue > 2020-07-03 09:21:25.845 [DEBUG - pool-2-thread-1] (QueueUtil.java:62) Message: org.apache.kudu.subprocess.InboundRequest@6a0874e7 has been put on the queue > 2020-07-03 09:21:25.866 [DEBUG - pool-3-thread-1] (ProtocolHandler.java:53) Unpacking request > 2020-07-03 09:21:25.869 [DEBUG - pool-3-thread-1] (ProtocolHandler.java:60) Executing request > 2020-07-03 09:21:26.012 [DEBUG - pool-3-thread-1] (RangerProtocolHandler.java:53) Executing request > 2020-07-03 09:21:26.021 [DEBUG - pool-3-thread-1] (Groups.java:312) GroupCacheLoader - load. > 2020-07-03 09:21:26.037 [DEBUG - pool-3-thread-1] (UserGroupInformation.java:1737) Failed to get groups for user user by java.io.IOException: No groups found for user user > 2020-07-03 09:21:26.040 [DEBUG - pool-3-thread-1] (RangerKuduAuthorizer.java:125) calling plugin > 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (PolicyRefresher.java:489) ==> PolicyRefresher(serviceName=kudu).loadRoles() > 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:109) ==> RangerRolesProvider(serviceName= kudu serviceType= kudu).loadUserGroupRoles() > 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:118) In-Use memory: 14500632, Free memory:223526120 > 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:171) ==> RangerRolesProvider(serviceName=kudu).loadUserGroupRolesFromAdmin() > 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerAdminRESTClient.java:204) ==> RangerAdminRESTClient.getRolesIfUpdated(-1, 1593768085212) > 2020-07-03 09:21:26.295 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerAdminRESTClient.java:238) Checking Roles updated as user : slave (auth:SIMPLE) > Jul 03, 2020 9:21:26 AM com.sun.jersey.api.client.ClientResponse getEntity > SEVERE: A message body reader for Java class java.lang.String, and Java type class java.lang.String, and MIME media type application/octet-stream was not found > Jul 03, 2020 9:21:26 AM com.sun.jersey.api.client.ClientResponse getEntity > SEVERE: The registered message body readers compatible with the MIME media type are: > */* -> > com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General > com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$General > com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$General > ... the rest of the logs seem to be Ranger refreshing policies until Kudu times out{code} > The Ranger server is running on [commit f37f5407eee8d2627a4306a25938b151f8e2ba31|https://github.com/apache/ranger/commits/f37f5407eee8d2627a4306a25938b151f8e2ba31] and the Ranger plugin version is 2.0.0. More logs are attached in KUDU-3154. > Are there known issues with the plugin when refreshing the privilege interval at high frequencies that can cause such hanging behavior? -- This message was sent by Atlassian Jira (v8.3.4#803005)