ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wong (Jira)" <j...@apache.org>
Subject [jira] [Commented] (RANGER-2899) Ranger Kudu plugin seems to hang when authorizing requests in certain environments
Date Wed, 08 Jul 2020 22:28:12 GMT

    [ https://issues.apache.org/jira/browse/RANGER-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154053#comment-17154053

Andrew Wong commented on RANGER-2899:

[~abhayk] and I have been chatting about this. After updating our Kudu test, it seems that
we could avoid the hang by posting a policy to Ranger, which we hadn't done previously in
this particular test, which led to this hypothesis (unclear what the exact failure mechanism
is yet though):
{quote}Abhay Kulkarni: Does it mean that you have a service in Ranger Admin which contains
no policies to start with. If the service contains some (even one) policy, then the download
is successful and everything is hunky-dory?
 Andrew Wong: that appears to be the case – the only difference is the post to service/plugins/policies?deleteIfExists=true
Cross-posting from KUDU-3154:

Some updates here, given that we have other Ranger tests that succeed in this environment,
I had a hunch that the issue was due to the fact that the test doesn't post any policies before
sending requests. So I updated the Ranger initialization with the following snippet in our
  Status InitializeRanger() {
    ranger_.reset(new MiniRanger(""));                                          
    // Grant some policy to the user so the Ranger client policy refresher can
    // pick something up. In some environments the absense of policies can
    // cause the plugin to wait for a policy to appear.
    PolicyItem item;
    AuthorizationPolicy policy;
    client_.reset(new RangerClient(env_, metric_entity_));
    return client_->Start();
And lo and behold, the test succeeds. This implies the differentiating factor is the existence
of at least one policy being posted to the {{service/plugins/policies?deleteIfExists=true}}
endpoint. I tried with a different user too; it doesn't seem to matter whether the user making
the plugin request in the test has a policy or not.

This environment is using Ubuntu 14.04.6 LTS and the following Java version for both the Ranger
server and plugin
java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1~14.04-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) {code}

> Ranger Kudu plugin seems to hang when authorizing requests in certain environments
> ----------------------------------------------------------------------------------
>                 Key: RANGER-2899
>                 URL: https://issues.apache.org/jira/browse/RANGER-2899
>             Project: Ranger
>          Issue Type: Bug
>          Components: plugins
>    Affects Versions: 2.1.0
>            Reporter: Andrew Wong
>            Priority: Major
>         Attachments: jstack, kudu-3154_jstacks.txt
> Kudu's authorization provider is a Java process that communicates with the Kudu master
processes with IPC going over a pipe. The Java process enqueues requests to be authorized
by a ["kudu"-typed RangerBasePlugin|https://github.com/apache/kudu/blob/master/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ranger/authorization/RangerKuduAuthorizer.java].
> In some of our pre-commit tests that run on Ubuntu 14.04, the plugin appears to hang,
periodically refreshing privileges (we've seen the issue with a refresh interval of 200ms
and 500ms) until the request eventually times out on the Kudu side:
> {code:java}
> 2020-07-03 09:21:25.793 [DEBUG - main] (PolicyRefresher.java:154) Scheduled policyDownloadRefresher
to download policies every 500 milliseconds
> 2020-07-03 09:21:25.794 [INFO - main] (RangerKuduAuthorizer.java:92) Finished Ranger
Kudu plugin initialization
> 2020-07-03 09:21:25.814 [DEBUG - pool-3-thread-1] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.829 [DEBUG - pool-2-thread-1] (QueueUtil.java:58) Message being put
onto the queue
> 2020-07-03 09:21:25.830 [DEBUG - pool-3-thread-2] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.834 [DEBUG - pool-3-thread-3] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.838 [DEBUG - pool-3-thread-1] (QueueUtil.java:44) Message: org.apache.kudu.subprocess.InboundRequest@6a0874e7
has been taken from the queue
> 2020-07-03 09:21:25.841 [DEBUG - pool-4-thread-1] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.845 [DEBUG - pool-2-thread-1] (QueueUtil.java:62) Message: org.apache.kudu.subprocess.InboundRequest@6a0874e7
has been put on the queue
> 2020-07-03 09:21:25.866 [DEBUG - pool-3-thread-1] (ProtocolHandler.java:53) Unpacking
> 2020-07-03 09:21:25.869 [DEBUG - pool-3-thread-1] (ProtocolHandler.java:60) Executing
> 2020-07-03 09:21:26.012 [DEBUG - pool-3-thread-1] (RangerProtocolHandler.java:53) Executing
> 2020-07-03 09:21:26.021 [DEBUG - pool-3-thread-1] (Groups.java:312) GroupCacheLoader
- load.
> 2020-07-03 09:21:26.037 [DEBUG - pool-3-thread-1] (UserGroupInformation.java:1737) Failed
to get groups for user user by java.io.IOException: No groups found for user user
> 2020-07-03 09:21:26.040 [DEBUG - pool-3-thread-1] (RangerKuduAuthorizer.java:125) calling
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (PolicyRefresher.java:489)
==> PolicyRefresher(serviceName=kudu).loadRoles()
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:109)
==> RangerRolesProvider(serviceName= kudu serviceType= kudu).loadUserGroupRoles()
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:118)
In-Use memory: 14500632, Free memory:223526120
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:171)
==> RangerRolesProvider(serviceName=kudu).loadUserGroupRolesFromAdmin()
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerAdminRESTClient.java:204)
==> RangerAdminRESTClient.getRolesIfUpdated(-1, 1593768085212)
> 2020-07-03 09:21:26.295 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerAdminRESTClient.java:238)
Checking Roles updated as user : slave (auth:SIMPLE)
> Jul 03, 2020 9:21:26 AM com.sun.jersey.api.client.ClientResponse getEntity
> SEVERE: A message body reader for Java class java.lang.String, and Java type class java.lang.String,
and MIME media type application/octet-stream was not found
> Jul 03, 2020 9:21:26 AM com.sun.jersey.api.client.ClientResponse getEntity
> SEVERE: The registered message body readers compatible with the MIME media type are:
> */* ->
>   com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General
>   com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$General
>   com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$General 
> ... the rest of the logs seem to be Ranger refreshing policies until Kudu times out{code}
> The Ranger server is running on [commit f37f5407eee8d2627a4306a25938b151f8e2ba31|https://github.com/apache/ranger/commits/f37f5407eee8d2627a4306a25938b151f8e2ba31] and
the Ranger plugin version is 2.0.0. More logs are attached in KUDU-3154.
> Are there known issues with the plugin when refreshing the privilege interval at high
frequencies that can cause such hanging behavior?

This message was sent by Atlassian Jira

View raw message