ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wong (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (RANGER-2899) Ranger Kudu plugin seems to hang when authorizing requests in certain environments
Date Wed, 08 Jul 2020 22:31:17 GMT

    [ https://issues.apache.org/jira/browse/RANGER-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154053#comment-17154053
] 

Andrew Wong edited comment on RANGER-2899 at 7/8/20, 10:30 PM:
---------------------------------------------------------------

[~abhayk] and I have been chatting about this. After updating our Kudu test, it seems that
we could avoid the hang by posting a policy to Ranger, which we hadn't done previously in
this particular test, which led to this hypothesis (unclear what the exact failure mechanism
is yet though):
{quote}Abhay Kulkarni: Does it mean that you have a service in Ranger Admin which contains
no policies to start with. If the service contains some (even one) policy, then the download
is successful and everything is hunky-dory?
 Andrew Wong: that appears to be the case – the only difference is the post to service/plugins/policies?deleteIfExists=true
{quote}
Cross-posting from KUDU-3154:

Some updates here, given that we have other Ranger tests that succeed in this environment,
I had a hunch that the issue was due to the fact that the test doesn't post any policies before
sending requests. So I updated the Ranger initialization with the following snippet in our
test:
{code:java}
  Status InitializeRanger() {
    ranger_.reset(new MiniRanger("127.0.0.1"));                                          
                                                                                         
                                                                                         
                                                                                         
        
    RETURN_NOT_OK(ranger_->Start());

    // BEGIN NEW LINES
    // Grant some policy to the user so the Ranger client policy refresher can
    // pick something up. In some environments the absense of policies can
    // cause the plugin to wait for a policy to appear.
    PolicyItem item;
    item.first.emplace_back("user");
    item.second.emplace_back(ActionPB::METADATA);
    AuthorizationPolicy policy;
    policy.databases.emplace_back("db");
    policy.tables.emplace_back("table");
    policy.items.emplace_back(std::move(item));
    RETURN_NOT_OK(ranger_->AddPolicy(std::move(policy)));
    // END NEW LINES

    RETURN_NOT_OK(ranger_->CreateClientConfig(test_dir_));
    client_.reset(new RangerClient(env_, metric_entity_));
    return client_->Start();
  }
{code}
Under the hood, the new lines are curling a policy to Ranger, as seen [here|https://github.com/apache/kudu/blob/master/src/kudu/ranger/mini_ranger.cc#L323].

And lo and behold, the test succeeds. This implies the differentiating factor is the existence
of at least one policy being posted to the {{service/plugins/policies?deleteIfExists=true}}
endpoint. I tried with a different user too; it doesn't seem to matter whether the user making
the plugin request in the test has a policy or not.

This environment is using Ubuntu 14.04.6 LTS and the following Java version for both the Ranger
server and plugin
{code:java}
java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1~14.04-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) {code}


was (Author: andrew.wong):
[~abhayk] and I have been chatting about this. After updating our Kudu test, it seems that
we could avoid the hang by posting a policy to Ranger, which we hadn't done previously in
this particular test, which led to this hypothesis (unclear what the exact failure mechanism
is yet though):
{quote}Abhay Kulkarni: Does it mean that you have a service in Ranger Admin which contains
no policies to start with. If the service contains some (even one) policy, then the download
is successful and everything is hunky-dory?
 Andrew Wong: that appears to be the case – the only difference is the post to service/plugins/policies?deleteIfExists=true
{quote}
Cross-posting from KUDU-3154:

Some updates here, given that we have other Ranger tests that succeed in this environment,
I had a hunch that the issue was due to the fact that the test doesn't post any policies before
sending requests. So I updated the Ranger initialization with the following snippet in our
test:
{code:java}
  Status InitializeRanger() {
    ranger_.reset(new MiniRanger("127.0.0.1"));                                          
                                                                                         
                                                                                         
                                                                                         
        
    RETURN_NOT_OK(ranger_->Start());
    // Grant some policy to the user so the Ranger client policy refresher can
    // pick something up. In some environments the absense of policies can
    // cause the plugin to wait for a policy to appear.
    PolicyItem item;
    item.first.emplace_back("user");
    item.second.emplace_back(ActionPB::METADATA);
    AuthorizationPolicy policy;
    policy.databases.emplace_back("db");
    policy.tables.emplace_back("table");
    policy.items.emplace_back(std::move(item));
    RETURN_NOT_OK(ranger_->AddPolicy(std::move(policy)));
    RETURN_NOT_OK(ranger_->CreateClientConfig(test_dir_));
    client_.reset(new RangerClient(env_, metric_entity_));
    return client_->Start();
  }
{code}
And lo and behold, the test succeeds. This implies the differentiating factor is the existence
of at least one policy being posted to the {{service/plugins/policies?deleteIfExists=true}}
endpoint. I tried with a different user too; it doesn't seem to matter whether the user making
the plugin request in the test has a policy or not.

This environment is using Ubuntu 14.04.6 LTS and the following Java version for both the Ranger
server and plugin
{code:java}
java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1~14.04-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) {code}

> Ranger Kudu plugin seems to hang when authorizing requests in certain environments
> ----------------------------------------------------------------------------------
>
>                 Key: RANGER-2899
>                 URL: https://issues.apache.org/jira/browse/RANGER-2899
>             Project: Ranger
>          Issue Type: Bug
>          Components: plugins
>    Affects Versions: 2.1.0
>            Reporter: Andrew Wong
>            Priority: Major
>         Attachments: jstack, kudu-3154_jstacks.txt
>
>
> Kudu's authorization provider is a Java process that communicates with the Kudu master
processes with IPC going over a pipe. The Java process enqueues requests to be authorized
by a ["kudu"-typed RangerBasePlugin|https://github.com/apache/kudu/blob/master/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ranger/authorization/RangerKuduAuthorizer.java].
> In some of our pre-commit tests that run on Ubuntu 14.04, the plugin appears to hang,
periodically refreshing privileges (we've seen the issue with a refresh interval of 200ms
and 500ms) until the request eventually times out on the Kudu side:
> {code:java}
> 2020-07-03 09:21:25.793 [DEBUG - main] (PolicyRefresher.java:154) Scheduled policyDownloadRefresher
to download policies every 500 milliseconds
> 2020-07-03 09:21:25.794 [INFO - main] (RangerKuduAuthorizer.java:92) Finished Ranger
Kudu plugin initialization
> 2020-07-03 09:21:25.814 [DEBUG - pool-3-thread-1] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.829 [DEBUG - pool-2-thread-1] (QueueUtil.java:58) Message being put
onto the queue
> 2020-07-03 09:21:25.830 [DEBUG - pool-3-thread-2] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.834 [DEBUG - pool-3-thread-3] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.838 [DEBUG - pool-3-thread-1] (QueueUtil.java:44) Message: org.apache.kudu.subprocess.InboundRequest@6a0874e7
has been taken from the queue
> 2020-07-03 09:21:25.841 [DEBUG - pool-4-thread-1] (QueueUtil.java:40) Message being taken
from the queue
> 2020-07-03 09:21:25.845 [DEBUG - pool-2-thread-1] (QueueUtil.java:62) Message: org.apache.kudu.subprocess.InboundRequest@6a0874e7
has been put on the queue
> 2020-07-03 09:21:25.866 [DEBUG - pool-3-thread-1] (ProtocolHandler.java:53) Unpacking
request
> 2020-07-03 09:21:25.869 [DEBUG - pool-3-thread-1] (ProtocolHandler.java:60) Executing
request
> 2020-07-03 09:21:26.012 [DEBUG - pool-3-thread-1] (RangerProtocolHandler.java:53) Executing
request
> 2020-07-03 09:21:26.021 [DEBUG - pool-3-thread-1] (Groups.java:312) GroupCacheLoader
- load.
> 2020-07-03 09:21:26.037 [DEBUG - pool-3-thread-1] (UserGroupInformation.java:1737) Failed
to get groups for user user by java.io.IOException: No groups found for user user
> 2020-07-03 09:21:26.040 [DEBUG - pool-3-thread-1] (RangerKuduAuthorizer.java:125) calling
plugin
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (PolicyRefresher.java:489)
==> PolicyRefresher(serviceName=kudu).loadRoles()
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:109)
==> RangerRolesProvider(serviceName= kudu serviceType= kudu).loadUserGroupRoles()
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:118)
In-Use memory: 14500632, Free memory:223526120
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerRolesProvider.java:171)
==> RangerRolesProvider(serviceName=kudu).loadUserGroupRolesFromAdmin()
> 2020-07-03 09:21:26.294 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerAdminRESTClient.java:204)
==> RangerAdminRESTClient.getRolesIfUpdated(-1, 1593768085212)
> 2020-07-03 09:21:26.295 [DEBUG - PolicyRefresher(serviceName=kudu)-12] (RangerAdminRESTClient.java:238)
Checking Roles updated as user : slave (auth:SIMPLE)
> Jul 03, 2020 9:21:26 AM com.sun.jersey.api.client.ClientResponse getEntity
> SEVERE: A message body reader for Java class java.lang.String, and Java type class java.lang.String,
and MIME media type application/octet-stream was not found
> Jul 03, 2020 9:21:26 AM com.sun.jersey.api.client.ClientResponse getEntity
> SEVERE: The registered message body readers compatible with the MIME media type are:
> */* ->
>   com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General
>   com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$General
>   com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$General 
> ... the rest of the logs seem to be Ranger refreshing policies until Kudu times out{code}
> The Ranger server is running on [commit f37f5407eee8d2627a4306a25938b151f8e2ba31|https://github.com/apache/ranger/commits/f37f5407eee8d2627a4306a25938b151f8e2ba31] and
the Ranger plugin version is 2.0.0. More logs are attached in KUDU-3154.
> Are there known issues with the plugin when refreshing the privilege interval at high
frequencies that can cause such hanging behavior?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message