ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Don Bosco Durai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (RANGER-2341) Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine
Date Fri, 01 Mar 2019 02:16:00 GMT

    [ https://issues.apache.org/jira/browse/RANGER-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781211#comment-16781211
] 

Don Bosco Durai commented on RANGER-2341:
-----------------------------------------

[~abhayk] this would be a good feature. Few questions...

> Cache management in ranger-admin is enhanced to use this table to figure out changes
using a previously known version number (provided by module requesting updated policies).
Seems more like more like redo logs in database, which I feel is a good approach.

> Backward compatibility is maintained with older plugins by adding another parameter to
REST API for downloading policies.
Should we do the other way? New plugins should pass the addition param, so that older plugins
will work without change?
 
> Policy deltas are disabled by default. 
I feel, we should enable this by default. This is a good feature and let the plugins decide
whether to use or not.

> Policy delta table is cleared of records older than a week on restart of ranger-admin.
I not sure whether restart should be the trigger, but might be okay for now till have an inbuilt
scheduler. I assume, we will make the the retention period configurable.



> Support for Incremental policy updates to improve performance of ranger-admin and plugins
by optimal building of policy-engine
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: RANGER-2341
>                 URL: https://issues.apache.org/jira/browse/RANGER-2341
>             Project: Ranger
>          Issue Type: Improvement
>          Components: Ranger
>    Affects Versions: master
>            Reporter: Abhay Kulkarni
>            Assignee: Abhay Kulkarni
>            Priority: Major
>             Fix For: master
>
>
> Requirements:
> Currently, every change to any policy causes rebuilding of policy-engine from scratch.
There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large demand on JVM memory system resulting in frequent garbage collection and pauses
of JVM.
> It will be more optimal to communicate only the changes and apply them to existing policy-engine.
> Design notes:
> Policy changes are logged into a new database table.
> Cache management in ranger-admin is enhanced to use this table to figure out changes
using a previously known version number (provided by module requesting updated policies).
> Policy engine supports update operation that accepts policy-deltas and returns a new
policy engine with deltas applied.
> Resource Trie structures are copied from older policy-engine selectively, and not rebuilt
from scratch.
> Backward compatibility is maintained with older plugins by adding another parameter to
REST API for downloading policies.
> Ranger admin as well as component plugins may be configured to optionally use policy
deltas for its internal policy-engines. Policy deltas are disabled by default. In ranger-admin,
policy-deltas are enabled in the ranger-admin by setting configuration variable 'ranger.admin.supports.policy.deltas'
to true. In individual plugins, policy-deltas are enabled by setting configuration variable
'ranger.plugin.<service-type>.policy.rest.supports.policy.deltas' to "true".
> Policy delta table is cleared of records older than a week on restart of ranger-admin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message