ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhan <>
Subject Re: Review Request 70078: RANGER-2341: Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine
Date Wed, 06 Mar 2019 08:35:09 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70078/#review213467
-----------------------------------------------------------


Ship it!




Ship It!

- madhan


On March 6, 2019, 4:29 a.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70078/
> -----------------------------------------------------------
> 
> (Updated March 6, 2019, 4:29 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Ramesh Mani, Sailaja Polavarapu, and Velmurugan
Periasamy.
> 
> 
> Bugs: RANGER-2341
>     https://issues.apache.org/jira/browse/RANGER-2341
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Optimize policy engine construction by applying only the changes to existing policy-engine
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java
dddfbc7fe 
>   agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
2a0797c92 
>   agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicy.java 3bafd5c0f

>   agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicyDelta.java PRE-CREATION

>   agents-common/src/main/java/org/apache/ranger/plugin/model/validation/RangerServiceDefHelper.java
6df5d8d00 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngine.java
63fcbd095 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineCache.java
8642dbee4 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
b29f15289 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
5498545a5 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
e5c8d0cc4 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java
289ec9bf2 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyresourcematcher/RangerDefaultPolicyResourceMatcher.java
8d35319ee 
>   agents-common/src/main/java/org/apache/ranger/plugin/service/RangerAuthContext.java
9ae334898 
>   agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
1c870f7b9 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 01ce9b2e4

>   agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java b5b4f1636

>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPolicyDeltaUtil.java
PRE-CREATION 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTClient.java f592ed4e7

>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTUtils.java f9ef1d301

>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java a2d52a049

>   agents-common/src/main/java/org/apache/ranger/plugin/util/ServicePolicies.java 664523e55

>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java
9d9be6c98 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_add.json
PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_delete.json
PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_update.json
PRE-CREATION 
>   knox-agent/src/main/java/org/apache/ranger/admin/client/RangerAdminJersey2RESTClient.java
d856f898b 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql b46a48155 
>   security-admin/db/mysql/patches/038-add-policy-change-log-table.sql PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 35c70c7f4 
>   security-admin/db/oracle/patches/038-add-policy-change-log-table.sql PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql dfa8c829c

>   security-admin/db/postgres/patches/038-add-policy-change-log-table.sql PRE-CREATION

>   security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 81c6172a6

>   security-admin/db/sqlanywhere/patches/038-add-policy-change-log-table.sql PRE-CREATION

>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 845e0891c

>   security-admin/db/sqlserver/patches/038-add-policy-change-log-table.sql PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java e1b244d45 
>   security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java
072495212 
>   security-admin/src/main/java/org/apache/ranger/common/db/JPABeanCallbacks.java 70ad44b48

>   security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java be1492282

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java a79ba7c77 
>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyChangeLog.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java d7089275b 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 0281c94a5 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java a43d076fd 
>   security-admin/src/main/java/org/apache/ranger/service/RangerSecurityZoneService.java
ab8931910 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml a6606010c 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java bf19efd31

>   security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java a1b0e45d5 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java ef4865afd

> 
> 
> Diff: https://reviews.apache.org/r/70078/diff/3/
> 
> 
> Testing
> -------
> 
> - Successfully executed all tests. 
> - Ensured that ranger admin and plugin functionality is indentical with or without policy
deltas enabled.
> - Ran performance test to ensure that JVM Garbage collection behavior is positively impacted
(No Full GC on ranger-admin). 
> - Noticed the policy-engine building time reduced by 80% with ~1800 large policies, with
policy-engine rebuilding within a second in the - plugin when a single policy was updated.

> - With ~3000 large policies, the policy-engine was built in about 2 seconds as compared
to 10 seconds without incremental policy updates.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message