ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhay Kulkarni <akulka...@hortonworks.com>
Subject Re: Review Request 70078: RANGER-2341: Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine
Date Mon, 04 Mar 2019 01:14:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70078/
-----------------------------------------------------------

(Updated March 4, 2019, 1:14 a.m.)


Review request for ranger, Madhan Neethiraj, Ramesh Mani, Sailaja Polavarapu, and Velmurugan
Periasamy.


Changes
-------

Addressed review comments


Bugs: RANGER-2341
    https://issues.apache.org/jira/browse/RANGER-2341


Repository: ranger


Description
-------

Optimize policy engine construction by applying only the changes to existing policy-engine


Diffs (updated)
-----

  agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java dddfbc7fe

  agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
2a0797c92 
  agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicy.java 3bafd5c0f 
  agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicyDelta.java PRE-CREATION

  agents-common/src/main/java/org/apache/ranger/plugin/model/validation/RangerServiceDefHelper.java
6df5d8d00 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngine.java
63fcbd095 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineCache.java
8642dbee4 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
b29f15289 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
5498545a5 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
e5c8d0cc4 
  agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java
289ec9bf2 
  agents-common/src/main/java/org/apache/ranger/plugin/policyresourcematcher/RangerDefaultPolicyResourceMatcher.java
8d35319ee 
  agents-common/src/main/java/org/apache/ranger/plugin/service/RangerAuthContext.java 9ae334898

  agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java 1c870f7b9

  agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 01ce9b2e4 
  agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java b5b4f1636

  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPolicyDeltaUtil.java PRE-CREATION

  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTClient.java f592ed4e7

  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTUtils.java f9ef1d301

  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java a2d52a049

  agents-common/src/main/java/org/apache/ranger/plugin/util/ServicePolicies.java 664523e55

  agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java
9d9be6c98 
  agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_add.json
PRE-CREATION 
  agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_delete.json
PRE-CREATION 
  agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_update.json
PRE-CREATION 
  knox-agent/src/main/java/org/apache/ranger/admin/client/RangerAdminJersey2RESTClient.java
d856f898b 
  security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql b46a48155 
  security-admin/db/mysql/patches/038-add-policy-change-log-table.sql PRE-CREATION 
  security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 35c70c7f4 
  security-admin/db/oracle/patches/038-add-policy-change-log-table.sql PRE-CREATION 
  security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql dfa8c829c 
  security-admin/db/postgres/patches/038-add-policy-change-log-table.sql PRE-CREATION 
  security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 81c6172a6

  security-admin/db/sqlanywhere/patches/038-add-policy-change-log-table.sql PRE-CREATION 
  security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 845e0891c 
  security-admin/db/sqlserver/patches/038-add-policy-change-log-table.sql PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java e1b244d45 
  security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java 072495212

  security-admin/src/main/java/org/apache/ranger/common/db/JPABeanCallbacks.java 70ad44b48

  security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java be1492282 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java a79ba7c77 
  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyChangeLog.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java d7089275b 
  security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 0281c94a5 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java a43d076fd 
  security-admin/src/main/java/org/apache/ranger/service/RangerSecurityZoneService.java ab8931910

  security-admin/src/main/resources/META-INF/jpa_named_queries.xml a6606010c 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java bf19efd31 
  security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java a1b0e45d5 
  security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java ef4865afd 


Diff: https://reviews.apache.org/r/70078/diff/2/

Changes: https://reviews.apache.org/r/70078/diff/1-2/


Testing
-------

- Successfully executed all tests. 
- Ensured that ranger admin and plugin functionality is indentical with or without policy
deltas enabled.
- Ran performance test to ensure that JVM Garbage collection behavior is positively impacted
(No Full GC on ranger-admin). 
- Noticed the policy-engine building time reduced by 80% with ~1800 large policies, with policy-engine
rebuilding within a second in the - plugin when a single policy was updated. 
- With ~3000 large policies, the policy-engine was built in about 2 seconds as compared to
10 seconds without incremental policy updates.


Thanks,

Abhay Kulkarni


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message