ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kent Yao <yaooq...@hotmail.com>
Subject Re: Review Request 70676: Implement Spark SQL plugin
Date Sat, 18 May 2019 15:28:57 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70676/
-----------------------------------------------------------

(Updated 五月 18, 2019, 3:28 p.m.)


Review request for ranger and Don Bosco Durai.


Bugs: https://issues.apache.org/jira/browse/RANGER-2128
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2128


Repository: ranger


Description
-------

Ranger Spark SQL Plugin for Spark SQL/Dataframe/Dataset


Diffs
-----

  .gitignore c2def9d 
  agents-audit/pom.xml 4aaec09 
  agents-common/pom.xml 4486af3 
  agents-cred/pom.xml c622683 
  agents-installer/pom.xml 20902cb 
  credentialbuilder/pom.xml 461dcd0 
  embeddedwebserver/pom.xml a8c7cba 
  hbase-agent/pom.xml c044ef0 
  hive-agent/pom.xml 0a091b2 
  jisql/pom.xml 07fb08b 
  kms/pom.xml 043c67f 
  knox-agent/pom.xml 0c45c84 
  plugin-atlas/pom.xml 87ff40d 
  plugin-elasticsearch/pom.xml b997310 
  plugin-kafka/pom.xml 132baf7 
  plugin-kms/pom.xml 1940f41 
  plugin-kylin/pom.xml 69badfc 
  plugin-nifi-registry/pom.xml 3e99f04 
  plugin-nifi/pom.xml c510d87 
  plugin-solr/pom.xml 619cc9e 
  plugin-spark/.gitignore PRE-CREATION 
  plugin-spark/pom.xml PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkAccessRequest.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkAuditHandler.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkAuthorizer.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkPlugin.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkResource.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkSQLExtension.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkAccessControlException.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkAccessType.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkObjectType.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkOperationType.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkPrivObjectActionType.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkPrivilegeObject.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkPrivilegeObjectType.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/authorizer.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/AuthzUtils.scala PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkAuthorizerExtension.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkMaskingExtension.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkOptimizer.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkRowFilterExtension.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RangerSparkMasking.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RangerSparkRowFilter.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/execution/RangerShowDatabasesCommand.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/execution/RangerShowTablesCommand.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/execution/RangerSparkPlanOmitStrategy.scala
PRE-CREATION 
  plugin-spark/src/main/scala/org/apache/spark/sql/hive/PrivilegesBuilder.scala PRE-CREATION

  plugin-spark/src/test/resources/data/files/kv1.txt PRE-CREATION 
  plugin-spark/src/test/resources/log4j.properties PRE-CREATION 
  plugin-spark/src/test/resources/ranger-spark-security.xml PRE-CREATION 
  plugin-spark/src/test/resources/sparkSql_hive_jenkins.json PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q1.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q10.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q11.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q12.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q13.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q14a.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q14b.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q15.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q16.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q17.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q18.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q19.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q2.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q20.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q21.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q22.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q23a.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q23b.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q24a.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q24b.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q25.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q26.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q27.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q28.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q29.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q3.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q30.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q31.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q32.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q33.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q34.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q35.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q36.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q37.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q38.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q39a.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q39b.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q4.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q40.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q41.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q42.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q43.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q44.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q45.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q46.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q47.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q48.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q49.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q5.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q50.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q51.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q52.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q53.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q54.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q55.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q56.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q57.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q58.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q59.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q6.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q60.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q61.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q62.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q63.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q64.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q65.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q66.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q67.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q68.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q69.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q7.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q70.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q71.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q72.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q73.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q74.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q75.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q76.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q77.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q78.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q79.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q8.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q80.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q81.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q82.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q83.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q84.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q85.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q86.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q87.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q88.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q89.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q9.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q90.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q91.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q92.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q93.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q94.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q95.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q96.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q97.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q98.sql PRE-CREATION 
  plugin-spark/src/test/resources/tpcds/q99.sql PRE-CREATION 
  plugin-spark/src/test/scala/org/apache/ranger/services/spark/RangerAdminClientImpl.scala
PRE-CREATION 
  plugin-spark/src/test/scala/org/apache/ranger/services/spark/SparkRangerAuthorizerTest.scala
PRE-CREATION 
  plugin-spark/src/test/scala/org/apache/spark/sql/RangerSparkTestUtils.scala PRE-CREATION

  plugin-spark/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkAuthorizerExtensionTest.scala
PRE-CREATION 
  plugin-spark/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkMaskingExtensionTest.scala
PRE-CREATION 
  plugin-spark/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkRowFilterExtensionTest.scala
PRE-CREATION 
  plugin-spark/src/test/scala/org/apache/spark/sql/execution/RangerSparkPlanOmitStrategyTest.scala
PRE-CREATION 
  plugin-sqoop/pom.xml af00e97 
  plugin-yarn/pom.xml f273569 
  pom.xml 2a2540a 
  ranger-atlas-plugin-shim/pom.xml 86d6a26 
  ranger-elasticsearch-plugin-shim/pom.xml 59a653b 
  ranger-examples/pom.xml fa4cb20 
  ranger-examples/sampleapp/pom.xml add9546 
  ranger-hbase-plugin-shim/pom.xml 9ed0aeb 
  ranger-hdfs-plugin-shim/pom.xml f45621c 
  ranger-hive-plugin-shim/pom.xml 26258d7 
  ranger-kafka-plugin-shim/pom.xml 5a7353f 
  ranger-kms-plugin-shim/pom.xml 0b505f6 
  ranger-knox-plugin-shim/pom.xml db342b7 
  ranger-kylin-plugin-shim/pom.xml 4bb9f34 
  ranger-plugin-classloader/pom.xml 0aa71c3 
  ranger-solr-plugin-shim/pom.xml 46b7821 
  ranger-sqoop-plugin-shim/pom.xml 86ab432 
  src/main/assembly/plugin-spark.xml PRE-CREATION 
  ugsync/pom.xml c30d8ca 
  unixauthclient/pom.xml 4625925 
  unixauthpam/pom.xml 0f2a459 


Diff: https://reviews.apache.org/r/70676/diff/1/


Testing (updated)
-------

https://issues.apache.org/jira/browse/RANGER-2128

## UTs
add Unit Tests
add TPCDS 99 queries as unit tests.

Can be verified via:

1. cd plugin-spark
2. mvn clean package -Pspark-2.3 or mvn clean package -Pspark-2.4

## Test manually

1. add the the shaded jar which can be found plugin-spark/target after mvn package to SPARK_HOME/jars
2. add/create ranger-spark-security.xml etc as we do for hive, then mv all the plugin configuration
file to SPARK_HOME/jars
3. cd $SPARK_HOME
4. run bin/spark-sql --conf spark.sql.extensions=org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension

**No need to configure HiveAuthorizerFactory etc in hive-site.xml**


Thanks,

Kent Yao


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message