ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qiang Zhang <zhangqia...@zte.com.cn>
Subject Re: Review Request 68128: RANGER-2170:Ranger supports plugin to enable, monitor and manage Elasticsearch
Date Thu, 13 Sep 2018 02:29:57 GMT


> On 九月 6, 2018, 6:30 p.m., Ramesh Mani wrote:
> > ranger-elasticsearch-plugin-shim/conf/plugin-descriptor.properties
> > Lines 46 (patched)
> > <https://reviews.apache.org/r/68128/diff/2/?file=2072368#file2072368line46>
> >
> >     Why is -shim- is having the conf and classes which are core to the plugin it
should part of /plugin-elasticseach/conf folder.
> >     -shim is to hold the ranger-classloader.
> >     Please refer the existing plugin and see if that can be done.
> 
> Qiang Zhang wrote:
>     Because unlike other Hadoop components, Elasticseach is designed to be pluggable.
>     To implement a new extension function, 
>     we need to organize code and configuration files according to the requirements of
Elasticseach.
>     Some classes in ranger-elasticsearch-plugin-shim is necessary to mount on Elasticseach.
>     Other classes is due to the fact that Elasticseach itself does not support user authentication,
>     so this plugin should work with other Elasticsearch plugin to authenticate users,
>     such as Basic Authentication, Kerberos, LDAP, etc.
>     Or, in the future, we can realize user authentication in ranger-elasticsearch-plugin-shim.
>     So these classes can't sink into plugin-elasticseach.
>     And plugin-descriptor.properties is for Elasticseach to start the Ranger Elasticseach
plugin.
>     But the files in the plugin-elasticseach/conf directory are for Ranger Elasticseach
plugin.
>     
>     The related directory structure after plugin installed is as follows:
>     1.ranger-elasticsearch-plugin-shim/conf/?
>     ```
>     [elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ pwd
>     /home/elasticsearch/elasticsearch-6.2.2/plugins/ranger-elasticsearch-plugin
>     [elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ ll
>     -rwxrwxrwx. 1 elasticsearch hadoop 588337 4?  25 2017 commons-collections-3.2.2.jar
>     -rwxrwxrwx. 1 elasticsearch hadoop 284220 4?  25 2017 commons-lang-2.6.jar
>     -rwxrwxrwx. 1 elasticsearch hadoop   2547 6?  26 09:41 plugin-descriptor.properties
>     -rwxrwxrwx. 1 elasticsearch hadoop   1754 6?  26 09:27 plugin-security.policy
>     drwxrwxrwx. 2 elasticsearch hadoop   4096 7?  13 09:40 ranger-elasticsearch-plugin-impl
>     -rwxrwxrwx. 1 elasticsearch hadoop  20627 6?  26 09:36 ranger-elasticsearch-plugin-shim-1.1.0-SNAPSHOT.jar
>     -rwxrwxrwx. 1 elasticsearch hadoop  16799 6?  26 09:35 ranger-plugin-classloader-1.1.0-SNAPSHOT.jar
>     -rwxrwxrwx. 1 elasticsearch hadoop  26084 4?  25 2017 slf4j-api-1.7.5.jar
>     -rwxrwxrwx. 1 elasticsearch hadoop   8866 6?  26 15:30 slf4j-log4j12-1.7.10.jar
>     ```
>     
>     2.plugin-elasticseach/conf?
>     ```
>     [elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ pwd
>     /home/elasticsearch/elasticsearch-6.2.2/config/ranger-elasticsearch-plugin
>     [elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ ll
>     -rwxrwxrwx. 1 elasticsearch hadoop 9548 6?  26 14:15 ranger-elasticsearch-audit.xml
>     -rwxrwxrwx. 1 elasticsearch hadoop 2773 6?  26 14:15 ranger-elasticsearch-security.xml
>     -rwxrwxrwx. 1 elasticsearch hadoop 1917 6?  26 14:15 ranger-policymgr-ssl.xml
>     -rwxrwxrwx. 1 elasticsearch hadoop   83 6?  26 14:15 ranger-security.xml
>     ```
>     
>     In addition, I have developed 2 plugins: Ranger Kylin Plugin and Ranger Sqoop2 Plugin.
>     In order to meet the requirements of Elasticsearch plugins 
>     and take into account the general design principles of Ranger plugins,
>     I think the implementation of Ranger Elasticsearch plugin is reasonable at present.
>     Try install this plugin , and you can get a better understanding of the implementation
of it.
> 
> Ramesh Mani wrote:
>     If you are moving the configs from ranger-elasticsearch-plugin-shim/conf to /home/elasticsearch/elasticsearch-6.2.2/plugins/ranger-elasticsearch-plugin,
you can have all the configs in plugin-elasticsearch/conf and during installtion you can move
it /home/elasticsearch/elasticsearch-6.2.2/plugins/ranger-elasticsearch-plugin. Now you have
config folder in both shim and plugin folder and this is confusing.
>     
>     I know that there are elastic-search specific classes which needs to be there in
shim folder which is fine.

The configs of ranger-elasticsearch-plugin-shim/conf can not be moved to 
/home/elasticsearch/elasticsearch-6.2.2/plugins/ranger-elasticsearch-plugin,
because they are the necessary components of Elasticsearch plugin,
If they are removed, it will result in Ranger Elasticsearch plugin not loaded or startup failure.
Especially this plugin-descriptor.properties,
please refer the following official description:
```
# Elasticsearch plugin descriptor file
# This file must exist as 'plugin-descriptor.properties' in a folder named `elasticsearch`
# inside all plugins.
#
### example plugin for "foo"
#
# foo.zip <-- zip file for the plugin, with this structure:
#|____elasticsearch/
#| |____   <arbitrary name1>.jar <-- classes, resources, dependencies
#| |____   <arbitrary nameN>.jar <-- any number of jars
#| |____   plugin-descriptor.properties <-- example contents below:
#
# classname=foo.bar.BazPlugin
# description=My cool plugin
# version=2.0
# elasticsearch.version=2.0
# java.version=1.7
#
```
These configuration files are strongly related to Elasticsearch,
which ard loaded by Elasticsearch.
and usually they will not be changed.
But the configs of plugin-elasticsearch/conf are related to Ranger,
which ard loaded by Ranger plugin.
So they should be separated,
but putting them together will be confusing.


- Qiang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68128/#review208418
-----------------------------------------------------------


On 八月 14, 2018, 8:46 a.m., Qiang Zhang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68128/
> -----------------------------------------------------------
> 
> (Updated 八月 14, 2018, 8:46 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O hEigeartaigh, Gautam
Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, Selvamohan Neethiraj, sam  rome, Venkat
Ranganathan, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2170
>     https://issues.apache.org/jira/browse/RANGER-2170
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Elasticsearch is a distributed, RESTful search and analytics engine capable of solving
a growing number of use cases. 
> Like Apache Solr, it is also an index server based on Lucence.
> Ranger supports plugin to enable, monitor and manage Elasticsearch,
> to control index security of Elasticsearch.
> 
> As there is X-Pack plugin for the Elasticsearch, but it is not free.
> X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting,

> and graph capabilities into one easy-to-install package.
> We refer to the Indices Privileges design of X-Pack,
> by keeping the permissions consistent,
> to make user use ranger Elasticsearch plugin easily.
> Reference X-Pack Indices Privileges:
> https://www.elastic.co/guide/en/x-pack/current/security-privileges.html
> 
> Here we develop Ranger Elasticsearch plugin, based on Elasticsearch version 6.2.2.
> Elasticsearch 6.2.2 was released in February 20, 2018, reference release-notes:
> https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.2.html
> Not like other system, Elasticsearch has no basic authentication, 
> it uses X-pack plugin to support basic authentication, 
> role-based access control, SSL/TLS encryption, LDAP and so on.
> Not like X-pack, our Ranger Elasticsearch plugin is designed to do authorization,
> it is to control index of Elasticsearch without authentication,
> this plugin should work with other Elasticsearch plugin to authenticate users.
> 
> 
> Diffs
> -----
> 
>   agents-common/scripts/enable-agent.sh ce0dc8c 
>   agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java e654f2b

>   agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
5e74da8 
>   agents-common/src/main/resources/service-defs/ranger-servicedef-elasticsearch.json
PRE-CREATION 
>   plugin-elasticsearch/.gitignore PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-audit.xml PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-security-changes.cfg PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-security.xml PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-policymgr-ssl.xml PRE-CREATION 
>   plugin-elasticsearch/pom.xml PRE-CREATION 
>   plugin-elasticsearch/scripts/install.properties PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/RangerServiceElasticsearch.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchClient.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchResourceMgr.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/privilege/IndexPrivilege.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/privilege/IndexPrivilegeUtils.java
PRE-CREATION 
>   pom.xml 8d7ea13 
>   ranger-elasticsearch-plugin-shim/.gitignore PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/conf/plugin-descriptor.properties PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/conf/plugin-security.policy PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/pom.xml PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAccessControl.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/RangerElasticsearchPlugin.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/action/filter/RangerSecurityActionFilter.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/authc/user/UsernamePasswordToken.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/rest/filter/RangerSecurityRestFilter.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/utils/RequestUtils.java
PRE-CREATION 
>   src/main/assembly/admin-web.xml d0f3545 
>   src/main/assembly/plugin-elasticsearch.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68128/diff/2/
> 
> 
> Testing
> -------
> 
> #Test Steps:
> 
> 1.Intall
> Ranger Elasticsearch Plugin Installation Guide	
> https://cwiki.apache.org/confluence/display/RANGER/Elasticsearch+Plugin
> Include install Elasticsearch and Ranger Elasticsearch Plugin,
> and verify install result.
> 
> 2.Create policy in Ranger Admin
> User "elasticsearch" has all permissions on all indices.
> User "yuwen" has permission "read" on index "twitter".
> 
> 3.Test permission
> 
> 3.1 successful:
> curl -u elasticsearch:xxx -X GET "localhost:9200/twitter/_stats?pretty"
> curl -u elasticsearch:xxx -X GET "localhost:9200/twitter2/_stats?pretty"
> curl -u yuwen:xxx -X GET "localhost:9200/twitter/_stats?pretty"
> 
> 3.2 failed:
> curl -X GET "localhost:9200/twitter/_stats?pretty"
> {
>   "error" : {
>     "root_cause" : [
>       {
>         "type" : "status_exception",
>         "reason" : "Error: User is null, the request requires user authentication."
>       }
>     ],
>     "type" : "status_exception",
>     "reason" : "Error: User is null, the request requires user authentication."
>   },
>   "status" : 401
> }
> 
> curl -u yuwen:xxx -X GET "localhost:9200/twitter2/_stats?pretty"
> {
>   "error" : {
>     "root_cause" : [
>       {
>         "type" : "status_exception",
>         "reason" : "Error: User[yuwen] could not do action[indices:monitor/stats] on
index[twitter2]"
>       }
>     ],
>     "type" : "status_exception",
>     "reason" : "Error: User[yuwen] could not do action[indices:monitor/stats] on index[twitter2]"
>   },
>   "status" : 403
> }
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message