ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kent Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
Date Mon, 16 Jul 2018 05:45:00 GMT

    [ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544822#comment-16544822

Kent Yao commented on RANGER-2128:

Emmm..weird, ClassCastException shouldn't occur if the hive client was correctly replaced
by reflecting. When I tried thrift server, I did not see such an exception.  

Although I don’t want to admit it, I have to say that the current implementation of Spark
thrift server is really a mess. To meet spark's architecture, it was designed as an single
spark application without multi tenancy support.

Livy Server is multi tenant. It launches spark applications separately in different JVMs,
which is easy to work with our current work.

I guess that we may implement  spark sql plugin step by step,

1.  Resolve the hive client and jersey incompatibility 
2.  Implement plugin for spark client mode
3.  Implement plugin for spark cluster mode
4.  Separate metastore layer from PhysicalScan layer in Spark Thrift Server
5.  Implement plugin for thrift server

By 1-3, I guess we can supply all real multi tenant servers for spark, for spark thrift server,
we may need to make some efforts to spark community to support this such as step 4 and some
related work.

> Implement SparkSQL plugin
> -------------------------
>                 Key: RANGER-2128
>                 URL: https://issues.apache.org/jira/browse/RANGER-2128
>             Project: Ranger
>          Issue Type: New Feature
>          Components: plugins, Ranger
>    Affects Versions: 1.1.0
>            Reporter: t oo
>            Assignee: Kent Yao
>            Priority: Major
>             Fix For: 2.0.0
>         Attachments: support_ranger11.tgz
> Implement SparkSQL plugin

This message was sent by Atlassian JIRA

View raw message