spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <cloud0...@gmail.com>
Subject Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Date Mon, 30 Dec 2019 04:32:57 GMT
+1 for the new thrift server to get rid of the Hive dependencies!

On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <wgyumg@gmail.com> wrote:

> I'm  +1 for this SPIP for these two reasons:
>
> 1. The current thriftserver has some issues that are not easy to solve,
> such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>.
> 2. The difference between the version of ORC we are using and the built-in
> Hive is using is getting bigger and bigger. We can't ensure that there will
> be no compatibility issues in the future. If thriftserver does not depend
> on Hive, it will be much easier to upgrade the built-in Hive in the future.
>
> On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <angers.zhu@gmail.com> wrote:
>
>> Hi all,
>>
>> I have complete a Design doc about how to use and config this new thrift
>> server, and some design detail about change and impersonation.
>>
>> Hope for your suggestions and ideas.
>>
>> SPIP DOC :
>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0
>> Design DOC :
>> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14
>> Thrift server about configurations
>> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web&ouid=110963191229426834922
>>
>> Best Regards
>> angers.zhu
>> angers.zhu@gmail.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>
>> --------- Forwarded Message ---------
>> From: angers.zhu <angers.zhu@gmail.com> <angers.zhu@gmail.com>
>> Date: 12/18/2019 22:29
>> To: dev-owner@spark.apache.org <dev-owner@spark.apache.org>
>> <dev-owner@spark.apache.org>
>> Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based
>> on protocol v11
>>
>> Add spark-dev group access privilege to google.
>> angers.zhu
>> angers.zhu@gmail.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>
>> On 12/18/2019 22:02,Sandeep Katta<sandeep0102.opensource@gmail.com>
>> <sandeep0102.opensource@gmail.com> wrote:
>>
>> I couldn't access the doc, please give permission to the spark-dev group
>>
>> On Wed, 18 Dec 2019 at 18:05, angers.zhu <angers.zhu@gmail.com> wrote:
>>
>>> With the development of Spark and Hive,in current sql/hive-thriftserver
>>> module,
>>>
>>> we need to do a lot of work to solve code conflicts for different
>>> built-in hive versions.
>>>
>>> It's an annoying and unending work in current ways. And these issues
>>> have limited
>>>
>>> our ability and convenience to develop new features for Spark’s thrift
>>> server.
>>>
>>>     We suppose to implement a new thrift server and JDBC driver based on Hive’s
>>> latest v11
>>>
>>> TCLService.thrift thrift protocol. Finally, the new thrift server have
>>> below feature:
>>>
>>>    1.
>>>
>>>    Build new module spark-service as spark’s thrift server
>>>    2.
>>>
>>>    Don't need as much reflection and inherited code as `hive-thriftser`
>>>    modules
>>>    3.
>>>
>>>    Support all functions current `sql/hive-thriftserver` support
>>>    4.
>>>
>>>    Use all code maintained by spark itself, won’t depend on Hive
>>>    5.
>>>
>>>    Support origin functions use spark’s own way, won't limited by
>>>    Hive's code
>>>    6.
>>>
>>>    Support running without hive metastore or with hive metastore
>>>    7.
>>>
>>>    Support user impersonation by Multi-tenant splited hive
>>>    authentication and DFS authentication
>>>    8.
>>>
>>>    Support session hook for with spark’s own code
>>>    9.
>>>
>>>    Add a new jdbc driver spark-jdbc, with spark’s own connection url
>>>    “jdbc:spark:<host>:<port>/<db>”
>>>    10.
>>>
>>>    Support both hive-jdbc and spark-jdbc client, then we can support
>>>    most clients and BI platform
>>>
>>>
>>>
>>> https://issues.apache.org/jira/browse/SPARK-29018
>>>
>>> Google Doc:
>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>> I'll start with my +1
>>> angers.zhu
>>> angers.zhu@gmail.com
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>
>>>

Mime
View raw message