spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gengliang <ltn...@gmail.com>
Subject Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Date Mon, 30 Dec 2019 04:53:38 GMT
+1

On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <cloud0fan@gmail.com> wrote:

> +1 for the new thrift server to get rid of the Hive dependencies!
>
> On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <wgyumg@gmail.com> wrote:
>
>> I'm  +1 for this SPIP for these two reasons:
>>
>> 1. The current thriftserver has some issues that are not easy to solve,
>> such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>.
>> 2. The difference between the version of ORC we are using and the
>> built-in Hive is using is getting bigger and bigger. We can't ensure that
>> there will be no compatibility issues in the future. If thriftserver does
>> not depend on Hive, it will be much easier to upgrade the built-in Hive in
>> the future.
>>
>> On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <angers.zhu@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I have complete a Design doc about how to use and config this new thrift
>>> server, and some design detail about change and impersonation.
>>>
>>> Hope for your suggestions and ideas.
>>>
>>> SPIP DOC :
>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0
>>> Design DOC :
>>> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14
>>> Thrift server about configurations
>>> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web&ouid=110963191229426834922
>>>
>>> Best Regards
>>> angers.zhu
>>> angers.zhu@gmail.com
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>
>>> --------- Forwarded Message ---------
>>> From: angers.zhu <angers.zhu@gmail.com> <angers.zhu@gmail.com>
>>> Date: 12/18/2019 22:29
>>> To: dev-owner@spark.apache.org <dev-owner@spark.apache.org>
>>> <dev-owner@spark.apache.org>
>>> Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based
>>> on protocol v11
>>>
>>> Add spark-dev group access privilege to google.
>>> angers.zhu
>>> angers.zhu@gmail.com
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>
>>> On 12/18/2019 22:02,Sandeep Katta<sandeep0102.opensource@gmail.com>
>>> <sandeep0102.opensource@gmail.com> wrote:
>>>
>>> I couldn't access the doc, please give permission to the spark-dev group
>>>
>>> On Wed, 18 Dec 2019 at 18:05, angers.zhu <angers.zhu@gmail.com> wrote:
>>>
>>>> With the development of Spark and Hive,in current sql/hive-thriftserver
>>>> module,
>>>>
>>>> we need to do a lot of work to solve code conflicts for different
>>>> built-in hive versions.
>>>>
>>>> It's an annoying and unending work in current ways. And these issues
>>>> have limited
>>>>
>>>> our ability and convenience to develop new features for Spark’s thrift
>>>> server.
>>>>
>>>>     We suppose to implement a new thrift server and JDBC driver based
>>>> on Hive’s latest v11
>>>>
>>>> TCLService.thrift thrift protocol. Finally, the new thrift server have
>>>> below feature:
>>>>
>>>>    1.
>>>>
>>>>    Build new module spark-service as spark’s thrift server
>>>>    2.
>>>>
>>>>    Don't need as much reflection and inherited code as
>>>>    `hive-thriftser` modules
>>>>    3.
>>>>
>>>>    Support all functions current `sql/hive-thriftserver` support
>>>>    4.
>>>>
>>>>    Use all code maintained by spark itself, won’t depend on Hive
>>>>    5.
>>>>
>>>>    Support origin functions use spark’s own way, won't limited by
>>>>    Hive's code
>>>>    6.
>>>>
>>>>    Support running without hive metastore or with hive metastore
>>>>    7.
>>>>
>>>>    Support user impersonation by Multi-tenant splited hive
>>>>    authentication and DFS authentication
>>>>    8.
>>>>
>>>>    Support session hook for with spark’s own code
>>>>    9.
>>>>
>>>>    Add a new jdbc driver spark-jdbc, with spark’s own connection url
>>>>    “jdbc:spark:<host>:<port>/<db>”
>>>>    10.
>>>>
>>>>    Support both hive-jdbc and spark-jdbc client, then we can support
>>>>    most clients and BI platform
>>>>
>>>>
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-29018
>>>>
>>>> Google Doc:
>>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don't think this is a good idea because ...
>>>>
>>>> I'll start with my +1
>>>> angers.zhu
>>>> angers.zhu@gmail.com
>>>>
>>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>>
>>>>

Mime
View raw message