spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Katta <sandeep0102.opensou...@gmail.com>
Subject Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Date Mon, 30 Dec 2019 06:38:29 GMT
+1

On Mon, 30 Dec 2019 at 10:24, Gengliang <ltnwgl@gmail.com> wrote:

> +1
>
> On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <cloud0fan@gmail.com> wrote:
>
>> +1 for the new thrift server to get rid of the Hive dependencies!
>>
>> On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <wgyumg@gmail.com> wrote:
>>
>>> I'm  +1 for this SPIP for these two reasons:
>>>
>>> 1. The current thriftserver has some issues that are not easy to solve,
>>> such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>
>>> .
>>> 2. The difference between the version of ORC we are using and the
>>> built-in Hive is using is getting bigger and bigger. We can't ensure that
>>> there will be no compatibility issues in the future. If thriftserver does
>>> not depend on Hive, it will be much easier to upgrade the built-in Hive in
>>> the future.
>>>
>>> On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <angers.zhu@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have complete a Design doc about how to use and config this new
>>>> thrift server, and some design detail about change and impersonation.
>>>>
>>>> Hope for your suggestions and ideas.
>>>>
>>>> SPIP DOC :
>>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0
>>>> Design DOC :
>>>> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14
>>>> Thrift server about configurations
>>>> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web&ouid=110963191229426834922
>>>>
>>>> Best Regards
>>>> angers.zhu
>>>> angers.zhu@gmail.com
>>>>
>>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>>
>>>> --------- Forwarded Message ---------
>>>> From: angers.zhu <angers.zhu@gmail.com> <angers.zhu@gmail.com>
>>>> Date: 12/18/2019 22:29
>>>> To: dev-owner@spark.apache.org <dev-owner@spark.apache.org>
>>>> <dev-owner@spark.apache.org>
>>>> Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based
>>>> on protocol v11
>>>>
>>>> Add spark-dev group access privilege to google.
>>>> angers.zhu
>>>> angers.zhu@gmail.com
>>>>
>>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>>
>>>> On 12/18/2019 22:02,Sandeep Katta<sandeep0102.opensource@gmail.com>
>>>> <sandeep0102.opensource@gmail.com> wrote:
>>>>
>>>> I couldn't access the doc, please give permission to the spark-dev group
>>>>
>>>> On Wed, 18 Dec 2019 at 18:05, angers.zhu <angers.zhu@gmail.com> wrote:
>>>>
>>>>> With the development of Spark and Hive,in current
>>>>> sql/hive-thriftserver module,
>>>>>
>>>>> we need to do a lot of work to solve code conflicts for different
>>>>> built-in hive versions.
>>>>>
>>>>> It's an annoying and unending work in current ways. And these issues
>>>>> have limited
>>>>>
>>>>> our ability and convenience to develop new features for Spark’s thrift
>>>>> server.
>>>>>
>>>>>     We suppose to implement a new thrift server and JDBC driver based
>>>>> on Hive’s latest v11
>>>>>
>>>>> TCLService.thrift thrift protocol. Finally, the new thrift server
>>>>> have below feature:
>>>>>
>>>>>    1.
>>>>>
>>>>>    Build new module spark-service as spark’s thrift server
>>>>>    2.
>>>>>
>>>>>    Don't need as much reflection and inherited code as
>>>>>    `hive-thriftser` modules
>>>>>    3.
>>>>>
>>>>>    Support all functions current `sql/hive-thriftserver` support
>>>>>    4.
>>>>>
>>>>>    Use all code maintained by spark itself, won’t depend on Hive
>>>>>    5.
>>>>>
>>>>>    Support origin functions use spark’s own way, won't limited by
>>>>>    Hive's code
>>>>>    6.
>>>>>
>>>>>    Support running without hive metastore or with hive metastore
>>>>>    7.
>>>>>
>>>>>    Support user impersonation by Multi-tenant splited hive
>>>>>    authentication and DFS authentication
>>>>>    8.
>>>>>
>>>>>    Support session hook for with spark’s own code
>>>>>    9.
>>>>>
>>>>>    Add a new jdbc driver spark-jdbc, with spark’s own connection url
>>>>>    “jdbc:spark:<host>:<port>/<db>”
>>>>>    10.
>>>>>
>>>>>    Support both hive-jdbc and spark-jdbc client, then we can support
>>>>>    most clients and BI platform
>>>>>
>>>>>
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-29018
>>>>>
>>>>> Google Doc:
>>>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>
>>>>> I'll start with my +1
>>>>> angers.zhu
>>>>> angers.zhu@gmail.com
>>>>>
>>>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81>
定制
>>>>>
>>>>>

Mime
View raw message