spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From angers.zhu <angers....@gmail.com>
Subject [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Date Wed, 18 Dec 2019 12:35:02 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.6;
    }
    ul,ol{
        padding-left: 20px;
        list-style-position: inside;
    }
</style>
<div style = 'font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;'>
    <div ></div>
<style>
    font{
        line-height: 1.6;
    }
    ul,ol{
        padding-left: 20px;
        list-style-position: inside;
    }
</style>
<div style="font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;">
    <div></div><div>
    <div>
        <span id="docs-internal-guid-29f11947-7fff-fd65-8ed8-f65063dc4df3"><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;
font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">
   With the development of Spark and Hive,in current </span><span style="font-size:
11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0); font-variant-ligatures:
normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;">sql/hive-thriftserver</span><span style="font-size: 11pt;
font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">
module,&nbsp;</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal;
font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;">we need to do a lot of work to solve code conflicts for different
built-in hive versions.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal;
font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;">It's an annoying and unending work in current ways. And these issues
have limited&nbsp;</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal;
font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;">our ability and convenience to develop new features for Spark’s
thrift server.&nbsp;</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal;
font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;">&nbsp;&nbsp;&nbsp;&nbsp;We suppose to implement
a new thrift server and JDBC driver based on&nbsp;</span><span style="font-size:
11pt; font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Hive’s
latest v11&nbsp;</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: rgb(255,
255, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre-wrap;">TCLService.thrift</span><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); font-variant-ligatures: normal;
font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;"> thrift protocol. Finally, the new thrift server have below feature:</span></p><ol><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Build
new module </span><span style="font-size: 11pt; background-color: rgb(255, 255, 0);
font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal;
vertical-align: baseline; white-space: pre-wrap;">spark-service</span><span style="font-size:
11pt; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre-wrap;"> as spark’s thrift server&nbsp;</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Don't
need as much reflection and inherited code as `hive-thriftser` modules</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Support
all functions current `sql/hive-thriftserver` support</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Use
all code maintained by spark itself, won’t depend on Hive</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Support
origin functions use spark’s own way, won't limited by Hive's code</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Support
running without hive metastore or with hive metastore</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Support
user impersonation by Multi-tenant splited hive authentication and DFS authentication</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Support
session hook for with spark’s own code</span></p></li><li dir="ltr"
style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0);
font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal;
vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Add
a new jdbc driver </span><span style="font-size: 11pt; color: rgb(152, 0, 0); background-color:
rgb(255, 255, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre-wrap;">spark-jdbc</span><span
style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre-wrap;">, with spark’s own connection
url&nbsp; “jdbc:spark:&lt;host&gt;:&lt;port&gt;/&lt;db&gt;”</span></p></li><li
dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre;"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Support
both </span><span style="font-size: 11pt; background-color: rgb(255, 255, 0); font-variant-ligatures:
normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;">hive-jdbc</span><span style="font-size: 11pt; font-variant-ligatures:
normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;
white-space: pre-wrap;"> and </span><span style="font-size: 11pt; background-color:
rgb(255, 255, 0); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre-wrap;">spark-jdbc</span><span
style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position:
normal; vertical-align: baseline; white-space: pre-wrap;"> client, then we can support
most clients and BI platform</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span style="font-size: 11pt; font-variant-ligatures: normal; font-variant-east-asian:
normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br></span></p></li></ol></span><span>
            
        </span>
    </div>
    <div><a href="https://issues.apache.org/jira/browse/SPARK-29018">https://issues.apache.org/jira/browse/SPARK-29018</a></div><div><span>&nbsp;</span></div><div><span>Google
Doc: &nbsp;</span><a href="https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#">https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#</a></div><div><br
style="font-family: Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica,
微软雅黑, 宋体;">[ ] +1: Accept the proposal as an official SPIP</span><br
style="font-family: Helvetica, 微软雅黑, 宋体;"><span style="font-family: Helvetica,
微软雅黑, 宋体;">[ ] +0</span><br style="font-family: Helvetica, 微软雅黑,
宋体;"><span style="font-family: Helvetica, 微软雅黑, 宋体;">[ ] -1: I don't
think this is a good idea because ...</span><br style="font-family: Helvetica, 微软雅黑,
宋体;"><br style="font-family: Helvetica, 微软雅黑, 宋体;"><span style="font-family:
Helvetica, 微软雅黑, 宋体;">I'll start with my +1</span></div>
    <div id="ntes-pcmac-signature" style="font-family:'Helvetica','Microsoft Yahei', '微软雅黑'">
      
    <div style="font-size:14px; padding: 0;  margin:0;line-height: 14px;">
        <div style="border-bottom:1px solid #e6e6e6;display:inline-block;">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=angers.zhu&amp;uid=angers.zhu%40gmail.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&amp;items=%5B%22angers.zhu%40gmail.com%22%5D"
style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px
0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none
!important;text-size-adjust:none !important;">
            <table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed;
border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none
!important;text-size-adjust:none !important;">
                <tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi
Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none
!important;text-size-adjust:none !important;">
                    <tr class="firstRow">
                            <td width="38" style="padding:0; box-sizing: border-box; width:
38px;">
                                <img width="38" height="38" style="vertical-align:middle;
width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
                            </td>
                            <td style="padding: 0 0 0 10px; color: #31353b;">
                                <div style="font-size: 16px;font-weight:bold; width:100%;
white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">angers.zhu</div>
                            </td>
                    </tr>
                        <tr width="100%" style="font-size: 14px !important; width: 100%;">
                            <td colspan="2" style="padding:10px 0 0 0; font-size:14px !important;
width: 100%;">
                                    <div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">angers.zhu@gmail.com</div>
                            </td>
                        </tr>
                </tbody>
            </table>
        </a>
        </div>
    </div>
    <div style="font-size:12px;color:#b5b9bd;margin-top:8px;line-height: 18px;">
        <span>签名由</span>
        <a style="text-decoration: none;color:#4196ff;padding:0px;" href="https://mail.163.com/dashi/dlpro.html?from=mail81">网易邮箱大师</a>
        <span>定制</span>
    </div>
 </div>
    <br>
</div><!--�-->
</div><!--�-->
</div>
</body>
</html>
Mime
View raw message