spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 崔苗(数据与人工智能产品开发部) <0049003...@znv.com>
Subject Re: how to use cluster sparkSession like localSession
Date Fri, 02 Nov 2018 06:01:42 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.6;
    }
    ul,ol{
        padding-left: 20px;
        list-style-position: inside;
    }
</style>
<div style = 'font-family:微软雅黑,Verdana,&quot;Microsoft Yahei&quot;,SimSun,sans-serif;
line-height:1.6;'>
    <div ></div><div>
    <div>
        <span>we use spark in web server , no application.jar and submit on the cluster
        </span></div>
    <div>
        <span>
            <br>
        </span>
    </div>
    <div id="ntes-pcmac-signature" style="font-family:'微软雅黑'">
     
    <div style="font-size:14px; padding: 0;  margin:0;line-height:14px;">
        <div style="padding-bottom:6px;margin-bottom:10px;border-bottom:1px solid #e6e6e6;display:inline-block;">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=0049003208&amp;uid=0049003208%40znv.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&amp;items=%5B%220049003208%40znv.com%22%5D"
style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px
0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none
!important;text-size-adjust:none !important;">
            <table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed;
border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none
!important;text-size-adjust:none !important;">
                <tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi
Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none
!important;text-size-adjust:none !important;">
                    <tr>
                            <td width="38" style="padding:0; box-sizing: border-box; width:
38px;">
                                <img width="38" height="38" style="vertical-align:middle;
width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
                            </td>
                            <td style="padding: 0 0 0 10px; color: #31353b;">
                                <div style="font-size: 16px;font-weight:bold; width:100%;
white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">0049003208</div>
                            </td>
                    </tr>
                        <tr width="100%" style="font-size: 14px !important; width: 100%;">
                            <td colspan="2" style="padding:10px 0 0 0; font-size:14px !important;
width: 100%;">
                                    <div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">0049003208@znv.com</div>
                            </td>
                        </tr>
                </tbody>
            </table>
        </a>
        </div>
    </div>
    <div style="font-size:12px;color:#b5b9bd;line-height:18px;">
        <span>签名由</span>
        <a style="text-decoration: none;color:#4196ff;padding:0 5px;" href="https://mail.163.com/dashi/dlpro.html?from=mail81">网易邮箱大师</a>
        <span>定制</span>
    </div>
 </div>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:12px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">11/2/2018 13:56</span>,<a class="mail-to" style="text-decoration:none;color:#2a83f2;"
href="mailto:akhalil@an10.io">Arbab Khalil&lt;akhalil@an10.io&gt;</a> wrote:
</div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family:
'微软雅黑';">
<div dir="ltr">remove master configuration from code and then submit it to any cluster,
it should work.</div><br><div class="gmail_quote"><div dir="ltr">On
Fri, Nov 2, 2018 at 10:52 AM 崔苗(数据与人工智能产品开发部) &lt;<a href="mailto:0049003208@znv.com">0049003208@znv.com</a>&gt;
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">

    

<div>

<div style="font-family:微软雅黑,Verdana,&quot;Microsoft Yahei&quot;,SimSun,sans-serif;line-height:1.6">
    <div></div><div>
    <div>
        <span>
            <br>
        </span>
    </div>
    <div>
        <span>then how about spark sql and spark MLlib , we use them at most time
        </span></div>
    <div id="m_3286316634789577518ntes-pcmac-signature" style="font-family:'\005fae\008f6f\0096c5\009ed1'">
     
    <div style="font-size:14px;padding:0;margin:0;line-height:14px">
        <div style="padding-bottom:6px;margin-bottom:10px;border-bottom:1px solid #e6e6e6;display:inline-block">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=0049003208&amp;uid=0049003208%40znv.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&amp;items=%5B%220049003208%40znv.com%22%5D"
style="display:block;background:#fff;max-width:400px;padding:15px 0 10px 0;text-decoration:none;outline:none"
target="_blank">
            <table cellpadding="0" style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3">
                <tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro
Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all">
                    <tr class="firstRow">
                            <td width="38" style="padding:0;box-sizing:border-box;width:38px">
                                <img width="38" height="38" style="vertical-align:middle;width:38px;height:38px;border-radius:50%"
src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
                            </td>
                            <td style="padding:0 0 0 10px;color:#31353b">
                                <div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">0049003208</div>
                            </td>
                    </tr>
                        <tr width="100%" style="font-size:14px!important;width:100%">
                            <td colspan="2" style="padding:10px 0 0 0;font-size:14px!important;width:100%">
                                    <div style="width:100%;font-size:14px!important;word-wrap:break-word;word-break:break-all">0049003208@znv.com</div>
                            </td>
                        </tr>
                </tbody>
            </table>
        </a>
        </div>
    </div>
    <div style="font-size:12px;color:#b5b9bd;line-height:18px">
        <span>签名由</span>
        <a style="text-decoration:none;color:#4196ff;padding:0 5px" href="https://mail.163.com/dashi/dlpro.html?from=mail81"
target="_blank">网易邮箱大师</a>
        <span>定制</span>
    </div>
 </div>
</div><div class="m_3286316634789577518J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'\005fae\008f6f\0096c5\009ed1'">
    <div style="font-size:12px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="m_3286316634789577518mail-date">11/2/2018 11:58</span>,<a class="m_3286316634789577518mail-to"
style="text-decoration:none;color:#2a83f2" href="mailto:daniel.oliveira.mantovani@gmail.com"
target="_blank">Daniel de Oliveira Mantovani&lt;daniel.oliveira.mantovani@gmail.com&gt;</a>
wrote: </div>
</div>
<blockquote id="m_3286316634789577518ntes-pcmail-quote" style="margin:0;padding:0;font-size:14px;font-family:'\005fae\008f6f\0096c5\009ed1'">
<div><div dir="auto">Please, read about Spark Streaming or Spark Structured Streaming.
Your web application can easily communicate through some API and you won’t have the overhead
of start a new spark job, which is pretty heavy.</div></div><div><br><div
class="gmail_quote"><div dir="ltr">On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部)
&lt;<a href="mailto:0049003208@znv.com" target="_blank">0049003208@znv.com</a>&gt;
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">

    

<div>

<div style="font-family:微软雅黑,Verdana,&quot;Microsoft Yahei&quot;,SimSun,sans-serif;line-height:1.6">
    <div></div><div>
    <div>
        <span>
            <br>
        </span>
    </div>
    <div><span>Hi,
        </span></div><div>we want to execute spark code with out submit
application.jar,like this code:</div><div><br></div><div>public
static void main(String args[]) throws Exception{
</div><div>
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; SparkSession spark = SparkSession
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; .builder()
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; .master("local[*]")
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; .appName("spark test")
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; .getOrCreate();
</div><div>&nbsp; &nbsp; &nbsp;&nbsp;</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; Dataset&lt;Row&gt; testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; testData.printSchema();
</div><div>&nbsp; &nbsp; &nbsp; &nbsp; testData.show();
</div><div>&nbsp; &nbsp; }</div><div><br></div><div>the
above code can work well with idea , do not need to generate jar file and submit , but if
we replace&nbsp;<span style="line-height:22.4px">master("local[*]") with&nbsp;</span><span
style="line-height:22.4px">master("yarn") , it can't work , so is there a way to use cluster
sparkSession like local sparkSession ? &nbsp;we need to dynamically execute spark code
in web server according to the different request , &nbsp;such as filter request will call
dataset.filter() , so there is no application.jar to submit .<br></span><span
style="line-height:22.4px">&nbsp;</span></div>
    <div id="m_3286316634789577518m_418094061433616952ntes-pcmac-signature" style="font-family:'\005fae\008f6f\0096c5\009ed1'">
     
    <div style="font-size:14px;padding:0;margin:0;line-height:14px">
        <div style="padding-bottom:6px;margin-bottom:10px;border-bottom:1px solid #e6e6e6;display:inline-block">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=0049003208&amp;uid=0049003208%40znv.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&amp;items=%5B%220049003208%40znv.com%22%5D"
style="display:block;background:#fff;max-width:400px;padding:15px 0 10px 0;text-decoration:none;outline:none"
target="_blank">
            <table cellpadding="0" style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3">
                <tbody style="font-family:'PingFang SC','Hiragino Sans GB','WenQuanYi Micro
Hei','Microsoft Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all">
                    <tr class="m_3286316634789577518m_418094061433616952firstRow">
                            <td width="38" style="padding:0;box-sizing:border-box;width:38px">
                                <img width="38" height="38" style="vertical-align:middle;width:38px;height:38px;border-radius:50%"
src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png">
                            </td>
                            <td style="padding:0 0 0 10px;color:#31353b">
                                <div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">0049003208</div>
                            </td>
                    </tr>
                        <tr width="100%" style="font-size:14px!important;width:100%">
                            <td colspan="2" style="padding:10px 0 0 0;font-size:14px!important;width:100%">
                                    <div style="width:100%;font-size:14px!important;word-wrap:break-word;word-break:break-all">0049003208@znv.com</div>
                            </td>
                        </tr>
                </tbody>
            </table>
        </a>
        </div>
    </div>
    <div style="font-size:12px;color:#b5b9bd;line-height:18px">
        <span>签名由</span>
        <a style="text-decoration:none;color:#4196ff;padding:0 5px" href="https://mail.163.com/dashi/dlpro.html?from=mail81"
target="_blank">网易邮箱大师</a>
        <span>定制</span>
    </div>
 </div>
</div>
</div>
</div>

---------------------------------------------------------------------
To unsubscribe e-mail: <a href="mailto:user-unsubscribe@spark.apache.org" target="_blank">user-unsubscribe@spark.apache.org</a>

</blockquote></div></div>-- <br><div dir="ltr" class="m_3286316634789577518gmail_signature"
data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div
dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr">--</div><div
dir="ltr">Daniel de Oliveira Mantovani</div><div><span style="font-size:12.8px">Perl
Evangelist/Data Hacker</span><br></div><div dir="ltr"><span style="font-size:12.8px">+1
786 459 1341</span></div></div></div></div></div></div></div>

</blockquote>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div
dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<div><font
color="#444444" size="2">Arbab Khalil</font></div><div><font color="#444444"
size="2">Software Design Engineer</font></div></div></div>

</blockquote><!--😀-->
</div>
</body>
</html>
Mime
View raw message