spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumedh Wale <sw...@snappydata.io>
Subject Re: how to use cluster sparkSession like localSession
Date Mon, 05 Nov 2018 05:16:22 GMT
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Hi,<br>
    <br>
    I think what you need is to have a long running Spark cluster to
    which you can submit jobs dynamically.<br>
    <br>
    For SQL, you can start Spark's HiveServer2:
<a class="moz-txt-link-freetext" href="https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine">https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine</a><br>
    This will start a long running Spark cluster with a fixed
    configuration (executors, cores etc) and allows Spark to act more
    like a regular database. Then you can create jdbc:hive2:// JDBC
    connections from your app and run SQL queries/DDLs.<br>
    <br>
    For other components (or even SQL), you can start a Spark jobserver:
    <a class="moz-txt-link-freetext" href="https://github.com/spark-jobserver/spark-jobserver">https://github.com/spark-jobserver/spark-jobserver</a><br>
    This will again start a long running Spark cluster. It also allows
    you create new SparkContexts on-the-fly though that should not be
    done from a web app rather configured separately by admin if
    required. It will require you to implement your job as a
    SparkJob/SparkSessionJob that will be provided pre-created
    SparkContext/SparkSession, and these take parameters that can be
    read dynamically in your implementation. You register your classes
    in jars separately before-hand. Then you can call those methods
    using REST API from your application providing it the required
    parameters like a remote procedure call.<br>
    <div class="moz-signature"><br>
      Or you can try SnappyData that provides both of these (and much
      more) out of the box.<br>
      <br>
      Regards,<br>
      Sumedh Wale<br>
      SnappyData (<a class="moz-txt-link-freetext" href="http://www.snappydata.io">http://www.snappydata.io</a>)<br>
      <a href="https://snappydatainc.github.io/snappydata/">Documentation</a>
      <a href="http://www.snappydata.io/download">Download</a><br>
      <br>
    </div>
    <div class="moz-cite-prefix">On 02/11/18 11:22, 崔苗(数据与人工智能产品开发部)
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:D48D32B7-A493-4059-95FE-8C23AC0A6925@znv.com">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <style>
    font{
        line-height: 1.6;
    }
    ul,ol{
        padding-left: 20px;
        list-style-position: inside;
    }
</style>
      <div style="font-family:微软雅黑,Verdana,&quot;Microsoft
        Yahei&quot;,SimSun,sans-serif; line-height:1.6;">
        <div>
          <div> <span> <br>
            </span> </div>
          <div> <span>then how about spark sql and spark MLlib , we use
              them at most time </span></div>
          <div id="ntes-pcmac-signature" style="font-family:'微软雅黑'">
            <div style="font-size:14px; padding: 0;
              margin:0;line-height:14px;">
              <div
                style="padding-bottom:6px;margin-bottom:10px;border-bottom:1px
                solid #e6e6e6;display:inline-block;"> <a
href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=0049003208&amp;uid=0049003208%40znv.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&amp;items=%5B%220049003208%40znv.com%22%5D"
                  style="display:block;background:#fff; max-width:
                  400px; _width: 400px;padding:15px 0 10px
                  0;text-decoration: none;
outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none
                  !important;text-size-adjust:none !important;"
                  moz-do-not-send="true">
                  <table style="width: 100%; max-width: 100%;
                    table-layout: fixed; border-collapse:
                    collapse;color: #9b9ea1;font-size:
                    14px;line-height:1.3;-webkit-text-size-adjust:none
                    !important;text-size-adjust:none !important;"
                    cellpadding="0">
                    <tbody style="font-family: 'PingFang SC', 'Hiragino
                      Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei',
                      '微软雅黑', verdana !important; word-wrap:break-word;
                      word-break:break-all;-webkit-text-size-adjust:none
                      !important;text-size-adjust:none !important;">
                      <tr>
                        <td style="padding:0; box-sizing: border-box;
                          width: 38px;" width="38"> <img
                            style="vertical-align:middle; width: 38px;
                            height: 38px; border-radius:50%;"
                            src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png"
                            moz-do-not-send="true" width="38"
                            height="38"> </td>
                        <td style="padding: 0 0 0 10px; color: #31353b;">
                          <div style="font-size: 16px;font-weight:bold;
                            width:100%; white-space: nowrap;
                            overflow:hidden;text-overflow: ellipsis;">0049003208</div>
                        </td>
                      </tr>
                      <tr width="100%" style="font-size: 14px
                        !important; width: 100%;">
                        <td colspan="2" style="padding:10px 0 0 0;
                          font-size:14px !important; width: 100%;">
                          <div style="width: 100%;font-size: 14px
                            !important;word-wrap:break-word;word-break:break-all;">0049003208@znv.com</div>
                        </td>
                      </tr>
                    </tbody>
                  </table>
                </a> </div>
            </div>
            <div style="font-size:12px;color:#b5b9bd;line-height:18px;">
              <span>签名由</span> <a style="text-decoration:
                none;color:#4196ff;padding:0 5px;"
                href="https://mail.163.com/dashi/dlpro.html?from=mail81"
                moz-do-not-send="true">网易邮箱大师</a> <span>定制</span>
</div>
          </div>
        </div>
        <div class="J-reply"
style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
          <div
style="font-size:12px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
            <span class="mail-date">11/2/2018 11:58</span>,<a
              class="mail-to"
              style="text-decoration:none;color:#2a83f2;"
              href="mailto:daniel.oliveira.mantovani@gmail.com"
              moz-do-not-send="true">Daniel de Oliveira
              Mantovani&lt;daniel.oliveira.mantovani@gmail.com&gt;</a>
            wrote: </div>
        </div>
        <blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0;
          font-size: 14px; font-family: '微软雅黑';">
          <div>
            <div dir="auto">Please, read about Spark Streaming or Spark
              Structured Streaming. Your web application can easily
              communicate through some API and you won’t have the
              overhead of start a new spark job, which is pretty heavy.</div>
          </div>
          <div><br>
            <div class="gmail_quote">
              <div dir="ltr">On Thu, Nov 1, 2018 at 23:01
                崔苗(数据与人工智能产品开发部) &lt;<a href="mailto:0049003208@znv.com"
                  moz-do-not-send="true">0049003208@znv.com</a>&gt;
                wrote:<br>
              </div>
              <blockquote class="gmail_quote" style="margin:0 0 0
                .8ex;border-left:1px #ccc solid;padding-left:1ex">
                <div>
                  <div style="font-family:微软雅黑,Verdana,&quot;Microsoft
                    Yahei&quot;,SimSun,sans-serif;line-height:1.6">
                    <div>
                      <div> <span> <br>
                        </span> </div>
                      <div><span>Hi, </span></div>
                      <div>we want to execute spark code with out submit
                        application.jar,like this code:</div>
                      <div><br>
                      </div>
                      <div>public static void main(String args[]) throws
                        Exception{
                      </div>
                      <div>
                      </div>
                      <div>        SparkSession spark = SparkSession
                      </div>
                      <div>                .builder()
                      </div>
                      <div>                .master("local[*]")
                      </div>
                      <div>                .appName("spark test")
                      </div>
                      <div>                .getOrCreate();
                      </div>
                      <div>      </div>
                      <div>        Dataset&lt;Row&gt; testData =
                        spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
                      </div>
                      <div>        testData.printSchema();
                      </div>
                      <div>        testData.show();
                      </div>
                      <div>    }</div>
                      <div><br>
                      </div>
                      <div>the above code can work well with idea , do
                        not need to generate jar file and submit , but
                        if we replace <span style="line-height:22.4px">master("local[*]")
                          with </span><span style="line-height:22.4px">master("yarn")
                          , it can't work , so is there a way to use
                          cluster sparkSession like local sparkSession ?
                           we need to dynamically execute spark code in
                          web server according to the different request
                          ,  such as filter request will call
                          dataset.filter() , so there is no
                          application.jar to submit .<br>
                        </span><span style="line-height:22.4px"> </span></div>
                      <div id="m_418094061433616952ntes-pcmac-signature"
style="font-family:'\005fae\008f6f\0096c5\009ed1'">
                        <div
                          style="font-size:14px;padding:0;margin:0;line-height:14px">
                          <div
                            style="padding-bottom:6px;margin-bottom:10px;border-bottom:1px
                            solid #e6e6e6;display:inline-block"> <a
href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=0049003208&amp;uid=0049003208%40znv.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&amp;items=%5B%220049003208%40znv.com%22%5D"
style="display:block;background:#fff;max-width:400px;padding:15px 0 10px
                              0;text-decoration:none;outline:none"
                              target="_blank" moz-do-not-send="true">
                              <table
style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:#9b9ea1;font-size:14px;line-height:1.3"
                                cellpadding="0">
                                <tbody style="font-family:'PingFang
                                  SC','Hiragino Sans GB','WenQuanYi
                                  Micro Hei','Microsoft
Yahei','\005fae\008f6f\0096c5\009ed1',verdana!important;word-wrap:break-word;word-break:break-all">
                                  <tr
                                    class="m_418094061433616952firstRow">
                                    <td
                                      style="padding:0;box-sizing:border-box;width:38px"
                                      width="38"> <img
                                        style="vertical-align:middle;width:38px;height:38px;border-radius:50%"
src="https://mail-online.nosdn.127.net/qiyelogo/defaultAvatar.png"
                                        moz-do-not-send="true"
                                        width="38" height="38"> </td>
                                    <td style="padding:0 0 0
                                      10px;color:#31353b">
                                      <div
style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis">0049003208</div>
                                    </td>
                                  </tr>
                                  <tr width="100%"
                                    style="font-size:14px!important;width:100%">
                                    <td colspan="2" style="padding:10px
                                      0 0
                                      0;font-size:14px!important;width:100%">
                                      <div
style="width:100%;font-size:14px!important;word-wrap:break-word;word-break:break-all">0049003208@znv.com</div>
                                    </td>
                                  </tr>
                                </tbody>
                              </table>
                            </a> </div>
                        </div>
                        <div
                          style="font-size:12px;color:#b5b9bd;line-height:18px">
                          <span>签名由</span> <a
                            style="text-decoration:none;color:#4196ff;padding:0
                            5px"
                            href="https://mail.163.com/dashi/dlpro.html?from=mail81"
                            target="_blank" moz-do-not-send="true">网易邮箱大师</a>
                          <span>定制</span> </div>
                      </div>
                    </div>
                  </div>
                </div>
---------------------------------------------------------------------
                To unsubscribe e-mail: <a
                  href="mailto:user-unsubscribe@spark.apache.org"
                  target="_blank" moz-do-not-send="true">user-unsubscribe@spark.apache.org</a>
              </blockquote>
            </div>
          </div>
          -- <br>
          <div dir="ltr" class="gmail_signature"
            data-smartmail="gmail_signature">
            <div dir="ltr">
              <div>
                <div dir="ltr">
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr"><br>
                      </div>
                      <div dir="ltr">--</div>
                      <div dir="ltr">Daniel de Oliveira Mantovani</div>
                      <div><span style="font-size:12.8px">Perl
                          Evangelist/Data Hacker</span><br>
                      </div>
                      <div dir="ltr"><span style="font-size:12.8px">+1
                          786 459 1341</span></div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </blockquote>
        <!--😀-->
      </div>
---------------------------------------------------------------------
      To unsubscribe e-mail: <a class="moz-txt-link-abbreviated" href="mailto:user-unsubscribe@spark.apache.org">user-unsubscribe@spark.apache.org</a>
    </blockquote>
    <br>
  </body>
</html>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message