spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Markey <kevin.mar...@oracle.com>
Subject Re: Is there a way to get the current progress of the job?
Date Tue, 01 Apr 2014 18:18:15 GMT
<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    The discussion there hits on the distinction of jobs and stages.&nbsp;
    When looking at one application, there are hundreds of stages,
    sometimes thousands.&nbsp; Depends on the data and the task.&nbsp; And the UI
    seems to track stages.&nbsp; And one could independently track them for
    such a job.&nbsp; But what if -- as occurs in another application --
    there's only one or two stages, but lots of data passing through
    those 1 or 2 stages?<br>
    <br>
    Kevin Markey<br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 04/01/2014 09:55 AM, Mark Hamstra
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAAsvFPmGyPM5cj-49L9cf90g=Q0GLrdyORhpv=eBOcz4JrpVxw@mail.gmail.com"
      type="cite">
      <div dir="ltr">Some related discussion:&nbsp;<a moz-do-not-send="true"
          href="https://github.com/apache/spark/pull/246">https://github.com/apache/spark/pull/246</a></div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">On Tue, Apr 1, 2014 at 8:43 AM, Philip
          Ogren <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:philip.ogren@oracle.com" target="_blank">philip.ogren@oracle.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi DB,<br>
            <br>
            Just wondering if you ever got an answer to your question
            about monitoring progress - either offline or through your
            own investigation. &nbsp;Any findings would be appreciated.<br>
            <br>
            Thanks,<br>
            Philip
            <div class="HOEnZb">
              <div class="h5"><br>
                <br>
                On 01/30/2014 10:32 PM, DB Tsai wrote:<br>
                <blockquote class="gmail_quote" style="margin:0 0 0
                  .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  Hi guys,<br>
                  <br>
                  When we're running a very long job, we would like to
                  show users the current progress of map and reduce job.
                  After looking at the api document, I don't find
                  anything for this. However, in Spark UI, I could see
                  the progress of the task. Is there anything I miss?<br>
                  <br>
                  Thanks.<br>
                  <br>
                  Sincerely,<br>
                  <br>
                  DB Tsai<br>
                  Machine Learning Engineer<br>
                  Alpine Data Labs<br>
                  --------------------------------------<br>
                  Web: <a moz-do-not-send="true"
                    href="http://alpinenow.com/" target="_blank">http://alpinenow.com/</a><br>
                </blockquote>
                <br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>

Mime
View raw message