spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pillis Work <pillis.w...@gmail.com>
Subject Re: About Spark job web ui persist(JIRA-969)
Date Mon, 13 Jan 2014 18:43:15 GMT
The listeners in SparkUI which update the counters can trigger saves along
the way.
The save can be on a 500ms delay after the last update, to batch changes.
This solution would not require save on stop().



On Mon, Jan 13, 2014 at 6:15 AM, Tom Graves <tgraves_cs@yahoo.com> wrote:

> So the downside to just saving stuff at the end is that if the app crashes
> or exits badly you don't have anything.   Hadoop has taken the approach of
> saving events along the way.  But Hadoop also uses that history file to
> start where it left off at if something bad happens and it gets restarted.
>  I don't think the latter really applies to spark though.
>
> Does mesos have a history server?
>
> Tom
>
>
>
> On Sunday, January 12, 2014 9:22 PM, Pillis Work <pillis.work@gmail.com>
> wrote:
>
> IMHO from a pure Spark standpoint, I don't know if having a dedicated
> history service makes sense as of now - considering that cluster managers
> have their own history servers. Just showing UI of history runs might be
> too thin a requirement for a full service. Spark should store history
> information that can later be exposed in required ways.
>
> Since each SparkContext is the logical entry and exit point for doing
> something useful in Spark, during its stop(), it should serialize that
> run's statistics into a JSON file - like "sc_run_[name]_[start-time].json".
> When SparkUI.stop() is called, it in turn asks its UI objects (which should
> implement a trait) to provide either a flat or hierarchical Map of String
> key/value pairs. This map (flat, hierarchical) is then serialized to a
> configured path (default being "var/history").
>
> With regards to Mesos or YARN, their applications during shutdown can have
> API to import this Spark history into their history servers - by making API
> calls etc.
>
> This way Spark's history information is persisted independent of cluster
> framework, and cluster frameworks can import the history when/as needed.
> Hope this helps.
> Regards,
> pillis
>
>
>
> On Thu, Jan 9, 2014 at 6:13 AM, Tom Graves <tgraves_cs@yahoo.com> wrote:
>
> > Note that it looks like we are planning on adding support for application
> > specific frameworks to YARN sooner rather then later. There is an initial
> > design up here: https://issues.apache.org/jira/browse/YARN-1530. Note
> > this has not been reviewed yet so changes are likely but gives an idea of
> > the general direction.  If anyone has comments on how that might work
> with
> > SPARK I encourage you to post to the jira.
> >
> > As Sandy mentioned it would be very nice if the solution could be
> > compatible with that.
> >
> > Tom
> >
> >
> >
> > On Wednesday, January 8, 2014 12:44 AM, Sandy Ryza <
> > sandy.ryza@cloudera.com> wrote:
> >
> > Hey,
> >
> > YARN-321 is targeted for the Hadoop 2.4.  The minimum feature set doesn't
> > include application-specific data, so that probably won't be part of 2.4
> > unless other things delay the release for a while.  There are no APIs for
> > it yet and pluggable UIs have been discussed but not agreed upon.  I
> think
> > requirements from Spark could be useful in helping shape what gets done
> > there.
> >
> > -Sandy
> >
> >
> >
> > On Tue, Jan 7, 2014 at 4:13 PM, Patrick Wendell <pwendell@gmail.com>
> > wrote:
> >
> > > Hey Sandy,
> > >
> > > Do you know what the status is for YARN-321 and what version of YARN
> > > it's targeted for? Also, is there any kind of documentation or API for
> > > this? Does it control the presentation of the data itself (e.g. it
> > > actually has its own UI)?
> > >
> > > @Tom - having an optional history server sounds like a good idea.
> > >
> > > One question is what format to use for storing the data and how the
> > > persisted format relates to XML/HTML generation in the live UI. One
> > > idea would be to add JSON as an intermediate format inside of the
> > > current WebUI, and then any JSON page could be persisted and rendered
> > > by the history server using the same code. Once a SparkContext exits
> > > it could dump a series of named paths each with a JSON file. Then the
> > > history server could load those paths and pass them through the second
> > > rendering stage (JSON => XML) to create each page.
> > >
> > > It would be good if SPARK-969 had a good design doc before anyone
> > > starts working on it.
> > >
> > > - Patrick
> > >
> > > On Tue, Jan 7, 2014 at 3:18 PM, Sandy Ryza <sandy.ryza@cloudera.com>
> > > wrote:
> > > > As a sidenote, it would be nice to make sure that whatever done here
> > will
> > > > work with the YARN Application History Server (YARN-321), a generic
> > > history
> > > > server that functions similarly to MapReduce's JobHistoryServer.  It
> > will
> > > > eventually have the ability to store application-specific data.
> > > >
> > > > -Sandy
> > > >
> > > >
> > > > On Tue, Jan 7, 2014 at 2:51 PM, Tom Graves <tgraves_cs@yahoo.com>
> > wrote:
> > > >
> > > >> I don't think you want to save the html/xml files. I would rather
> see
> > > the
> > > >> info saved into a history file in like a json format that could then
> > be
> > > >> re-read and the web ui display the info, hopefully without much
> change
> > > to
> > > >> the UI parts.  For instance perhaps the history server could read
> the
> > > file
> > > >> and populate the appropriate Spark data structures that the web ui
> > > already
> > > >> uses.
> > > >>
> > > >> I would suggest making it so the history server is an optional
> server
> > > and
> > > >> could be run on any node. That way if the load on a particular node
> > > becomes
> > > >> to much it could be moved, but you also could run it on the same
> node
> > as
> > > >> the Master.  All it really needs to know is where to get the history
> > > files
> > > >> from and have access to that location.
> > > >>
> > > >> Hadoop actually has a history server for MapReduce which works very
> > > >> similar to what I mention above.   One thing to keep in minds here
> is
> > > >> security.  You want to make sure that the history files can only be
> > > read by
> > > >> users who have the appropriate permissions.  The history server
> itself
> > > >> could run as  a superuser who has permission to server up the files
> > > based
> > > >> on the acls.
> > > >>
> > > >>
> > > >>
> > > >> On Tuesday, January 7, 2014 8:06 AM, "Xia, Junluan" <
> > > junluan.xia@intel.com>
> > > >> wrote:
> > > >>
> > > >> Hi all
> > > >>          Spark job web ui will not be available when job is over,
> but
> > it
> > > >> is convenient for developer to debug with persisting job web ui. I
> > just
> > > >> come up with draft for this issue.
> > > >>
> > > >> 1.       We could simply save the web page with html/xml
> > > >> format(stages/executors/storages/environment) to certain location
> when
> > > job
> > > >> finished
> > > >>
> > > >> 2.       But it is not easy for user to review the job info with #1,
> > we
> > > >> could build extra job history service for developers
> > > >>
> > > >> 3.       But where will we build this history service? In Driver
> node
> > or
> > > >> Master node?
> > > >>
> > > >> Any suggestions about this improvement?
> > > >>
> > > >> regards,
> > > >> Andrew
> > > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message