spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Friedman <>
Subject Re: action progress in ipython notebook?
Date Mon, 29 Dec 2014 21:24:43 GMT

> On Dec 29, 2014, at 1:52 PM, Matei Zaharia <> wrote:
> Hey Eric, sounds like you are running into several issues, but thanks for reporting them.
Just to comment on a few of these:
>> I'm not seeing RDDs or SRDDs cached in the Spark UI. That page remains empty despite
my calling cache().
> This is expected until you compute the RDDs the first time and they actually get cached,
though I can see why it would be confusing. They don't get registered with the UI until this

Yes, I'm aware of this. In the cases I observed, I saw nothing even after a collect() action.
I'll attempt to reproduce as PEBCAK is always a possibility and will log a jira if I'm able.

>> I think that attempts to access a directory of parquet files still requires reading
the schema from the footer of every file. Painfully slow for terabytes of data.
> This is also expected unfortunately due to the way Parquet works, but if you use Spark
SQL to read these, the metadata gets cached after the first time you access a particular file.

It would be helpful if one could "hint" the scheme for a homogeneous data set, by indicating
that the footer of any single file represents truth for all files in the directory. This is
certainly the norm and is actually pretty similar to how inferSchema works when it take()s
a single row from an RDD. If I tried to slip a heterogeneous set in there, well, shame on

>> Exceptions are still often reflective of a symptom rather than a root cause. For
example, I had a join that was blowing up, but it was variously reported as insufficient kryro
buffers and even an AST error in the SQL parser. 
>> Saving an SRDD to a table in Hive doesn't work. I had to sneak it in by saving to
a file and the creating an external table.
> Yeah, unfortunately this is a bug in 1.2.0 (
It should be fixed for 1.2.1.
>> In interactive work, it would be nice if I could interrupt the current job without
killing the whole session.
> You can actually do this with the "kill" button for that stage on the application web
UI. It's a bit non-obvious but it does work.
> Anyway, thanks for reporting this stuff. Don't be afraid to open JIRAs for such issues
and for usability suggestions, especially if you have a way to reproduce them. Some of the
usability things are "obvious" to people who know the UI inside-out but not to anyone else.

Thank you very much.  


> Matei
>> The lower latency potential of Sparrow is also very intriguing. 
>> Getting GraphX for PySpark would be very welcome. 
>> It's easy to find fault, of course. I do want to say again how grateful I am to have
a usable release in 1.2 and look forward to 1.3 and beyond with real excitement. 
>> ----
>> Eric Friedman
>>> On Dec 28, 2014, at 5:40 PM, Patrick Wendell <> wrote:
>>> Hey Eric,
>>> I'm just curious - which specific features in 1.2 do you find most
>>> help with usability? This is a theme we're focusing on for 1.3 as
>>> well, so it's helpful to hear what makes a difference.
>>> - Patrick
>>> On Sun, Dec 28, 2014 at 1:36 AM, Eric Friedman
>>> <> wrote:
>>>> Hi Josh,
>>>> Thanks for the informative answer. Sounds like one should await your changes
>>>> in 1.3. As information, I found the following set of options for doing the
>>>> visual in a notebook.
>>>> On Dec 27, 2014, at 4:07 PM, Josh Rosen <> wrote:
>>>> The console progress bars are implemented on top of a new stable "status
>>>> API" that was added in Spark 1.2.  It's possible to query job progress using
>>>> this interface (in older versions of Spark, you could implement a custom
>>>> SparkListener and maintain the counts of completed / running / failed tasks
>>>> / stages yourself).
>>>> There are actually several subtleties involved in implementing "job-level"
>>>> progress bars which behave in an intuitive way; there's a pretty extensive
>>>> discussion of the challenges at
>>>> Also, check out the pull request for the console progress bars for an
>>>> interesting design discussion around how they handle parallel stages:
>>>> I'm not sure about the plumbing that would be necessary to display live
>>>> progress updates in the IPython notebook UI, though.  The general pattern
>>>> would probably involve a mapping to relate notebook cells to Spark jobs (you
>>>> can do this with job groups, I think), plus some periodic timer that polls
>>>> the driver for the status of the current job in order to update the progress
>>>> bar.
>>>> For Spark 1.3, I'm working on designing a REST interface to accesses this
>>>> type of job / stage / task progress information, as well as expanding the
>>>> types of information exposed through the stable status API interface.
>>>> - Josh
>>>> On Thu, Dec 25, 2014 at 10:01 AM, Eric Friedman <>
>>>> wrote:
>>>>> Spark 1.2.0 is SO much more usable than previous releases -- many thanks
>>>>> to the team for this release.
>>>>> A question about progress of actions.  I can see how things are
>>>>> progressing using the Spark UI.  I can also see the nice ASCII art animation
>>>>> on the spark driver console.
>>>>> Has anyone come up with a way to accomplish something similar in an
>>>>> iPython notebook using pyspark?
>>>>> Thanks
>>>>> Eric
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message