spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: Spark-shell return results when the job is executing?
Date Tue, 02 Sep 2014 14:38:36 GMT
Spark-shell, or any other Spark application, returns the full results of
the job until it has finished executing. You could add a hook for it to
write partial results to a file, but you may want to do so sparingly to
incur fewer I/Os. If you have a large file and the result contains many
lines, it is unlikely to fully fit in memory anyway, so it's probably not a
bad idea to just write your results to a file in batches while the
application is still running.

-Andrew


2014-09-01 22:16 GMT-07:00 Hao Wang <wh.sjtu@gmail.com>:

> Hi, all
>
> I am wondering if I use Spark-shell to scan a large file to obtain lines
> containing "error", whether the shell returns results while the job is
> executing, or the job has been totally finished.
>
> Regards,
> Wang Hao(王灏)
>
> CloudTeam | School of Software Engineering
> Shanghai Jiao Tong University
> Address:800 Dongchuan Road, Minhang District, Shanghai, 200240
> Email:wh.sjtu@gmail.com
>

Mime
View raw message