spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <>
Subject Re: Spark-shell return results when the job is executing?
Date Tue, 02 Sep 2014 14:38:36 GMT
Spark-shell, or any other Spark application, returns the full results of
the job until it has finished executing. You could add a hook for it to
write partial results to a file, but you may want to do so sparingly to
incur fewer I/Os. If you have a large file and the result contains many
lines, it is unlikely to fully fit in memory anyway, so it's probably not a
bad idea to just write your results to a file in batches while the
application is still running.


2014-09-01 22:16 GMT-07:00 Hao Wang <>:

> Hi, all
> I am wondering if I use Spark-shell to scan a large file to obtain lines
> containing "error", whether the shell returns results while the job is
> executing, or the job has been totally finished.
> Regards,
> Wang Hao(王灏)
> CloudTeam | School of Software Engineering
> Shanghai Jiao Tong University
> Address:800 Dongchuan Road, Minhang District, Shanghai, 200240

View raw message