spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject previous stage results are not saved?
Date Mon, 17 Oct 2016 23:11:56 GMT
I'm trying out 2.0, and ran a long job with 10 stages, in spark-shell

it seems that after all 10 finished successfully, if I run the last, or the
9th again,
spark reruns all the previous stages from scratch, instead of utilizing the
partial results.

this is quite serious since I can't experiment while making small changes
to the code.

any idea what part of the spark framework might have caused this ?

thanks
Yang

Mime
View raw message