spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Is there setup and cleanup function in spark?
Date Tue, 18 Nov 2014 02:28:01 GMT
Hi,

On Fri, Nov 14, 2014 at 2:49 PM, Jianshi Huang <jianshi.huang@gmail.com>
wrote:

> Ok, then we need another trick.
>
> let's have an *implicit lazy var connection/context* around our code. And
> setup() will trigger the eval and initialization.
>

Due to lazy evaluation, I think having setup/teardown is a bit tricky. In
particular teardown, because it is not easy to execute code after all
computation is done. You can check
http://apache-spark-user-list.1001560.n3.nabble.com/Keep-state-inside-map-function-tp10968p11009.html
for an example of what worked for me.

Tobias

Mime
View raw message