spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ehrlich <and...@aehrlich.com>
Subject Re: Changing Spark configuration midway through application.
Date Wed, 10 Aug 2016 16:59:49 GMT
If you're changing properties for the SparkContext, then I believe you will
have to start a new SparkContext with the new properties.

On Wed, Aug 10, 2016 at 8:47 AM, Jestin Ma <jestinwith.an.e@gmail.com>
wrote:

> If I run an application, for example with 3 joins:
>
> [join 1]
> [join 2]
> [join 3]
>
> [final join and save to disk]
>
> Could I change Spark properties in between each join?
>
> [join 1]
> [change properties]
> [join 2]
> [change properties]
> ...
>
> Or would I have to create a separate application with different properties
> for each of the three joins and also save each intermediate join result to
> disk?
>
> Jestin
>

Mime
View raw message