spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Hadoop configuration for checkpointing
Date Tue, 04 Nov 2014 18:49:13 GMT
Let me crash this thread to suggest this *might* be related to this
problem I'm trying to solve:

Basically the question there is: this blank Configuration object gets
made on the driver in the saveAsNewAPIHadoopFiles call, and seems to
need to be serialized to use it in foreachRDD. This fails for me and
at least 2 other users I know. But I feel like I am missing something.

If you're investigating handling of Configuration when enabling
checkpointing with the getOrCreate method, have a look and see if you
have any comments vis-a-vis this JIRA.

On Tue, Nov 4, 2014 at 5:48 PM, Marcelo Vanzin <> wrote:
> On Tue, Nov 4, 2014 at 9:34 AM, Cody Koeninger <> wrote:
>> 2.  Is there a reason StreamingContext.getOrCreate defaults to a blank
>> hadoop configuration rather than
>> org.apache.spark.deploy.SparkHadoopUtil.get.conf,
>> which would pull values from spark config?
> This is probably something I overlooked when I changed the rest of the
> code to use SparkHadoopUtil.get.conf. Feel free to send a PR for it,
> we should get it into 1.2 so that all code creates configuration
> objects in a consistent way.
> --
> Marcelo
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message