spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: serialization issue with mapPartitions
Date Fri, 26 Dec 2014 01:37:41 GMT
Hi,

On Fri, Dec 26, 2014 at 10:13 AM, ey-chih chow <eychih@hotmail.com> wrote:

> I should rephrase my question as follows:
>
> How to use the corresponding Hadoop Configuration of a HadoopRDD in
> defining
> a function as an input parameter to the MapPartitions function?
>

Well, you could try to pull the `val config = job.getConfiguration()` out
of the function and just use `config` inside the function, hoping that this
one is serializable.

Tobias

Mime
View raw message