spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil <ak...@sigmoidanalytics.com>
Subject Re: serialization issue with mapPartitions
Date Fri, 26 Dec 2014 11:22:08 GMT
You cannot pass your jobConf object inside any of the transformation function
in spark (like map, mapPartitions, etc.) since 
 org.apache.hadoop.mapreduce.Job is not Serializable. You can use
KryoSerializer (See this doc
http://spark.apache.org/docs/latest/tuning.html#data-serialization), We
usually converts the JobConf into ByteArray and pass over the byteArray
object inside the map and from there we creates the jobConf (new variable)
with the data inside byteArray object.


> I should rephrase my question as follows:
> 
> How to use the corresponding Hadoop Configuration of a HadoopRDD in
> defining a function as an input parameter to the MapPartitions function?
> 
> Thanks.
> 
> Ey-Chih Chow





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/serialization-issue-with-mapPartitions-tp20858p20865.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message