spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshuata <joshaspl...@gmail.com>
Subject Re: Task not serializable: java.io.NotSerializableException: org.json4s.Serialization$$anon$1
Date Tue, 19 Jul 2016 20:51:25 GMT
It looks like the problem is that the parse function non-serializeable. This
is most likely because the formats variable is local to the ParseJson
object, and therefore not globally accessible to the cluster. Generally this
problem can be solved by moving the variable inside the closure so that it
is distributed to each worker.

In this specific instance, it makes far more sense to use the  json
datasource
<http://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets>  
provided by newer versions of Spark.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Task-not-serializable-java-io-NotSerializableException-org-json4s-Serialization-anon-1-tp8233p27359.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message