spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Tuero <>
Subject Broadcasting Complex Custom Objects
Date Mon, 17 Oct 2016 21:32:00 GMT
Hi guys,

I'm  trying to do a a job with Spark, using Java.

The thing is I need to have an index of words of about 3 GB in each
machine, so I'm trying to broadcast custom objects to represent the index
and the interface with it.
I'm using java standard serialization, so I tried to implement serializable
interface in each class involved, but some objects come from libraries so I
can't go any further.

Is there another way to make it works?
Should I try with Kryo?
Is there a way to work with non-serializable objects?

I use a fat-jar, so the code is available in all workers really. I thing it
should be a way to use it instead of being serializing and deserializing


View raw message