spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Narrell <>
Subject Re: Serialized 3rd party libs
Date Tue, 02 Sep 2014 18:33:53 GMT

Thanks for point this out.  I’d have to experiment with the mapPartitions method, but you’re
right, this seems to address this issue directly.  I’m also connecting to Zookeeper to retrieve
SparkConf parameters.  I run into the same issue with my Zookeeper driver, however, this is
before any Spark contexts are created, and I assume before the job is partitioned.  


On Sep 2, 2014, at 11:00 AM, Sean Owen <> wrote:

> The problem is not using the drivers per se, but writing your
> functions in a way that you are trying to serialize them. You can't
> serialize them, and indeed don't want to.  Instead your code needs to
> reopen connections and so forth when the function is instantiated on
> the remote worker.
> static variables are a crude way to do that, probably too crude in general.
> No, there's certainly no reason you can't access these things in Spark.
> Since it answers exactly this point, I don't mind promoting today's blog post:
> ... which repeats Tobias's good formulation of how to deal with things
> like drivers in an efficient way that doesn't trip over serialization.
> On Tue, Sep 2, 2014 at 5:45 PM, Matt Narrell <> wrote:
>> Hello,
>> I’m using Spark streaming to aggregate data from a Kafka topic in sliding windows.
 Usually we want to persist this aggregated data to a MongoDB cluster, or republish to a different
Kafka topic.  When I include these 3rd party drivers, I usually get a NotSerializableException
due to the parallelization of the job.  To side step this, I’ve used static class variables
which seem to help, e.g., I can run my jobs.
>> Is this the proper way to provide 3rd party libs to Spark jobs?
>> Does having these drivers declared as static prohibit me from parallelizing my job?
>> Is this even a proper way to design jobs?
>> An alternative (I assume) would be to aggregate my data into HDFS and have another
process (perhaps non-Spark?) consume it and republish/persist?
>> Thanks,
>> Matt
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message