spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Khaled Zaouk <>
Subject Re: Pickling Keras models for use in UDFs
Date Fri, 04 May 2018 09:28:21 GMT
Why don't you try to encapsulate your keras model within a wrapper class
(an estimator let's say), and you implement inside this wrapper class the
two functions: __getstate__ and __setstate__

On Thu, May 3, 2018 at 5:27 PM erp12 <> wrote:

> I would like to create a Spark UDF which returns the a prediction made
> with a
> trained Keras model. Keras models are not typically pickle-able, however I
> have used the monkey patch approach to making Keras models pickle-able, as
> described here:
> This allows for models to be sent from the PySpark driver to the workers,
> however the worker python processes do not have the monkey patched Model
> class, and thus cannot properly un-pickle the models. To fix this issue, I
> know I must call the monkey patching function (make_keras_picklable()) once
> on each worker, however I have been unable to figure out how to do this.
> I am curious to hear if anyone has a fix for this issue, or would like to
> offer an alternative way to make predictions with a Keras model within a
> Spark UDF.
> Here is a Stack Overflow question with more details:
> Thank you!
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message