spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From erp12 <eddie.pantri...@gmail.com>
Subject Pickling Keras models for use in UDFs
Date Thu, 03 May 2018 15:26:57 GMT
I would like to create a Spark UDF which returns the a prediction made with a
trained Keras model. Keras models are not typically pickle-able, however I
have used the monkey patch approach to making Keras models pickle-able, as
described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html

This allows for models to be sent from the PySpark driver to the workers,
however the worker python processes do not have the monkey patched Model
class, and thus cannot properly un-pickle the models. To fix this issue, I
know I must call the monkey patching function (make_keras_picklable()) once
on each worker, however I have been unable to figure out how to do this.

I am curious to hear if anyone has a fix for this issue, or would like to
offer an alternative way to make predictions with a Keras model within a
Spark UDF. 

Here is a Stack Overflow question with more details:
https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark

Thank you!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message