From user-return-74774-apmail-spark-user-archive=spark.apache.org@spark.apache.org Fri May 4 09:28:44 2018 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 67DF3188B8 for ; Fri, 4 May 2018 09:28:44 +0000 (UTC) Received: (qmail 48521 invoked by uid 500); 4 May 2018 09:28:38 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 48354 invoked by uid 500); 4 May 2018 09:28:37 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 48344 invoked by uid 99); 4 May 2018 09:28:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 May 2018 09:28:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 65CA91A176C for ; Fri, 4 May 2018 09:28:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.192 X-Spam-Level: *** X-Spam-Status: No, score=3.192 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 7aCHfsyMZJSQ for ; Fri, 4 May 2018 09:28:33 +0000 (UTC) Received: from mail-lf0-f52.google.com (mail-lf0-f52.google.com [209.85.215.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5D8045F59D for ; Fri, 4 May 2018 09:28:33 +0000 (UTC) Received: by mail-lf0-f52.google.com with SMTP id o123-v6so29948996lfe.8 for ; Fri, 04 May 2018 02:28:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YeImqn/c2wDHTUaP+MYClxowcs4pmEI7shNPMzO83gE=; b=cYBSFlBQDX0GqCpRNmP0TAvOcZgGD143+BzUApzSAhlNtvxb3sBvNY8mGG2XOWRoJL tfxNYqcoGLhKNMO89HnC199gmoYSgo2OMRt9QmDIDIW0bJwQ2MUXDrdfrAQGHs8DFQLN 2I+s7X9a6ttLAiLIGklq5o7cQ/BWGawY9FpG8hzRLT6UQv8DBqszotZYKIJN5D6uf4cg 9ofo5Sdd15SNQ6T9J/ZkUTHewARj9EUSEjSzuPD4nd0F9cFkFGNH666YUyJR26JSprOE v9CWJsPtIx7j6S+IkaY+K8fpSmnLojUzDcqFjvIC0iKtqdcuoNxYR+ZGOkzEpFba7ivh BWnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YeImqn/c2wDHTUaP+MYClxowcs4pmEI7shNPMzO83gE=; b=U/04UGwhdl27plBCMxFJzt52SOWttU185iXWXhfY48H6tuzhXz+60bcO8EHHaCv6eP YH4yhF8Vhcbg9Zw+AkyGTWU5ltqcgg4oKDlslA02teoRmXGwI7pWCkqS/RzZcBavcY4t /bp9MA3y6gGnUq3+uPilXLs81+ZbBI/y6NjBZFgtEb/XILd8YyohVhMnXxObcLsD3LJa Vu0ZH8PnO6BHoaC3/pg6rp322nnnyP5Y4tWYUFNnEvlWOPQgt6nEteMd+nZ+n43tCV/b vEHnYjqPH75XCj1PkKyTzi0WcFi9AjsH6wXzMA7P/13zBBOGU9KcyT2ucMAmGW3ZCWDQ vqXQ== X-Gm-Message-State: ALQs6tAnbDIGSE2djYjpdnXRKDV/g+gczb+YayoGSJfTH+CJ/vAmcC3o yQRhfEuhYzCvd0uluDq38SocpLEuY7mm+OruzSM= X-Google-Smtp-Source: AB8JxZqgqd5UKss1Vx6nPzXYLNpz9zsEBBbJsYVh0xovJpCu8p3WTyQAqiEP2yQOuMD2v3EAyVKq1pMVF8UxSX4WPag= X-Received: by 2002:a19:f106:: with SMTP id p6-v6mr14730335lfh.118.1525426112178; Fri, 04 May 2018 02:28:32 -0700 (PDT) MIME-Version: 1.0 References: <1525361217756-0.post@n3.nabble.com> In-Reply-To: <1525361217756-0.post@n3.nabble.com> From: Khaled Zaouk Date: Fri, 04 May 2018 09:28:21 +0000 Message-ID: Subject: Re: Pickling Keras models for use in UDFs To: eddie.pantridge@gmail.com Cc: user@spark.apache.org Content-Type: multipart/alternative; boundary="0000000000008c2eaa056b5df1dc" --0000000000008c2eaa056b5df1dc Content-Type: text/plain; charset="UTF-8" Why don't you try to encapsulate your keras model within a wrapper class (an estimator let's say), and you implement inside this wrapper class the two functions: __getstate__ and __setstate__ On Thu, May 3, 2018 at 5:27 PM erp12 wrote: > I would like to create a Spark UDF which returns the a prediction made > with a > trained Keras model. Keras models are not typically pickle-able, however I > have used the monkey patch approach to making Keras models pickle-able, as > described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html > > This allows for models to be sent from the PySpark driver to the workers, > however the worker python processes do not have the monkey patched Model > class, and thus cannot properly un-pickle the models. To fix this issue, I > know I must call the monkey patching function (make_keras_picklable()) once > on each worker, however I have been unable to figure out how to do this. > > I am curious to hear if anyone has a fix for this issue, or would like to > offer an alternative way to make predictions with a Keras model within a > Spark UDF. > > Here is a Stack Overflow question with more details: > > https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark > > Thank you! > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscribe@spark.apache.org > > --0000000000008c2eaa056b5df1dc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Why don't you try to encapsulate your keras model with= in a wrapper class (an estimator let's say), and you implement inside t= his wrapper class the two functions: __getstate__ and __setstate__
On Thu, May 3, 2018 at 5:27 PM= erp12 <eddie.pantridge@gma= il.com> wrote:
I would like = to create a Spark UDF which returns the a prediction made with a
trained Keras model. Keras models are not typically pickle-able, however I<= br> have used the monkey patch approach to making Keras models pickle-able, as<= br> described here: http://zachmoshe.com/2017/= 04/03/pickling-keras-models.html

This allows for models to be sent from the PySpark driver to the workers, however the worker python processes do not have the monkey patched Model class, and thus cannot properly un-pickle the models. To fix this issue, I<= br> know I must call the monkey patching function (make_keras_picklable()) once=
on each worker, however I have been unable to figure out how to do this.
I am curious to hear if anyone has a fix for this issue, or would like to offer an alternative way to make predictions with a Keras model within a Spark UDF.

Here is a Stack Overflow question with more details:
h= ttps://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-m= odel-for-use-in-pyspark

Thank you!



--
Sent from: http://apache-spark-user-list.1001560= .n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

--0000000000008c2eaa056b5df1dc--