spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Cutler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-25147) GroupedData.apply pandas_udf crashing
Date Wed, 22 Aug 2018 17:49:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-25147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589177#comment-16589177
] 

Bryan Cutler commented on SPARK-25147:
--------------------------------------

Works for me on linux with:
Python 3.6.6
pyarrow 0.10.0
pandas 23.4
numpy 1.14.3

Maybe only on MacOS?

> GroupedData.apply pandas_udf crashing
> -------------------------------------
>
>                 Key: SPARK-25147
>                 URL: https://issues.apache.org/jira/browse/SPARK-25147
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.3.1
>         Environment: OS: Mac OS 10.13.6
> Python: 2.7.15, 3.6.6
> PyArrow: 0.10.0
> Pandas: 0.23.4
> Numpy: 1.15.0
>            Reporter: Mike Sukmanowsky
>            Priority: Major
>
> Running the following example taken straight from the docs results in {{org.apache.spark.SparkException:
Python worker exited unexpectedly (crashed)}} for reasons that aren't clear from any logs
I can see:
> {code:java}
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> spark = (
>     SparkSession
>     .builder
>     .appName("pandas_udf")
>     .getOrCreate()
> )
> df = spark.createDataFrame(
>     [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
>     ("id", "v")
> )
> @F.pandas_udf("id long, v double", F.PandasUDFType.GROUPED_MAP)
> def normalize(pdf):
>     v = pdf.v
>     return pdf.assign(v=(v - v.mean()) / v.std())
> (
>     df
>     .groupby("id")
>     .apply(normalize)
>     .show()
> )
> {code}
>  See output.log for [stacktrace|https://gist.github.com/msukmanowsky/b9cb6700e8ccaf93f265962000403f28]. 
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message