spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <>
Subject [jira] [Created] (SPARK-22939) registerFunction also accepts Spark UDF
Date Tue, 02 Jan 2018 15:34:00 GMT
Xiao Li created SPARK-22939:

             Summary: registerFunction also accepts Spark UDF 
                 Key: SPARK-22939
             Project: Spark
          Issue Type: Bug
          Components: PySpark, SQL
    Affects Versions: 2.3.0
            Reporter: Xiao Li

import random
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType, StringType
random_udf = udf(lambda: int(random.random() * 100), IntegerType()).asNondeterministic()
spark.catalog.registerFunction("random_udf", random_udf, StringType())
spark.sql("SELECT random_udf()").collect()

We will get the following error.
Py4JError: An error occurred while calling o29.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(
	at py4j.reflection.ReflectionEngine.getMethod(
	at py4j.Gateway.invoke(
	at py4j.commands.AbstractCommand.invokeMethod(
	at py4j.commands.CallCommand.execute(

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message