spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bago Amirbekian (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-22232) Row objects in pyspark using the `Row(**kwars)` syntax do not get serialized/deserialized properly
Date Tue, 10 Oct 2017 02:53:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Bago Amirbekian updated SPARK-22232:
------------------------------------
    Description: 
The fields in a Row object created from a dict (ie {{Row(**kwargs)}}) should be accessed by
field name, not by position because `Row.__new__` sorts the fields alphabetically by name.
It seems like this promise is not being honored when these Row objects are shuffled. I've
included an example to help reproduce the issue.



{{
from pyspark.sql.types import *
from pyspark.sql import *

def toRow(i):
  return Row(a="a", c=3.0, b=2)

schema = StructType([
  StructField("a", StringType(),  False),
  StructField("c", FloatType(), False),
  StructField("b", IntegerType(), False),
])

rdd = sc.parallelize(range(10)).repartition(2).map(lambda i: toRow(i))

# As long as we don't shuffle things work fine.
print rdd.toDF(schema).take(2)

# If we introduce a shuffle we have issues
print rdd.repartition(3).toDF(schema).take(2)
}}


  was:
bq. The fields in a Row object created from a dict (ie `Row(**kwargs)`) should be accessed
by field name, not by position because `Row.__new__` sorts the fields alphabetically by name.
It seems like this promise is not being honored when these Row objects are shuffled. I've
included an example to help reproduce the issue.



{{
from pyspark.sql.types import *
from pyspark.sql import *

def toRow(i):
  return Row(a="a", c=3.0, b=2)

schema = StructType([
  StructField("a", StringType(),  False),
  StructField("c", FloatType(), False),
  StructField("b", IntegerType(), False),
])

rdd = sc.parallelize(range(10)).repartition(2).map(lambda i: toRow(i))

# As long as we don't shuffle things work fine.
print rdd.toDF(schema).take(2)

# If we introduce a shuffle we have issues
print rdd.repartition(3).toDF(schema).take(2)
}}



> Row objects in pyspark using the `Row(**kwars)` syntax do not get serialized/deserialized
properly
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22232
>                 URL: https://issues.apache.org/jira/browse/SPARK-22232
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.2.0
>            Reporter: Bago Amirbekian
>
> The fields in a Row object created from a dict (ie {{Row(**kwargs)}}) should be accessed
by field name, not by position because `Row.__new__` sorts the fields alphabetically by name.
It seems like this promise is not being honored when these Row objects are shuffled. I've
included an example to help reproduce the issue.
> {{
> from pyspark.sql.types import *
> from pyspark.sql import *
> def toRow(i):
>   return Row(a="a", c=3.0, b=2)
> schema = StructType([
>   StructField("a", StringType(),  False),
>   StructField("c", FloatType(), False),
>   StructField("b", IntegerType(), False),
> ])
> rdd = sc.parallelize(range(10)).repartition(2).map(lambda i: toRow(i))
> # As long as we don't shuffle things work fine.
> print rdd.toDF(schema).take(2)
> # If we introduce a shuffle we have issues
> print rdd.repartition(3).toDF(schema).take(2)
> }}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message