spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-28240) Fix Arrow tests to pass with Python 2.7 and latest PyArrow and Pandas in PySpark
Date Wed, 03 Jul 2019 08:05:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-28240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-28240:
------------------------------------

    Assignee:     (was: Apache Spark)

> Fix Arrow tests to pass with Python 2.7 and latest PyArrow and Pandas in PySpark
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-28240
>                 URL: https://issues.apache.org/jira/browse/SPARK-28240
>             Project: Spark
>          Issue Type: Test
>          Components: PySpark
>    Affects Versions: 3.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>
> {code}
> ======================================================================
> FAIL: test_createDataFrame_with_incorrect_schema (pyspark.sql.tests.test_arrow.ArrowTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/hyukjin.kwon/workspace/forked/spark/python/pyspark/sql/tests/test_arrow.py",
line 275, in test_createDataFrame_with_incorrect_schema
>     self.spark.createDataFrame(pdf, schema=wrong_schema)
> AssertionError: "integer.*required.*got.*str" does not match "('Exception thrown when
converting pandas.Series (object) to Arrow Array (int32). It can be caused by overflows or
other unsafe conversions warned by Arrow. Arrow safe type check can be disabled by using SQL
config `spark.sql.execution.pandas.arrowSafeTypeConversion`.', ArrowTypeError('an integer
is required',))"
> ======================================================================
> FAIL: test_createDataFrame_with_incorrect_schema (pyspark.sql.tests.test_arrow.EncryptionArrowTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/hyukjin.kwon/workspace/forked/spark/python/pyspark/sql/tests/test_arrow.py",
line 275, in test_createDataFrame_with_incorrect_schema
>     self.spark.createDataFrame(pdf, schema=wrong_schema)
> AssertionError: "integer.*required.*got.*str" does not match "('Exception thrown when
converting pandas.Series (object) to Arrow Array (int32). It can be caused by overflows or
other unsafe conversions warned by Arrow. Arrow safe type check can be disabled by using SQL
config `spark.sql.execution.pandas.arrowSafeTypeConversion`.', ArrowTypeError('an integer
is required',))"
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message