spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yucai (JIRA)" <>
Subject [jira] [Created] (SPARK-25206) Wrong data may be returned when enable pushdown
Date Thu, 23 Aug 2018 07:41:00 GMT
yucai created SPARK-25206:

             Summary: Wrong data may be returned when enable pushdown
                 Key: SPARK-25206
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.1
            Reporter: yucai

In current Spark 2.3.1, below query returns wrong data silently.
sql("DROP TABLE t")
sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
scala> sql("select * from t where id > 0").show
| ID|

*Root Cause*
Spark pushdowns"{color:#ff0000}ID{color}"), 0: Integer) into parquet,
but {color:#ff0000}ID{color} does not exist in /tmp/data (parquet is case sensitive, it has
{color:#ff0000}id{color} actually).
So no records are returned.

In Spark 2.1, the user will get Exception:
Caused by: java.lang.IllegalArgumentException: Column [ID] was not found in schema!{code}
But in Spark 2.3, they will get the wrong results sliently.


Since SPARK-24716, Spark uses Parquet schema instead of Hive metastore schema to do the pushdown,
perfect for this issue.

[~yuming] , [~cloud_fan], [~smilegator], any thoughts? Should we backport it?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message