spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <>
Subject [jira] [Commented] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases
Date Fri, 31 Aug 2018 04:26:00 GMT


Wenchen Fan commented on SPARK-25206:

It turns out we need to backport 3 non-trivial PRs to entirely fix the problem, which is risky.
Let's close this JIRA if the problem has been resolved in master.

> wrong records are returned when Hive metastore schema and parquet schema are in different
letter cases
> ------------------------------------------------------------------------------------------------------
>                 Key: SPARK-25206
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.2, 2.3.1
>            Reporter: yucai
>            Priority: Blocker
>              Labels: Parquet, correctness
>         Attachments: image-2018-08-24-18-05-23-485.png, image-2018-08-24-22-33-03-231.png,
image-2018-08-24-22-34-11-539.png, image-2018-08-24-22-46-05-346.png, image-2018-08-25-09-54-53-219.png,
image-2018-08-25-10-04-21-901.png, pr22183.png
> In current Spark 2.3.1, below query returns wrong data silently.
> {code:java}
> spark.range(10).write.parquet("/tmp/data")
> sql("DROP TABLE t")
> sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
> scala> sql("select * from t where id > 0").show
> +---+
> | ID|
> +---+
> +---+
> {code}
> *Root Cause*
> After deep dive, it has two issues, both are related to different letter cases between
Hive metastore schema and parquet schema.
> 1. Wrong column is pushdown.
> Spark pushdowns"{color:#ff0000}ID{color}"), 0: Integer) into
parquet, but {color:#ff0000}ID{color} does not exist in /tmp/data (parquet is case sensitive,
it has {color:#ff0000}id{color} actually).
> So no records are returned.
> Since SPARK-24716, Spark uses Parquet schema instead of Hive metastore schema to do the
pushdown, perfect for this issue.
> 2. Spark SQL returns NULL for a column whose Hive metastore schema and Parquet schema
are in different letter cases, even spark.sql.caseSensitive set to false.
> SPARK-25132 addressed this issue already.
> The biggest difference is, in Spark 2.1, user will get Exception for the same query:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Column [ID] was not found in schema!{code}
> So they will know the issue and fix the query.
> But in Spark 2.3, user will get the wrong results sliently.
> To make the above query work, we need both SPARK-25132 and -SPARK-24716.-
> [~yumwang] , [~cloud_fan], [~smilegator], any thoughts? Should we backport it?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message