spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yucai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-25207) Case-insensitve field resolution for filter pushdown when reading Parquet
Date Thu, 23 Aug 2018 07:57:00 GMT
yucai created SPARK-25207:
-----------------------------

             Summary: Case-insensitve field resolution for filter pushdown when reading Parquet
                 Key: SPARK-25207
                 URL: https://issues.apache.org/jira/browse/SPARK-25207
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: yucai


Currently, filter pushdown will not work if Parquet schema and Hive metastore schema are in
different letter cases even spark.sql.caseSensitive is false.

Like the below case:
{code:java}
spark.range(10).write.parquet("/tmp/data")
sql("DROP TABLE t")
sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
sql("select * from t where id > 0").show{code}
No filter will be pushed down.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message