spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guy Khazma <Guy.Kha...@ibm.com>
Subject [Spark 3.0] DataSourceV2 FileScan - Hive style partition pruning
Date Mon, 30 Dec 2019 14:41:08 GMT
Hi,

It seems that hive style partition pruning is not working for file based
data sources such as Parquet and ORC.
This causes serious performance degradation for non hive tables.

The reason for that is that the  FileScan
<https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala>
 
abstract class is not aware of the partition and data filters. 
The method for getting the selectedPartitions calls the FileIndex listFiles
method with empty sequence for both - see  here
<https://github.com/apache/spark/blob/5af77410bbb970059d9365b193987e0e44827c20/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala#L74>

.

In the v1 datasource the  FileSourceScanExec
<https://github.com/apache/spark/blob/5af77410bbb970059d9365b193987e0e44827c20/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L160>
 
class gets the partition and data filters and use them to filter unnecessary
partitions by passing them to the listFiles function - see  here
<https://github.com/apache/spark/blob/5af77410bbb970059d9365b193987e0e44827c20/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L210>

.

Are there any ongoing plans to add a support for that?

Thanks,
Guy



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message