spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-8756) Keep cached information and avoid re-calculating footers in ParquetRelation2
Date Wed, 01 Jul 2015 09:55:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609852#comment-14609852
] 

Apache Spark commented on SPARK-8756:
-------------------------------------

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/7154

> Keep cached information and avoid re-calculating footers in ParquetRelation2
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-8756
>                 URL: https://issues.apache.org/jira/browse/SPARK-8756
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Liang-Chi Hsieh
>
> Currently, in ParquetRelation2, footers are re-read every time refresh() is called. But
we can check if it is possibly changed before we do the reading because reading all footers
will be expensive when there are too many partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message