spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-8756) Keep cached information and avoid re-calculating footers in ParquetRelation2
Date Sun, 19 Jul 2015 11:41:05 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cheng Lian updated SPARK-8756:
------------------------------
    Shepherd: Cheng Lian

> Keep cached information and avoid re-calculating footers in ParquetRelation2
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-8756
>                 URL: https://issues.apache.org/jira/browse/SPARK-8756
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Liang-Chi Hsieh
>
> Currently, in ParquetRelation2, footers are re-read every time refresh() is called. But
we can check if it is possibly changed before we do the reading because reading all footers
will be expensive when there are too many partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message