spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6315) SparkSQL 1.3.0 (RC3) fails to read parquet file generated by 1.1.1
Date Mon, 16 Mar 2015 17:23:38 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363523#comment-14363523
] 

Cheng Lian commented on SPARK-6315:
-----------------------------------

Hi [~Michael Davies], really sorry for the trouble! [PR #5034| https://github.com/apache/spark/pull/5034]
was opened to fix this issue, and is expected to be included in 1.3.1. If you need this right
away, you may cherry-pick it into your own fork after it's been reviewed and merged (which
should be pretty soon).

> SparkSQL 1.3.0 (RC3) fails to read parquet file generated by 1.1.1
> ------------------------------------------------------------------
>
>                 Key: SPARK-6315
>                 URL: https://issues.apache.org/jira/browse/SPARK-6315
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.0
>            Reporter: Michael Armbrust
>            Assignee: Cheng Lian
>            Priority: Blocker
>
> Parquet files generated by Spark 1.1 have a deprecated representation of the schema.
 In Spark 1.3 we fail to read these files through the new Parquet code path.  We should continue
to read these files until we formally deprecate this representation.
> As a workaround:
> {code}
> SET spark.sql.parquet.useDataSourceApi=false
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message