spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-6330) newParquetRelation gets incorrect FileSystem
Date Mon, 16 Mar 2015 19:20:38 GMT

     [ https://issues.apache.org/jira/browse/SPARK-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron Davidson resolved SPARK-6330.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.3.1
         Assignee: Volodymyr Lyubinets

> newParquetRelation gets incorrect FileSystem
> --------------------------------------------
>
>                 Key: SPARK-6330
>                 URL: https://issues.apache.org/jira/browse/SPARK-6330
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.0
>            Reporter: Volodymyr Lyubinets
>            Assignee: Volodymyr Lyubinets
>             Fix For: 1.3.1
>
>
> Here's a snippet from newParquet.scala:
>     def refresh(): Unit = {
>       val fs = FileSystem.get(sparkContext.hadoopConfiguration)
>       // Support either reading a collection of raw Parquet part-files, or a collection
of folders
>       // containing Parquet files (e.g. partitioned Parquet table).
>       val baseStatuses = paths.distinct.map { p =>
>         val qualified = fs.makeQualified(new Path(p))
>         if (!fs.exists(qualified) && maybeSchema.isDefined) {
>           fs.mkdirs(qualified)
>           prepareMetadata(qualified, maybeSchema.get, sparkContext.hadoopConfiguration)
>         }
>         fs.getFileStatus(qualified)
>       }.toArray
> If we are running this locally and path points to S3, fs would be incorrect. A fix is
to construct fs for each file separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message