spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Commented] (SPARK-4687) SparkContext#addFile doesn't keep file folder information
Date Tue, 06 Jan 2015 00:37:35 GMT


Patrick Wendell commented on SPARK-4687:

I spent some more time looking at this and talking with [~sandyr] and [~joshrosen]. I think
having some limited version of this is fine given that, from what I can tell, this is pretty
difficult to implement outside of Spark. I am going to post further comments on the JIRA.

> SparkContext#addFile doesn't keep file folder information
> ---------------------------------------------------------
>                 Key: SPARK-4687
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Jimmy Xiang
> Files added with SparkContext#addFile are loaded with Utils#fetchFile before a task starts.
However, Utils#fetchFile puts all files under the Spart root on the worker node. We should
have an option to keep the folder information. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message