spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Commented] (SPARK-1529) Support setting spark.local.dirs to a hadoop FileSystem
Date Mon, 21 Apr 2014 19:44:15 GMT


Patrick Wendell commented on SPARK-1529:

[~liancheng] Hey Cheng, the tricky thing here is want to avoid _always_ going through the
HDFS filesystem interface when people are actually using local files. We might need to add
an intermediate abstraction to deal with this. We already do this elsehwere in the code base,
for instance the JobLogger will load an output stream either directly form a file or from
a hadoop file.

One thing to note is that the requirement here is really only for the shuffle files, not for
the other uses. But I realize we currently conflate these inside of Spark so that might not
buy us much. I'll look into this a bit more later.

> Support setting spark.local.dirs to a hadoop FileSystem 
> --------------------------------------------------------
>                 Key: SPARK-1529
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Patrick Wendell
>            Assignee: Cheng Lian
>             Fix For: 1.1.0
> In some environments, like with MapR, local volumes are accessed through the Hadoop filesystem
interface. We should allow setting spark.local.dir to a Hadoop filesystem location. 

This message was sent by Atlassian JIRA

View raw message