spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-4368) Ceph integration?
Date Sat, 24 Jan 2015 14:56:34 GMT


Sean Owen commented on SPARK-4368:

I don't think Spark does anything in particular to support GlusterFS; the message you cite
just says it works without any special support. I haven't heard Ceph come up. Are you suggesting
there is some change that needs to be made to support it? if so I think you should outline
how big the change is. I think the suggestion recently has been that third-party integration
projects belong outside the core project though.

> Ceph integration?
> -----------------
>                 Key: SPARK-4368
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>            Reporter: Serge Smertin
> There is a use-case of storing big number of relatively small BLOB objects (2-20Mb),
which has to have some ugly workarounds in HDFS environments. There is a need to process those
BLOBs close to data themselves, so that's why MapReduce paradigm is good, as it guarantees
data locality.
> Ceph seems to be one of the systems that maintains both of the properties (small files
and data locality) -
I know already that Spark supports GlusterFS -
> So i wonder, could there be an integration with this storage solution and what could
be the effort of doing that? 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message