spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Integration with Apache Ignite
Date Fri, 10 Apr 2015 18:25:01 GMT
There is some work to create an off-heap storage API for Spark. I think
with it, it will be easier to support different storage backends.

https://issues.apache.org/jira/browse/SPARK-6479

With that API in place, rest of the integration should probably just live
outside of Spark in Ignite or as a 3rd party package.


On Fri, Apr 10, 2015 at 5:21 AM, Devl Devel <devl.development@gmail.com>
wrote:

> Hi
>
> Having evaluated the ignite project and it's relationship to spark: Ignite
> vs. Spark
>
> https://wiki.apache.org/incubator/IgniteProposal
>
> The scope of the project are defined. However are there any current or
> future projects planned to have an Ignite file system RDD (like the Hadoop
> RDD) so that Spark can leverage the in-memory file system as an
> alternative/complement to Tachyon for example?
>
> The idea being to use MLLib and other Spark components but with the option
> to use ignite fs.
>
> I'm not an expert on Ignite FS but would appreciate any comments on the
> matter.
>
> Thanks
>
> Devl
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message