spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: Integration with Apache Ignite
Date Fri, 10 Apr 2015 18:25:01 GMT
There is some work to create an off-heap storage API for Spark. I think
with it, it will be easier to support different storage backends.

With that API in place, rest of the integration should probably just live
outside of Spark in Ignite or as a 3rd party package.

On Fri, Apr 10, 2015 at 5:21 AM, Devl Devel <>

> Hi
> Having evaluated the ignite project and it's relationship to spark: Ignite
> vs. Spark
> The scope of the project are defined. However are there any current or
> future projects planned to have an Ignite file system RDD (like the Hadoop
> RDD) so that Spark can leverage the in-memory file system as an
> alternative/complement to Tachyon for example?
> The idea being to use MLLib and other Spark components but with the option
> to use ignite fs.
> I'm not an expert on Ignite FS but would appreciate any comments on the
> matter.
> Thanks
> Devl

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message