ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Spark data frames integration merged
Date Fri, 29 Dec 2017 21:22:49 GMT

Great news! We completed and merged first part of integration with Spark
data frames [1]. It contains implementation of Spark data source which
allows to use DataFrame API to query Ignite data, as well as join it with
other data frames originated from different sources.

Next planned steps are the following:
- Implement custom execution strategy to avoid transferring data from
Ignite to Spark when possible [2]. This should give serious performance
improvement in cases when only Ignite tables participate in a query.
- Implement ability to save a data frame into Ignite via DataFrameWrite API

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://issues.apache.org/jira/browse/IGNITE-7077
[3] https://issues.apache.org/jira/browse/IGNITE-7337

Nikolay Izhikov, thanks for the contribution and for all the hard work!


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message