systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Acs S <>
Subject [DISCUSS] SystemML with Spark 2.0 support and roadmap
Date Wed, 17 Aug 2016 21:18:44 GMT

Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be uptodate with latest
version of Spark. This brings us a challenge to support our consumers until they move to Spark
2.0.Based on some brainstorming, I can propose following options to keep SystemML being supported
on latest Spark version quickly.

 Supporting SystemML on Spark 1.x We can continue to support SystemML on Spark 1.x code base
for short period of time by adding fixes and features on main branch.  We will release SystemML
with support to Spark 1.x next version (0.11) around beginning of Oct 2016 (Lets target for
Oct 1st 2016)
 Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark 2.0, we can make
SystemML on Spark 2.0 immediately based on branch created on top of latest master branch code.
Glen has some prototype code to transform SystemML code to be compatible with Spark 2.0, he
can merge his code with new branch targeted to support SystemML on Spark 2.0 This would be
"Preview" version code, and we can update it on frequent basis (on bi-monthy basis).  Supporting
SystemML on Spark 2.0 We will have full support of SystemML on Spark 2.0 before end of year
2016. We will formalize release date by end of Sept 2016. At the same time we will discuss
if we can move support of SystemML on Spark 1.x to maintenance mode (Only required bug fixes
will be merged from main branch) or we need to support both SystemML on Spark 2.0 and Spark
1.x for some additional time. 
SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep Learning (Library of
Network layers?) - Frame - New MLContext API        - Python DSL integration (Preview)
- Compressed Linear Algebra (Preview) - Hydra R integration - New Algorithms (?)
 0.12 (Spark 2.0)  (Targeted to 4Q 2016) - GPU support (Local mode/Distributed mode?)
 - New Algorithms (?)
Please feel free to comment on support and roadmap points.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message