systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deron Eriksson <deroneriks...@gmail.com>
Subject Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Date Tue, 23 Aug 2016 22:51:23 GMT
To simplify release candidate validation, I would like to propose that the
distribution profile only builds the following 7 (out of the current
included 10) artifacts:

systemml-0.11.0-incubating-SNAPSHOT-javadoc.jar
systemml-0.11.0-incubating-SNAPSHOT-sources.jar
systemml-0.11.0-incubating-SNAPSHOT-src.tar.gz
systemml-0.11.0-incubating-SNAPSHOT-src.zip
systemml-0.11.0-incubating-SNAPSHOT-standalone.tar.gz (rename w/o
"-standalone")
systemml-0.11.0-incubating-SNAPSHOT-standalone.zip (rename w/o
"-standalone")
systemml-0.11.0-incubating-SNAPSHOT.jar

The following could still be built using maven profiles but would not be in
the distribution profile:

systemml-0.11.0-incubating-SNAPSHOT-standalone.jar
systemml-0.11.0-incubating-SNAPSHOT.tar.gz (also rename)
systemml-0.11.0-incubating-SNAPSHOT.zip (also rename)

This would decrease the number of our artifacts by 30% which means that we
can validate the release faster, and the release candidate will also be
more likely to pass external validation/voting.

Deron


On Thu, Aug 18, 2016 at 12:05 AM, Berthold Reinwald <reinwald@us.ibm.com>
wrote:

> This makes sense. Couple of comments.
>
> - wrt SystemML on Spark 1.x, SystemML 0.11 target date should be the
> latest in Oct. Ideally it should be earlier - maybe September - depending
> on community demand as it will contain bug fixes for features introduced
> in 0.10. The master branch will stay on Spark 1.x til then.  New features
> can be integrated but as they are still partial, they will be marked as
> experimental.
>
> - wrt SystemML on Spark 2.0, as the timeline is fairly short, instead of
> bi-monthly I'd suggest more frequent sync with the branch on Spark 2.0
>
> - Comes early October, we should switch the master to Spark 2.0, and
> immediately create a release for 2.0.
>
> - wrt Roadmap 0.11 items, except for Frame support and MLContext and may
> be DL lib, all the other features should be marked experimental.
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinwald@us.ibm.com
>
>
>
> From:   dusenberrymw@gmail.com
> To:     dev@systemml.incubator.apache.org
> Date:   08/17/2016 04:52 PM
> Subject:        Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
>
>
>
> +1
>
> This sounds like a good plan to allow us to continue supporting the Spark
> 1.x line in the short term, with a plan for moving to Spark 2.x support
> soon.
>
> -Mike
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Aug 17, 2016, at 2:59 PM, Deron Eriksson <deroneriksson@gmail.com>
> wrote:
> >
> > +1
> > Continuing to support Spark 1.4/1.6 for now while setting a cutover date
> > for 2.0 sounds like a great idea. This allows for the creation of a
> really
> > solid release for 1.x, which greatly benefits SystemML users using Spark
> > 1.x. It also gives these users a general date that they can use to plan
> > migration to Spark 2.0 when that becomes the SystemML standard so that
> they
> > can benefit from the latest improvements to the project.
> >
> > Deron
> >
> >
> >> On Wed, Aug 17, 2016 at 2:32 PM, Acs S <acs_s@yahoo.com.invalid> wrote:
> >>
> >> Seems, mail is not retaining format. I am attaching same text through
> PDF
> >> file.
> >> If there is any other better option please let me know.
> >>
> >>
> >> -Arvind
> >>
> >>
> >>
> >>
> >>
> >> ----- Forwarded Message -----
> >> *From:* Acs S <acs_s@yahoo.com.INVALID>
> >> *To:* Dev <dev@systemml.incubator.apache.org>
> >> *Sent:* Wednesday, August 17, 2016 2:18 PM
> >> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
> >>
> >>
> >> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
> >> uptodate with latest version of Spark. This brings us a challenge to
> >> support our consumers until they move to Spark 2.0.Based on some
> >> brainstorming, I can propose following options to keep SystemML being
> >> supported on latest Spark version quickly.
> >>
> >> Supporting SystemML on Spark 1.x We can continue to support SystemML on
> >> Spark 1.x code base for short period of time by adding fixes and
> features
> >> on main branch.  We will release SystemML with support to Spark 1.x
> next
> >> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st
> 2016)
> >> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
> >> 2.0, we can make SystemML on Spark 2.0 immediately based on branch
> created
> >> on top of latest master branch code. Glen has some prototype code to
> >> transform SystemML code to be compatible with Spark 2.0, he can merge
> his
> >> code with new branch targeted to support SystemML on Spark 2.0 This
> would
> >> be "Preview" version code, and we can update it on frequent basis (on
> >> bi-monthy basis).  Supporting SystemML on Spark 2.0 We will have full
> >> support of SystemML on Spark 2.0 before end of year 2016. We will
> formalize
> >> release date by end of Sept 2016. At the same time we will discuss if
> we
> >> can move support of SystemML on Spark 1.x to maintenance mode (Only
> >> required bug fixes will be merged from main branch) or we need to
> support
> >> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
> >> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
> >> Learning (Library of Network layers?) - Frame - New MLContext API  -
> >> Python DSL integration (Preview) - Compressed Linear Algebra (Preview)
> -
> >> Hydra R integration - New Algorithms (?)
> >> 0.12 (Spark 2.0)  (Targeted to 4Q 2016) - GPU support (Local
> >> mode/Distributed mode?)
> >> - New Algorithms (?)
> >> Please feel free to comment on support and roadmap points.
> >>
> >>
> >> -Arvind
> >>
> >>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message