spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Pentreath (JIRA)" <>
Subject [jira] [Commented] (SPARK-15575) Remove breeze from dependencies?
Date Fri, 01 Jul 2016 11:26:11 GMT


Nick Pentreath commented on SPARK-15575:

So it seems from the various discussion on SPARK-15581, SPARK-13944, and due to the issue
in SPARK-14438 that there is enough interest and immediacy around this that it may make sense
to review linear algebra within Spark and JVM in general, to refresh the current state with
respect to features, performance, API / Java compat, and license.

I've created a (very initial and WIP) [Google doc|]
to begin discussion and collating the details. Please feel free to discuss further here or
on that doc.

I'm happy to move ahead with the evaluation and performance testing (also happy to split up
the work if desired). 

> Remove breeze from dependencies?
> --------------------------------
>                 Key: SPARK-15575
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>            Reporter: Joseph K. Bradley
> This JIRA is for discussing whether we should remove Breeze from the dependencies of
MLlib.  The main issues with Breeze are Scala 2.12 support and performance issues.
> There are a few paths:
> # Keep dependency.  This could be OK, especially if the Scala version issues are fixed
within Breeze.
> # Remove dependency
> ## Implement our own linear algebra operators as needed
> ## Design a way to build Spark using custom linalg libraries of the user's choice.  E.g.,
you could build MLlib using Breeze, or any other library supporting the required operations.
 This might require significant work.  See [SPARK-6442] for related discussion.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message