systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm" <>
Subject Re: [DISCUSS] Project Roadmap
Date Mon, 21 Dec 2015 11:09:06 GMT

>From my perspective, our roadmap for 2016 should cover the following
SystemML engine extensions with regard to runtime (R), optimizer (O), as
well as language and tools (L). Each sub-bullet in the following list will
be further broken down into multiple JIRAs.

R1) Extended Scale-Up Backend
* Support for large dense matrix blocks >16GB
* Extended multi-threaded operations (e.g., ctable, aggregate)
* NUMA-awareness (partitioning and multi-threaded operations)
* Extended update-in-place support

R2) Generalized Matrix Block Library
* Investigation interface design (abstraction)
* Boolean matrices and operations
* Different types of sparse matrix blocks
* Additional physical runtime operators

R3) HW Accelerators / Low-Level Optimizations
* Exploit GPU BLAS libraries (integration)
* Custom GPU kernels for complex operator patterns
* Low-level optimizations (source code gen, compression)

O1) Global Program Optimization
* Global data flow optimization (rewrites, holistic)
* Code motion (for cse, program block merge)
* Advanced loop vectorization (common patterns)
* Advanced function inlining (inlining multi-block functions)
* Extended inter-procedure analysis (independent constant propagation)

O2) Cost Model
* Update memory budgets wrt Spark 1.6 dynamic memory management
* Extended runtime cost model for Spark (incl lazy evaluation)
* Extended execution type selection based on FLOPs

O3) Dynamic Rewrites
* Extended matrix mult chain opt (sparsity, rewrites, ops)
* Rewrites exploiting additional statistics (e.g., min/max)

O4) Optimizer Support R2/R3
* Extended memory estimates for R2/R3
* Type inference for matrix operations
* Extended cost model and operator selection

L1) Extended Spark Interfaces
* Hardening MLContext (config, lazy eval, cleanup)
* Extended Spark ML wrappers for all algorithms
* Investigation of embedded DSL with sufficient optimization scope

L2) New/Extended Builtin Functions
* Second order functions (apply), incl optimizer/runtime support
* Generalization of existing functions from vectors to matrices
* Additional builtin functions (e.g., var, sd, rev, rep, sign, etc)

L3) Extended Dev Tools
* Extended statistics output (e.g., wrt Spark lazy evaluation)
* Extended benchmarking (data generators, test suites, etc)

Once we create the individual JIRAs, we should also include a list of new
algorithms as well as additional documentation guides.


From:	Luciano Resende <>
Date:	11/20/2015 01:50 AM
Subject:	[DISCUSS] Project Roadmap

Now that we are done with our 0.8.0 (non-apache) Release,  and have most of
our infrastructure in place at Apache, I would like to start some
discussion around what are some high level items you see we could be
working on the short/medium term, and start building a Roadmap, so new
contributors can easily find areas of their interest to start contributing.

Let's have items listed on this thread, and once we have our JIRA
available, we start updating it there.


Luciano Resende

  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message