spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eron Wright <>
Subject [sample code] deeplearning4j for Spark ML (@DeveloperAPI)
Date Mon, 08 Jun 2015 16:20:03 GMT

The deeplearning4j framework provides a variety of distributed, neural network-based learning
algorithms, including convolutional nets, deep auto-encoders, deep-belief nets, and recurrent
nets.      We’re working on integration with the Spark ML pipeline, leveraging the developer
API.   This announcement is to share some code and get feedback from the Spark community.

The integration code is located in the dl4j-spark-ml module in the deeplearning4j repository.

Major aspects of the integration work:
ML algorithms.  To bind the dl4j algorithms to the ML pipeline, we developed a new classifier
and a new unsupervised learning estimator.   
ML attributes. We strove to interoperate well with other pipeline components.   ML Attributes
are column-level metadata enabling information sharing between pipeline components.    See
here how the classifier reads label metadata from a column provided by the new StringIndexer.
Large binary data.  It is challenging to work with large binary data in Spark.   An effective
approach is to leverage PrunedScan and to carefully control partition sizes.  Here we explored
this with a custom data source based on the new relation API.   
Column-based record readers.  Here we explored how to construct rows from a Hadoop input split
by composing a number of column-level readers, with pruning support.
UDTs.   With Spark SQL it is possible to introduce new data types.   We prototyped an experimental
Tensor type, here.
Spark Package.   We developed a spark package to make it easy to use the dl4j framework in
spark-shell and with spark-submit.      See the deeplearning4j/dl4j-spark-ml repository for
useful snippets involving the sbt-spark-package plugin.
Example code.   Examples demonstrate how the standardized ML API simplifies interoperability,
such as with label preprocessing and feature scaling.   See the deeplearning4j/dl4j-spark-ml-examples
repository for an expanding set of example pipelines.
Hope this proves useful to the community as we transition to exciting new concepts in Spark
SQL and Spark ML.   Meanwhile, we have Spark working with multiple GPUs on AWS and we're looking
forward to optimizations that will speed neural net training even more. 

Eron Wright
Contributor |

View raw message