spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Pentreath <nick.pentre...@gmail.com>
Subject Re: [sample code] deeplearning4j for Spark ML (@DeveloperAPI)
Date Thu, 11 Jun 2015 05:48:29 GMT
Looks very interesting, thanks for sharing this.

I haven't had much chance to do more than a quick glance over the code.
Quick question - are the Word2Vec and GLOVE implementations fully parallel
on Spark?

On Mon, Jun 8, 2015 at 6:20 PM, Eron Wright <ewright@live.com> wrote:

>
> The deeplearning4j framework provides a variety of distributed, neural
> network-based learning algorithms, including convolutional nets, deep
> auto-encoders, deep-belief nets, and recurrent nets.      We’re working on
> integration with the Spark ML pipeline, leveraging the developer API.
> This announcement is to share some code and get feedback from the Spark
> community.
>
> The integration code is located in the dl4j-spark-ml module
> <https://github.com/deeplearning4j/deeplearning4j/tree/master/deeplearning4j-scaleout/spark/dl4j-spark-ml>
in
> the deeplearning4j repository.
>
> Major aspects of the integration work:
>
>    1. *ML algorithms.*  To bind the dl4j algorithms to the ML pipeline,
>    we developed a new classifier
>    <https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-scaleout/spark/dl4j-spark-ml/src/main/scala/org/deeplearning4j/spark/ml/classification/MultiLayerNetworkClassification.scala>
and
>    a new unsupervised learning estimator
>    <https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-scaleout/spark/dl4j-spark-ml/src/main/scala/org/deeplearning4j/spark/ml/Unsupervised.scala>.
>
>    2. *ML attributes.* We strove to interoperate well with other pipeline
>    components.   ML Attributes are column-level metadata enabling information
>    sharing between pipeline components.    See here
>    <https://github.com/deeplearning4j/deeplearning4j/blob/4d33302dd8a792906050eda82a7d50ff77a8d957/deeplearning4j-scaleout/spark/dl4j-spark-ml/src/main/scala/org/deeplearning4j/spark/ml/classification/MultiLayerNetworkClassification.scala#L89>
how
>    the classifier reads label metadata from a column provided by the new
>    StringIndexer
>    <http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc4-docs/api/scala/index.html#org.apache.spark.ml.feature.StringIndexer>
>    .
>    3. *Large binary data.*  It is challenging to work with large binary
>    data in Spark.   An effective approach is to leverage PrunedScan and to
>    carefully control partition sizes.  Here
>    <https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-scaleout/spark/dl4j-spark-ml/src/main/scala/org/deeplearning4j/spark/sql/sources/lfw/LfwRelation.scala>
we
>    explored this with a custom data source based on the new relation API.
>    4. *Column-based record readers.*  Here
>    <https://github.com/deeplearning4j/deeplearning4j/blob/b237385b56d42d24bd3c99d1eece6cb658f387f2/deeplearning4j-scaleout/spark/dl4j-spark-ml/src/main/scala/org/deeplearning4j/spark/sql/sources/lfw/LfwRelation.scala#L96>
we
>    explored how to construct rows from a Hadoop input split by composing a
>    number of column-level readers, with pruning support.
>    5. *UDTs*.   With Spark SQL it is possible to introduce new data
>    types.   We prototyped an experimental Tensor type, here
>    <https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-scaleout/spark/dl4j-spark-ml/src/main/scala/org/deeplearning4j/spark/sql/types/tensors.scala>
>    .
>    6. *Spark Package.*   We developed a spark package to make it easy to
>    use the dl4j framework in spark-shell and with spark-submit.      See the
>    deeplearning4j/dl4j-spark-ml
>    <https://github.com/deeplearning4j/dl4j-spark-ml> repository for
>    useful snippets involving the sbt-spark-package plugin.
>    7. *Example code.*   Examples demonstrate how the standardized ML API
>    simplifies interoperability, such as with label preprocessing and feature
>    scaling.   See the deeplearning4j/dl4j-spark-ml-examples
>    <https://github.com/deeplearning4j/dl4j-spark-ml-examples> repository
>    for an expanding set of example pipelines.
>
> Hope this proves useful to the community as we transition to exciting new
> concepts in Spark SQL and Spark ML.   Meanwhile, we have Spark working
> with multiple GPUs on AWS <http://deeplearning4j.org/gpu_aws.html> and
> we're looking forward to optimizations that will speed neural net training
> even more.
>
> Eron Wright
> Contributor | deeplearning4j.org
>
>

Mime
View raw message