hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From liuxun <neliu...@163.com>
Subject Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
Date Tue, 30 Oct 2018 14:11:34 GMT
HI, community


I updated the design document and submitted it to the hadoop community.
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>

If you have any questions, you can ask them in the email or directly in the document. Thank
you!

I developed the submarine interpreter module in zeppelin. Below is the interface for the system
to run.

1. zeppelin submarine interpreter properties

1.1 System administrators can configure multiple interpreters, and different interpreters
can have different resource configurations.
1.2 Allow different users to use different interpreters for resource allocation and management.

2. zeppelin submarine Tensorflow interpreter

2.1 Users use their own interpreter, First through the %submarine.tensorflow paragraph, To
write python code for tensorflow.
2.2 After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin
will upload the python code to the specified HDFS directory. The submarine is loaded into
the docker container when it is running.

3. zeppelin submarine interpreter

3.1 The user sets the call parameter value of the tensorflow python code, Then enter the job
run command.
3.2 The zeppelin submarine interpreter first checks that all parameters of the submarine run
are set completely. After the check is passed, the implementation of YARN is performed via
submarine.jar.
3.3 The progress and log information of the submarine will be displayed in zeppelin's note.
3.4 You can enter submarine's other commands in the submarine interpreter, view all the jobs
in the submarine, etc.


> 在 2018年10月30日,下午6:40,liuxun <neliuxun@163.com> 写道:
> 
> 
> HI,
> 
> 
> 	I updated the design document and submitted it to the hadoop community.
> 	https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#
<https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>

> 	If you have any questions, you can ask them in the email or directly in the document.
Thank you!
> 
> 	I developed the submarine interpreter module in zeppelin. Below is the interface for
the system to run.
> 
> zeppelin submarine interpreter properties
> 
> System administrators can configure multiple interpreters, and different interpreters
can have different resource configurations.
> Allow different users to use different interpreters for resource allocation and management.
> zeppelin submarine Tensorflow interpreter
> 
> Users use their own interpreter, First through the %submarine.tensorflow paragraph, To
write python code for tensorflow.
> After the user writes the python code of tensorflow, Click the [RUN] button and zeppelin
will upload the python code to the specified HDFS directory. The submarine is loaded into
the docker container when it is running.
> 
> zeppelin submarine interpreter
> 
> The user sets the call parameter value of the tensorflow python code, Then enter the
job run command.
> The zeppelin submarine interpreter first checks that all parameters of the submarine
run are set completely. After the check is passed, the implementation of YARN is performed
via submarine.jar.
> The progress and log information of the submarine will be displayed in zeppelin's note.
> You can enter submarine's other commands in the submarine interpreter, view all the jobs
in the submarine, etc.
> 
> 
> 
>> 在 2018年10月21日,上午6:26,Felix Cheung <felixcheung_m@hotmail.com>
写道:
>> 
>> Very cool!
>> 
>> 
>> ________________________________
>> From: Jeff Zhang <zjffdu@gmail.com>
>> Sent: Friday, October 19, 2018 7:14 AM
>> To: dev@zeppelin.apache.org
>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter
>> 
>> Thanks xun. This would be a great addon for zeppelin to support deep
>> learning. I will check the design later.
>> 
>> 
>> 
>> liu xun <neliuxun@163.com>于2018年10月19日周五 下午3:56写道:
>> 
>>> Hi,
>>> Hadoop Submarine is the latest machine learning framework subproject in
>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>>> full-featured system framework for machine learning algorithm development,
>>> distributed model training, model management, and model publishing,
>>> combined with hadoop's intrinsic data storage and data processing
>>> capabilities to enable data scientists to Good mining and the value of the
>>> data.
>>> 
>>> 
>>> I was involved in the development of the hadoop submarine project. So I
>>> plan to add the interpreter module of hadoop submarine to zeppelin.
>>> Let zeppeline increase the development of deep learning. This is my design
>>> document, let's see if there is any opinion, you can put it directly in the
>>> document, thank you!
>>> 
>>> 
>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd
>>> 
>>> 
>>> 
> 


Mime
View raw message