hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject Re: Any thoughts making Submarine a separate Apache project?
Date Tue, 13 Aug 2019 08:36:52 GMT
Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵 <dujunping@gmail.com> wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli <vinodkv@apache.org> 于2019年7月29日周一 下午9:46写道:
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan <wheeleast@gmail.com> wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo <neo@pingcap.com> wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu <liuxun@apache.org> 于2019年7月22日周一 下午4:07写道:
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣 <xinzhao@seu.edu.cn> 于2019年7月22日周一 上午9:38写道:
>> >>>
>> >>>> Hi
>> >>>>
>> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/).
We
>> >> are
>> >>>> a major in electrical engineering. Our teaching teams and students
>> use
>> >>>> bigoop submarine for big data analysis and automation control of
>> >>> electrical
>> >>>> equipment.
>> >>>>
>> >>>> Many thanks to the hadoop community for providing us with machine
>> >>> learning
>> >>>> tools like submarine.
>> >>>>
>> >>>> I wish hadoop submarine is getting better and better.
>> >>>>
>> >>>>
>> >>>> ==============================
>> >>>> 赵欣
>> >>>> 东南大学电气工程学院
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>> Zhao XIN
>> >>>>
>> >>>> School of Electrical Engineering
>> >>>>
>> >>>> ==============================
>> >>>> 2019-07-18
>> >>>>
>> >>>>
>> >>>> *From:* Xun Liu <liuxun@apache.org>
>> >>>> *Date:* 2019-07-18 09:46
>> >>>> *To:* xinzhao <xinzhao@seu.edu.cn>
>> >>>> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
>> >>>> project?
>> >>>>
>> >>>>
>> >>>> ---------- Forwarded message ---------
>> >>>> 发件人: dashuiguailuyun@gmail.com <dashuiguailuyun@gmail.com>
>> >>>> Date: 2019年7月17日周三 下午3:17
>> >>>> Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> >> project?
>> >>>> To: Szilard Nemeth <snemeth@cloudera.com.invalid>, runlin
zhang <
>> >>>> runlin512@gmail.com>
>> >>>> Cc: Xun Liu <liuxun@apache.org>, common-dev <
>> >>> common-dev@hadoop.apache.org>,
>> >>>> yarn-dev <yarn-dev@hadoop.apache.org>, hdfs-dev <
>> >>>> hdfs-dev@hadoop.apache.org>, mapreduce-dev <
>> >>>> mapreduce-dev@hadoop.apache.org>, submarine-dev <
>> >>>> submarine-dev@hadoop.apache.org>
>> >>>>
>> >>>>
>> >>>> +1 ,Good idea, we are very much looking forward to it.
>> >>>>
>> >>>> ------------------------------
>> >>>> dashuiguailuyun@gmail.com
>> >>>>
>> >>>>
>> >>>> *From:* Szilard Nemeth <snemeth@cloudera.com.INVALID>
>> >>>> *Date:* 2019-07-17 14:55
>> >>>> *To:* runlin zhang <runlin512@gmail.com>
>> >>>> *CC:* Xun Liu <liuxun@apache.org>; Hadoop Common
>> >>>> <common-dev@hadoop.apache.org>; yarn-dev <yarn-dev@hadoop.apache.org
>> >;
>> >>>> Hdfs-dev <hdfs-dev@hadoop.apache.org>; mapreduce-dev
>> >>>> <mapreduce-dev@hadoop.apache.org>; submarine-dev
>> >>>> <submarine-dev@hadoop.apache.org>
>> >>>> *Subject:* Re: Any thoughts making Submarine a separate Apache
>> project?
>> >>>> +1, this is a very great idea.
>> >>>> As Hadoop repository has already grown huge and contains many
>> >> projects, I
>> >>>> think in general it's a good idea to separate projects in the early
>> >>> phase.
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 17, 2019, 08:50 runlin zhang <runlin512@gmail.com>
>> wrote:
>> >>>>
>> >>>>> +1 ,That will be great !
>> >>>>>
>> >>>>>> 在 2019年7月10日,下午3:34,Xun Liu <liuxun@apache.org>
写道:
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> This is Xun Liu contributing to the Submarine project for
deep
>> >>> learning
>> >>>>>> workloads running with big data workloads together on Hadoop
>> >>> clusters.
>> >>>>>>
>> >>>>>> There are a bunch of integrations of Submarine to other
projects
>> >> are
>> >>>>>> finished or going on, such as Apache Zeppelin, TonY, Azkaban.
The
>> >>> next
>> >>>>> step
>> >>>>>> of Submarine is going to integrate with more projects like
Apache
>> >>>> Arrow,
>> >>>>>> Redis, MLflow, etc. & be able to handle end-to-end machine
learning
>> >>> use
>> >>>>>> cases like model serving, notebook management, advanced
training
>> >>>>>> optimizations (like auto parameter tuning, memory cache
>> >> optimizations
>> >>>> for
>> >>>>>> large datasets for training, etc.), and make it run on other
>> >>> platforms
>> >>>>> like
>> >>>>>> Kubernetes or natively on Cloud. LinkedIn also wants to
donate TonY
>> >>>>> project
>> >>>>>> to Apache so we can put Submarine and TonY together to the
same
>> >>>> codebase
>> >>>>>> (Page #30.
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
>> >>>>>> ).
>> >>>>>>
>> >>>>>> This expands the scope of the original Submarine project
in
>> >> exciting
>> >>>> new
>> >>>>>> ways. Toward that end, would it make sense to create a separate
>> >>>> Submarine
>> >>>>>> project at Apache? This can make faster adoption of Submarine,
and
>> >>>> allow
>> >>>>>> Submarine to grow to a full-blown machine learning platform.
>> >>>>>>
>> >>>>>> There will be lots of technical details to work out, but
any
>> >> initial
>> >>>>>> thoughts on this?
>> >>>>>>
>> >>>>>> Best Regards,
>> >>>>>> Xun Liu
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> >>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message