hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingliang Liu <lium...@gmail.com>
Subject Re: [DISCUSS] official docker image(s) for hadoop
Date Thu, 14 Sep 2017 04:19:41 GMT
> It would be very helpful for testing the RC.
For testing and voting, I have been using docker containers for a while, see code at: https://github.com/weiqingy/caochong

> TL;DR: I propose to create official hadoop images and upload them to the dockerhub
I’m +1 on this idea. The “official” docker image basically means a commitment to maintain
well documented and broadly tested images, which seems not a burden to us.

Ceph has a community docker project https://github.com/ceph/ceph-docker <https://github.com/ceph/ceph-docker>
and I think our scope here is similar to it.


> On Sep 13, 2017, at 11:39 AM, Yufei Gu <flyrain000@gmail.com> wrote:
> It would be very helpful for testing the RC. To vote a RC, committers and
> PMCs usually spend lots of time to compile, deploy the RC, do several
> sanity tests, then +1 for the RC. The docker image potentially saves the
> compilation and deployment time, and people can do more tests.
> Best,
> Yufei
> On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan <wheeleast@gmail.com> wrote:
>> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
>> super helpful!
>> Thanks,
>> Wangda
>> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
>> miklos.szegedi@cloudera.com> wrote:
>>> Marton, thank you for working on this. I think Official Docker images for
>>> Hadoop would be very useful for a lot of reasons. I think that it is
>> better
>>> to have a coordinated effort with production ready base images with
>>> dependent images for prototyping. Does anyone else have an opinion about
>>> this?
>>> Thank you,
>>> Miklos
>>> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek <hdp@anzix.net> wrote:
>>>> TL;DR: I propose to create official hadoop images and upload them to
>> the
>>>> dockerhub.
>>>> GOAL/SCOPE: I would like improve the existing documentation with
>>>> easy-to-use docker based recipes to start hadoop clusters with various
>>>> configuration.
>>>> The images also could be used to test experimental features. For
>> example
>>>> ozone could be tested easily with these compose file and configuration:
>>>> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
>>>> Or even the configuration could be included in the compose file:
>>>> https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
>>>> ker-compose.yaml
>>>> I would like to create separated example compose files for federation,
>>> ha,
>>>> metrics usage, etc. to make it easier to try out and understand the
>>>> features.
>>>> CONTEXT: There is an existing Jira https://issues.apache.org/jira
>>>> /browse/HADOOP-13397
>>>> But it’s about a tool to generate production quality docker images
>>>> (multiple types, in a flexible way). If no objections, I will create a
>>>> separated issue to create simplified docker images for rapid
>> prototyping
>>>> and investigating new features. And register the branch to the
>> dockerhub
>>> to
>>>> create the images automatically.
>>>> MY BACKGROUND: I am working with docker based hadoop/spark clusters
>> quite
>>>> a while and run them succesfully in different environments (kubernetes,
>>>> docker-swarm, nomad-based scheduling, etc.) My work is available from
>>> here:
>>>> https://github.com/flokkr but they could handle more complex use cases
>>>> (eg. instrumenting java processes with btrace, or read/reload
>>> configuration
>>>> from consul).
>>>> And IMHO in the official hadoop documentation it’s better to suggest
>> to
>>>> use official apache docker images and not external ones (which could be
>>>> changed).
>>>> Please let me know if you have any comments.
>>>> Marton
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message