hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miklos Szegedi <miklos.szeg...@cloudera.com>
Subject Re: [DISCUSS] official docker image(s) for hadoop
Date Wed, 13 Sep 2017 17:48:29 GMT
Marton, thank you for working on this. I think Official Docker images for
Hadoop would be very useful for a lot of reasons. I think that it is better
to have a coordinated effort with production ready base images with
dependent images for prototyping. Does anyone else have an opinion about
this?

Thank you,
Miklos

On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek <hdp@anzix.net> wrote:

>
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
>
> GOAL/SCOPE: I would like improve the existing documentation with
> easy-to-use docker based recipes to start hadoop clusters with various
> configuration.
>
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
>
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
>
> Or even the configuration could be included in the compose file:
>
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> ker-compose.yaml
>
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the
> features.
>
> CONTEXT: There is an existing Jira https://issues.apache.org/jira
> /browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images
> (multiple types, in a flexible way). If no objections, I will create a
> separated issue to create simplified docker images for rapid prototyping
> and investigating new features. And register the branch to the dockerhub to
> create the images automatically.
>
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite
> a while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases
> (eg. instrumenting java processes with btrace, or read/reload configuration
> from consul).
>  And IMHO in the official hadoop documentation it’s better to suggest to
> use official apache docker images and not external ones (which could be
> changed).
>
> Please let me know if you have any comments.
>
> Marton
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message