hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16092) Move the source of hadoop/ozone containers to the same repository
Date Tue, 28 May 2019 17:01:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849929#comment-16849929

Eric Yang commented on HADOOP-16092:

{quote}Please don't mix the two things. Usability and popularity are two different things.{quote}

Ease of use and popularity are often related.  Hadoop already have hadoop-build image for
developer to work locally.  Hadoop-runner image is a distraction that serves no purpose to
Hadoop development.  The discussion is digressing from the goal to match docker image build
source with Hadoop branches.  Hadoop development already have ability to build version specific
hadoop-build docker images from trunk, and branch-3.2 code base.  The same ability exists
in YARN as well.  Ozone is the only one that uses hadoop-runner image that is not in version
controlled by the same source code repository.  This ticket should focus on making Ozone hadoop-runner
image more aligned to Hadoop trunk source code.  

Fictional use case of having ability to run Hadoop 2.7.3 with latest hadoop-runner image can
not be supported in reality.  The latest hadoop-runner source code does not have versioning.
 It would be very difficult to maintain forever backward compatibility between latest hadoop-runner
source commits and past version of Hadoop.  This has been observed that trunk version of hadoop-runner:latest
is already not compatible with Ozone 0.4.0.  Trunk version of hadoop-runner image is trying
to avoid hard code of uid, which deviates from original hadoop-runner:latest when Ozone 0.4.0
was released.  A new user may come in and said that he likes the hard coded uid, and current
hadoop-runner:latest breaks his backward compatibility.  Hence, maintaining this forever backward
compatible code cross Hadoop major version is a non-obtainable goal and cost expensive community
drain.  Unlike latest release of a software, hadoop-runner:latest is a publicly exposed snapshot
that claim to have forever compatibility.  The statement is too bold to uphold.

By bringing hadoop-runner docker build process into regular Hadoop maintenance branches, there
is some version control to roughly hint developer which version of hadoop-runner image that
they require to rebuild hadoop-runner:latest for Ozone 0.4.0 release.  It releases Hadoop
community from the burden to carry forever backward compatible snapshot.

> Move the source of hadoop/ozone containers to the same repository
> -----------------------------------------------------------------
>                 Key: HADOOP-16092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16092
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Elek, Marton
>            Assignee: Gabor Bota
>            Priority: Major
> This is proposed by [~eyang] in [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
mailing thread.
> bq. Hadoop community can decide what is best for Hadoop.  My preference is to remove
ozone from source tree naming, if Ozone is intended to be subproject of Hadoop for long period
of time.  This enables Hadoop community to host docker images for various subproject without
having to check out several source tree to trigger a grand build
> As of now the source of  hadoop docker images are stored in the hadoop git repository
(docker-* branches) for hadoop and in hadoop-docker-ozone git repository for ozone (all branches).
> As it's discussed in HDDS-851 the biggest challenge to solve here is the mapping between
git branches and dockerhub tags. It's not possible to use the captured part of a github branch.
> For example it's not possible to define a rule to build all the ozone-(.*) branches and
use a tag $1 for it. Without this support we need to create a new mapping for all the releases
manually (with the help of the INFRA).
> Note: HADOOP-16091 can solve this problem as it doesn't require branch mapping any more.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message