nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Doran <>
Subject Re: Dockerfile and Docker Hub Management
Date Thu, 21 Sep 2017 16:42:22 GMT
Thanks, Aldrin.

I'm a +1 for a separate NiFi Docker project and release process, allowing improvements to
the containerization of NiFi artifacts to be made independently from the artifacts (and their
source files). I have observed many community members maintaining their own NiFi Dockerfiles
and Docker Compose files, and it would be nice if the most reusable variant(s) of these could
be well maintained in one location on an independent schedule.

I also agree a separate repository would be cleaner and preferable that a subtree of the existing
NiFi repository.


On 9/21/17, 11:12, "Aldrin Piri" <> wrote:

    Hey folks,
    ** This message turned out to be more detailed than anticipated.  In
    summary, I propose consolidating Docker/container work with a separate
    release process outside of the repository they are packaging.  Full
    thoughts and background follow.  Any input would be appreciated!
    I've been working through providing some additional Docker capabilities for
    the project and wanted to share some thoughts as well as possible plans to
    help us be a bit more nimble and responsive to curating Dockerfiles and
    their respective images on DockerHub.
    As a bit of context, we currently have the core NiFi project captured in
    two Dockerfiles, one that is used in conjunction with a Maven plugin for
    creating an image during the NiFi build (dockermaven), and another that is
    used for building tagged releases on Docker Hub (dockerhub).  Both of these
    artifacts, currently, reside in a nifi-docker project and are activated via
    Maven profile, (-P docker).
    We've seen at times that this is a very coupled process and limits our
    flexibility.  For instance, we had an ill-placed 'chown' which caused a
    duplicating layer and causes our image to be doubly large.  While this has
    been remedied, given current release processes, this is included with the
    core nifi release and we have been unable to rectify that issue.
    Another issue is a very specific sequence of actions that needs to happen
    with the current release for artifacts to be triggered correctly in Docker
    Hub.  This can be seen in Section 6 of the release guide [1].  While there
    are ways to rectify this if the timing isn't quite right and/or an error is
    made, it can impose an additional burden on the INFRA team to facilitate
    these requests as there currently is no capability for PMCs to manage their
    Docker repositories directly.
    Ultimately, I think we should consider a separate release process for NiFi
    Docker, and any associated efforts that may relate to those files.  In this
    context, NiFi is encompassing of all projects/efforts in the project.
    Additional efforts could comprise of examples of configuring NiFi to be
    secured or clustered, receive data from MiNiFi instances, or using Docker
    Compose or other orchestration frameworks. I have also noticed a number of
    different areas across our work that are using Docker for integration
    testing purposes.  With some planning and coordination, we could likely
    consolidate many of these core resources/templates to allow us to reuse
    them across efforts.
    I believe there are two approaches from an organizational standpoint that
    let us execute on the separate release process effectively:
    1.) Condense all Docker artifacts into the current NiFi repository [2].  We
    update our release for NiFi to exclude the Docker subtree to carry out our
    normal release flow and provide the build/tooling for the Docker subtree to
    be released on its own.
    2.)  Establish a new git repository to handle Docker and any other
    containerization efforts and migrate all existing resources into a file
    structure that makes sense.
    My inclination is toward (2).
    Regardless of path chosen above, this frees us to handle updates and
    improvements to container efforts when needed.  Any time we wanted to
    release updates to Docker images, we could perform a separate release on
    either the subtree of (1) or the repository of (2) and reference the
    associated latest artifacts of NiFi.
    If you've made it this far, thanks for working through the wall of text and
    would appreciate any thoughts or comments.

View raw message