nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <>
Subject Dockerfile and Docker Hub Management
Date Thu, 21 Sep 2017 15:11:44 GMT
Hey folks,

** This message turned out to be more detailed than anticipated.  In
summary, I propose consolidating Docker/container work with a separate
release process outside of the repository they are packaging.  Full
thoughts and background follow.  Any input would be appreciated!


I've been working through providing some additional Docker capabilities for
the project and wanted to share some thoughts as well as possible plans to
help us be a bit more nimble and responsive to curating Dockerfiles and
their respective images on DockerHub.

As a bit of context, we currently have the core NiFi project captured in
two Dockerfiles, one that is used in conjunction with a Maven plugin for
creating an image during the NiFi build (dockermaven), and another that is
used for building tagged releases on Docker Hub (dockerhub).  Both of these
artifacts, currently, reside in a nifi-docker project and are activated via
Maven profile, (-P docker).

We've seen at times that this is a very coupled process and limits our
flexibility.  For instance, we had an ill-placed 'chown' which caused a
duplicating layer and causes our image to be doubly large.  While this has
been remedied, given current release processes, this is included with the
core nifi release and we have been unable to rectify that issue.

Another issue is a very specific sequence of actions that needs to happen
with the current release for artifacts to be triggered correctly in Docker
Hub.  This can be seen in Section 6 of the release guide [1].  While there
are ways to rectify this if the timing isn't quite right and/or an error is
made, it can impose an additional burden on the INFRA team to facilitate
these requests as there currently is no capability for PMCs to manage their
Docker repositories directly.

Ultimately, I think we should consider a separate release process for NiFi
Docker, and any associated efforts that may relate to those files.  In this
context, NiFi is encompassing of all projects/efforts in the project.
Additional efforts could comprise of examples of configuring NiFi to be
secured or clustered, receive data from MiNiFi instances, or using Docker
Compose or other orchestration frameworks. I have also noticed a number of
different areas across our work that are using Docker for integration
testing purposes.  With some planning and coordination, we could likely
consolidate many of these core resources/templates to allow us to reuse
them across efforts.

I believe there are two approaches from an organizational standpoint that
let us execute on the separate release process effectively:

1.) Condense all Docker artifacts into the current NiFi repository [2].  We
update our release for NiFi to exclude the Docker subtree to carry out our
normal release flow and provide the build/tooling for the Docker subtree to
be released on its own.

2.)  Establish a new git repository to handle Docker and any other
containerization efforts and migrate all existing resources into a file
structure that makes sense.

My inclination is toward (2).

Regardless of path chosen above, this frees us to handle updates and
improvements to container efforts when needed.  Any time we wanted to
release updates to Docker images, we could perform a separate release on
either the subtree of (1) or the repository of (2) and reference the
associated latest artifacts of NiFi.

If you've made it this far, thanks for working through the wall of text and
would appreciate any thoughts or comments.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message