nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joew...@apache.org>
Subject [discuss] Splitting NiFi framework and extension repos and releases
Date Thu, 30 May 2019 15:18:48 GMT
Team,

We've discussed this a bit over the years in various forms but it again
seems time to progress this topic and enough has changed I think to warrant
it.

Tensions:
1) Our build times take too long.  In travis-ci for instance it takes 40
minutes when it works.
2) The number of builds we do has increased.  We do us/jp/fr builds on open
and oracle JDKs.  That is 6 builds.
3) We want to add Java 11 support such that one could build with 8 or 11
and the above still apply.  The becomes 6 builds.
4) With the progress in NiFi registry we can now load artifacts there and
could pull them into NiFi.  And this integration will only get better.
5) The NiFi build is too huge and cannot grow any longer or else we cannot
upload convenience binaries.

We cannot solve all the things just yet but we can make progress.  I
suggest we split apart the NiFi 'framework/application' in its own release
cycle and code repository from the 'nifi extensions' into its own
repository and release cycle.  The NiFi release would still pull in a
specific set of extension bundles so to our end users at this time there is
no change. In the future we could also just stop including the extensions
in nifi the application and they could be sourced at runtime as needed from
the registry (call that a NiFi 2.x thing).

Why does this help?
- Builds would only take as long as just extensions take or just core/app
takes.  This reduces time for each change cycle and reduces load on
travis-ci which runs the same tests over and over and over for each pull
request/push regardless of whether it was an extension or core.

- It moves us toward the direction we're heading anyway whereby extensions
can have their own lifecycle from the framework/app itself.

How is this not awesome:
- Doesn't yet solve for the large builds problem.  I think we'll get there
with a NiFi 2.x release which fully leverages nifi-registry for retrieval
of all extensions.
- Adds another 'thing we need to do a release cycle for'.  This is
generally unpleasant but it is paid for once a release cycle and it does
allow us to release independently for new cool extensions/fixes apart from
the framework itself.

Would be great to hear others thoughts if they too feel it is time to make
this happen.

Thanks
Joe

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message