nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: About Nar Classloader
Date Thu, 09 May 2019 12:05:08 GMT
https://www.youtube.com/watch?v=-ISycoP3TVA
Is a video demonstrating using the archetypes to create a bundle, loading
it via UI into Metron and starting a storm topology that loads it from HDFS.


On May 9, 2019 at 07:47:59, Otto Fowler (ottobackwards@gmail.com) wrote:

I think I am that person, I call them Bundles not NARs. ;)

https://github.com/palindromicity/bundles
https://github.com/palindromicity/test-bundles
https://github.com/palindromicity/bundles-testing
https://github.com/palindromicity/bundles-maven


A while ago, I forked and refactored the NAR bundle system  for possible
use in Apache Metron.  Although we did not end up using it, I created
github projects for the work.
While working on this for metron, I used it and archetypes I created for
metron to use bundles in storm ( loading bundles from HDFS ), and in spring
boot apps. The system is not up
to date with the very latest changes in bundles, but for your own app, you
may not need those changes. My maven plugin and configuration allow you to
change the names from .nar / NAR to whatever you want.

You are welcome to take a look, as is anyone.  The issues link is there as
well.
If you are going to use it, there is a slack that you can join as well (
though it is for my github org, which also has the simple-syslog* libs that
Nifi and Metron use ).

# Bundles

Bundles are a derivative of the [Apache Nifi](https://nifi.apache.org)
[NARs](https://nifi.apache.org/developer-guide.html).
This is an attempt to adapt the NAR system to be usable outside of [Apache
Nifi](https://nifi.apache.org) in other scenarios.

## bundles-maven-plugin
The [bundles-maven-plugin](https://github.com/palindromicity/bundles-maven)
is an adapted version of the jar dependency plugin whose function is to
bundle a jar of jars based on the dependencies for a project.  It also
creates metadata attributes.
A project's jar, and it's non-provided dependency jars are place in a /lib
entry in the bundle, with the bundle itself being in jar format.

## bundles-lib
The bundles-lib contains the functionality required to:
- discover bundles
- inspect bundles for exposed extension types
- load the bundles
- create special class loaders for bundles
- deliver instances of extension types for use

NAR exposed the bundles through many classes.  I have created the
BundleSystem interface to expose a more usable, simplified api for our use
cases.

### What is different
- Adaptation of nifi-nar-utils to be used outside of the [Apache Nifi](
https://nifi.apache.org) project
- Rudimentary extensibility to allow configuration and injection of service
types and other things that were hard coded to nifi
- Refactored from File based to [Apache Commons VFS](
https://commons.apache.org/proper/commons-vfs/) based, allowing for bundle
libraries to be loaded from HDFS
- Rebranding to Bundle from Nar ( although the lib and the plugin allow
that to be configured now )
- Added capability to the properties class to write to stream, adapted to
uri from paths
- Added integration tests for hdfs
- Changed to be ClassIndex based instead of ServiceLoader. Service loader
is slower, and Casey's ClassIndex work is great. This also removes the
NAR's required manual maintenance of the service file.
- Refactored to use VFS to load the bundle/nar into the classloader AND to
use VFS to load the dependency jars -> VFS as a composite filesystem. Thus
going from NAR's 'working directory', exploded NARS to just loading the
bundle/nar.
- Created a simple interface for interacting with the system
- Refactored such that it is possible to add new bundles to the running
system without having to restart the application


## Project info

- bundles-lib The bundles library
- bundles-testing a sample project showing how to use bundles
- test-bundles source for bundles used for bundles-lib tests



On May 8, 2019 at 22:53:04, Joe Witt (joe.witt@gmail.com) wrote:

Jianan

We have not done the work to fully isolate the concept of a NiFi Archive
(Nar) such that it could be used outside of NiFi as a general classloader
isolation pattern.

There was one other person interested in helping make this happen in the
past but I'm not sure where it has gone.

With some of the recent advances in Java it might well be the case that a
more general purpose option exists (modules).

The concept of a NAR is at this time pretty tightly coupled to the NiFi
project by design.

Thanks
Joe

On Wed, May 8, 2019 at 10:49 PM Jianan Zhang <william.jn.zhang@gmail.com>
wrote:

> Hi all,
>
> I really like the idea of Nar-Bundle, it create namespaces that separate
> several dependency environments.
> I want to use Nar-Bundle in other java project, and I found the
> <nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
> Does it exists standalone nar-utils or other similar utils?
>
> Thanks.
>

Mime
View raw message