nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Giovanni Lanzani" <giovannilanz...@godatadriven.com>
Subject Re: StoreInKiteDataset, wasb, and class path
Date Fri, 29 Sep 2017 16:50:36 GMT
On 29 Sep 2017, at 17:00, Bryan Bende wrote:

> Hello,
>
> Unfortunately it isn't really that easy...
>
> The way NiFi is setup you aren't really supposed to be modifying the
> classpath, other than adding new NARs to the lib directory.
>
> Each NAR has it's own classloader and isolates the dependencies so
> that other NARs don't impact each other, and all NARs see any JARs
> directly in the lib directory. Adding all the WASB JARs to the lib
> directory would make them visible to the Kite processors, but would
> also likely cause unintended side effects in other processors.
>
> We built a specific framework level capability to handle what you are
> wanting to do...
>
> You would add a new PropertyDescriptior to the processor like:
>
> public static final PropertyDescriptor ADDITIONAL_CLASSPATH_RESOURCES
> = new PropertyDescriptor.Builder()
>         .name("Additional Classpath Resources")
>         .description("A comma-separated list of paths to files and/or
> directories that will be added to the classpath. When specifying a " +
>                 "directory, all files with in the directory will be
> added to the classpath, but further sub-directories will not be
> included.")
>         .required(false)
>         .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
>         .dynamicallyModifiesClasspath(true)
>         .build();
>
> The framework will see the "dynamicallyModifiesClasspath" part and
> will automatically add these resources to the class loader for the
> given instance of the processor.
>
> The above example property is from the HDFS processors and is how we
> handled the WASB situation for Put/Get/List HDFS.

Now you definitely picked my interest.

I've opened https://issues.apache.org/jira/browse/NIFI-4443 and I'll try 
to work on it a bit. I might not tackle the Kerberos stuff just yet but 
I'm curious to see if this additional CLASSPATH thingy is enough.

Thanks,

Giovanni

Mime
View raw message