nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: problems with custom controller services, and other comments
Date Tue, 10 Jan 2017 18:29:29 GMT
Michael

That is indeed strange. 
Quick question, is there any chance we can look at your code (the custom CS). Also, what version
of NiFi you are using?
Will be looking into it, but any additional info is helpful 

Cheers
Oleg


> On Jan 10, 2017, at 1:14 PM, Knapp, Michael <Michael.Knapp@capitalone.com> wrote:
> 
> Devs,
> 
> For some reason NIFI is not working with some custom controller services I have written.
 I wrote new implementations of AWSCredentialsProviderService, that aim to work with session
tokens.  I am hoping to run NIFI from my local machine and to be able to connect with AWS
using session tokens.
> 
> For both implementations I tried, it fails when I try to create them from the web UI.
 I created a PutS3Object processor, and configured the “AWS Credentials Provider Service”
property.  From that property I tried to create a new service and selected my custom implementation.
 When I click “create” the value for the credentials provider service is the ID of the
controller service, not its name.  While my controller services require several properties
to be set, the web UI is not letting me set them.  Usually I see an arrow next to the property,
which allows me to configure a controller service, but I am not getting that now.  I looked
in the nifi-app logs, and I do not see any exception, I have even set the logging to TRACE
for all things, and still don’t see any problem in the logs.  The PutS3Object processor
is not validating because the controller service is found to be invalid.  I tried creating
a unit test, it seems to work for me in tests, but I can’t use TestRunners because that
is processor oriented, not meant for controller services.  I have a suspicion that spring’s
aspect oriented programming is somehow fuddling with my service.
> 
> Does anybody know what I am doing wrong here?
> 
> Other unrelated comments:
> 
> 1.       The first time you unpack NIFI it takes super long for it to start for me, like
a half hour or more.  I think you should make it easy for people to scale back their NIFI
implementation.  Really I would like to start it with just the minimum NAR files for it to
start, and I can add others that I need.  Maybe a sub-directory in lib for the essential nars
could help people separate the essential stuff from the optional nars.  The first time I tried
installing it, I thought it was broken when really it just was taking forever (over 30 minutes).
 I think that new users will probably abandon NIFI if they can’t get it to start quickly
out of the box.  Maybe split the optional nars into an “extra-lib”, and people can move
those into lib as necessary for their goals.
> 
> 2.       Building NIFI from source takes over an hour for me, really I just want to build
the bare minimum things to get it to start.  I tried creating maven profiles to have it build
just the minimum pieces, but this proved to be non-trivial as maven does not seem to respect
the “modules” tag in profiles, and the nifi-assembly project requires all of the optional
nars to also be built.  Creating this might be too complicated for me.  Has anybody thought
about supporting a quick/minimal build?
> 
> 3.       The “nifi-aws-processors” is challenging to use because in one project they
have defined the interfaces for controller services (AWSCredentialsProviderService) and also
included the services.  I tried creating my own nar with an implementation of AWSCredentialsProviderService,
but since it depended on “nifi-aws-processors”, my nar was also re-hosting all of the
AWS processors.  I was facing a lot of classpath issues because of this.  I worked around
this by using maven shade to shade in the “nifi-aws-processors” into my own jar, but excluding
the services it provided.  Then in my nar project I had to exclude the dependency on “nifi-aws-processors”.
 This was a lot of work on my part when all they needed to do was split that project into
api, api-nar, impl, and impl-nar.
> 
> 4.       I think it is very confusing how there is a “Controller Services” for the
entire NIFI canvas, and separate ones for individual processor groups.  It seems that processors
cannot use global controller services, and I am still uncertain about why I would ever create
a global one.  From Nifi settings, I would like to also see controller services in processor
groups, and vice versa.  From a processor, I would like to assign controller services that
are global in scope, not limited to a processor group.  I think this is something that will
confuse and frustrate a lot of new developers, driving them to consider competing products.
> 
> 5.       I think the developer guide needs some clarification on what jars are provided
and not.  New developers will be unsure if they should include “nifi-api” as a provided
or compile dependency, and same goes for nifi-framework-core.
> 
> 6.       Perhaps the maven-nar-plugin could let people tell NIFI to only use services
listed under a certain set of bundled-dependencies.  For example, my code depends on “nifi-aws-processors”,
but I don’t want my nar to also host the services in that jar.  From the MOJO you are able
to exclude entire artifacts, but you can’t exclude the services within certain artifacts.
 This might be a problem to fix on the classloader side instead of from the maven plugin,
I don’t know which is easier.
> 
> Please help,
> 
> Michael Knapp
> Capital One
> ________________________________________________________
> 
> The information contained in this e-mail is confidential and/or proprietary to Capital
One and/or its affiliates and may only be used solely in performance of work or services for
Capital One. The information transmitted herewith is intended only for use by the individual
or entity to which it is addressed. If the reader of this message is not the intended recipient,
you are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.

Mime
View raw message