nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: problems with custom controller services, and other comments
Date Thu, 12 Jan 2017 19:46:50 GMT
Michael

Sorry took a while, but we just had one of our guys reporting the same symptoms and as Bryan
suggested it was actually a problem in the POM, specifically the POM file for NAR modules
of your custom processor.
Basically once your processor is referencing the service it’s NRA module has to declare
dependency on such service.
I m pasting an example POM for a NAR module from one of our guys who is referencing DBCP service
from his custom bundle

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-sql-bundle</artifactId>
<version>1.1.0</version>
</parent>

<artifactId>nifi-sql-nar</artifactId>
<packaging>nar</packaging>

<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-nar</artifactId>
<type>nar</type>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-sql-processors</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>
</project>

Note the first dependency (wifi-dbcp-service-nar)

Hope that helps

Cheers
Oleg
On Jan 10, 2017, at 1:29 PM, Oleg Zhurakousky <ozhurakousky@hortonworks.com<mailto:ozhurakousky@hortonworks.com>>
wrote:

Michael

That is indeed strange.
Quick question, is there any chance we can look at your code (the custom CS). Also, what version
of NiFi you are using?
Will be looking into it, but any additional info is helpful

Cheers
Oleg


On Jan 10, 2017, at 1:14 PM, Knapp, Michael <Michael.Knapp@capitalone.com<mailto:Michael.Knapp@capitalone.com>>
wrote:

Devs,

For some reason NIFI is not working with some custom controller services I have written. 
I wrote new implementations of AWSCredentialsProviderService, that aim to work with session
tokens.  I am hoping to run NIFI from my local machine and to be able to connect with AWS
using session tokens.

For both implementations I tried, it fails when I try to create them from the web UI.  I created
a PutS3Object processor, and configured the “AWS Credentials Provider Service” property.
 From that property I tried to create a new service and selected my custom implementation.
 When I click “create” the value for the credentials provider service is the ID of the
controller service, not its name.  While my controller services require several properties
to be set, the web UI is not letting me set them.  Usually I see an arrow next to the property,
which allows me to configure a controller service, but I am not getting that now.  I looked
in the nifi-app logs, and I do not see any exception, I have even set the logging to TRACE
for all things, and still don’t see any problem in the logs.  The PutS3Object processor
is not validating because the controller service is found to be invalid.  I tried creating
a unit test, it seems to work for me in tests, but I can’t use TestRunners because that
is processor oriented, not meant for controller services.  I have a suspicion that spring’s
aspect oriented programming is somehow fuddling with my service.

Does anybody know what I am doing wrong here?

Other unrelated comments:

1.       The first time you unpack NIFI it takes super long for it to start for me, like a
half hour or more.  I think you should make it easy for people to scale back their NIFI implementation.
 Really I would like to start it with just the minimum NAR files for it to start, and I can
add others that I need.  Maybe a sub-directory in lib for the essential nars could help people
separate the essential stuff from the optional nars.  The first time I tried installing it,
I thought it was broken when really it just was taking forever (over 30 minutes).  I think
that new users will probably abandon NIFI if they can’t get it to start quickly out of the
box.  Maybe split the optional nars into an “extra-lib”, and people can move those into
lib as necessary for their goals.

2.       Building NIFI from source takes over an hour for me, really I just want to build
the bare minimum things to get it to start.  I tried creating maven profiles to have it build
just the minimum pieces, but this proved to be non-trivial as maven does not seem to respect
the “modules” tag in profiles, and the nifi-assembly project requires all of the optional
nars to also be built.  Creating this might be too complicated for me.  Has anybody thought
about supporting a quick/minimal build?

3.       The “nifi-aws-processors” is challenging to use because in one project they have
defined the interfaces for controller services (AWSCredentialsProviderService) and also included
the services.  I tried creating my own nar with an implementation of AWSCredentialsProviderService,
but since it depended on “nifi-aws-processors”, my nar was also re-hosting all of the
AWS processors.  I was facing a lot of classpath issues because of this.  I worked around
this by using maven shade to shade in the “nifi-aws-processors” into my own jar, but excluding
the services it provided.  Then in my nar project I had to exclude the dependency on “nifi-aws-processors”.
 This was a lot of work on my part when all they needed to do was split that project into
api, api-nar, impl, and impl-nar.

4.       I think it is very confusing how there is a “Controller Services” for the entire
NIFI canvas, and separate ones for individual processor groups.  It seems that processors
cannot use global controller services, and I am still uncertain about why I would ever create
a global one.  From Nifi settings, I would like to also see controller services in processor
groups, and vice versa.  From a processor, I would like to assign controller services that
are global in scope, not limited to a processor group.  I think this is something that will
confuse and frustrate a lot of new developers, driving them to consider competing products.

5.       I think the developer guide needs some clarification on what jars are provided and
not.  New developers will be unsure if they should include “nifi-api” as a provided or
compile dependency, and same goes for nifi-framework-core.

6.       Perhaps the maven-nar-plugin could let people tell NIFI to only use services listed
under a certain set of bundled-dependencies.  For example, my code depends on “nifi-aws-processors”,
but I don’t want my nar to also host the services in that jar.  From the MOJO you are able
to exclude entire artifacts, but you can’t exclude the services within certain artifacts.
 This might be a problem to fix on the classloader side instead of from the maven plugin,
I don’t know which is easier.

Please help,

Michael Knapp
Capital One
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message