nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Ghigiarelli <briang...@gmail.com>
Subject Re: Custom Controller Service Debugging
Date Fri, 14 Aug 2015 15:48:04 GMT
Cool - thanks for verifying that, Matt!
https://issues.apache.org/jira/browse/NIFI-854

On Fri, Aug 14, 2015 at 11:43 AM, Matt Gilman <matt.c.gilman@gmail.com>
wrote:

> Brian,
>
> I see the same behavior regarding Controller Service creation when
> referenced by Processor's in sub Groups. Please file the JIRA ticket and it
> will be addressed in our release. Thanks!
>
> Matt
>
> On Fri, Aug 14, 2015 at 11:30 AM, Brian Ghigiarelli <brianghig@gmail.com>
> wrote:
>
> > Matt and Bryan,
> >
> > Thanks for the detailed explanation.  The class loading aspect makes
> > perfect sense, and yes, importing as disabled is understood as well.
> >
> > I do think I've come across a small bug with automatically creating
> > Controller Services on template imports though.  NiFi is not
> automatically
> > creating Controller Services for Processors inside of Process Groups on
> > template imports. Can one of you check me on this before we write up a
> Jira
> > ticket?
> >
> > Steps to reproduce:
> > 1. Start fresh (clean canvas, no controller services)
> > 2. Create Process Group and go into it
> > 3. Add DetectDuplicate processor to Process Group
> > 4. Configure DetectDuplicate processor with a new
> > DistributedMapCacheClientService and enable that service so that the
> > DetectDuplicate processor is valid and ready to run
> > 5. Navigate to root canvas (see the Process Group)
> > 6. Create template from root canvas (name: "Controller Service Test")
> > 7. Delete everything from the canvas and disable + delete the controller
> > service so that we're essentially back to the start
> > 8. Drag and drop the "Controller Service Test" template onto the canvas
> > (expecting that everything will be auto-created)
> > 9. Navigate into the Process Group and notice that the DetectDuplicate
> > processor is invalid because the DistributedMapCacheClientService ID
> cannot
> > be found
> >
> > As an additional test, I created a template with a processor at the
> > top-level and one inside the Process Group that are using the same
> > Controller Service, and that works just fine.  My guess is that the
> > top-level processor is used to create the Controller Service and the
> > inner-processor picks it up by that same ID.
> >
> > I can provide a sample template in a Jira ticket if that would help.
> >
> > Thanks,
> > Brian
> >
> > On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende <bbende@gmail.com> wrote:
> >
> > > Brian,
> > >
> > > Glad it worked!
> > >
> > > I think it was just a combination of things... I had previously spent a
> > lot
> > > of time studying the existing dependency setup in nifi-standard-bundle,
> > as
> > > well as the dev
> > > guide section on NARs [1]. Also, someone else had ran into a similar
> > > problem recently and it turned out to be a dependency problem as well.
> > >
> > > There should be a Maven archetype for Controller Services [2] coming
> soon
> > > which might help nail down the correct dependency setup, also after
> > seeing
> > > your email last night I put together a bare-bones project [3] that has
> a
> > > custom processor using a custom controller service (both created using
> > > the archetypes) to serve as an example.
> > >
> > > -Bryan
> > >
> > > [1]
> > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
> > > [2] https://issues.apache.org/jira/browse/NIFI-736
> > > [3] https://github.com/bbende/nifi-dependency-example
> > >
> > >
> > >
> > > On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman <matt.c.gilman@gmail.com>
> > > wrote:
> > >
> > > > Brian,
> > > >
> > > > The NAR dependency stuff all breaks down to providing classpath
> > isolation
> > > > and not polluting the classpath of downstream NARs with dependencies
> > > > required for Controller Service implementation. We try to keep
> > Controller
> > > > Service interfaces lean by not depending on external libs in the
> > > interface
> > > > (generally a good practice anyways). Both the implementation of the
> > > > Controller Service and the any Processor NAR that want to consume the
> > > > Controller Service needs to depend on the Controller Service API NAR
> > > (where
> > > > the interface lives). This established a common parent ClassLoader
> > which
> > > > allows the service to be access across NARs. If your building
> Processor
> > > > NARs that depend on custom Controller Services and standard
> Controller
> > > > Services, you'll want to have your custom Controller Services API NAR
> > > > depend on the standard Controller Services API NAR. The NAR
> dependency
> > > tree
> > > > (if you will) maps to a ClassLoader hierarchy within NiFi.
> > > >
> > > > Controller Services being added in a disabled state is remaining
> > > consistent
> > > > with Processors (and other components) in template. Also, because
> > > sensitive
> > > > properties are not included in templates, they may require the user
> to
> > > > complete the configuration prior to enabling.
> > > >
> > > > Matt
> > > >
> > > > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli <
> > brianghig@gmail.com>
> > > > wrote:
> > > >
> > > > > Bryan,
> > > > >
> > > > > Thank you!! Adding a NAR dependency to my standard-processors NAR
> did
> > > the
> > > > > trick to get NiFi to recognize the service for these processors and
> > > > > auto-create a new instance of the Controller Service on template
> > > > import.  I
> > > > > owe ya a beer or six the next time our paths cross.
> > > > >
> > > > > How did you figure out the required NAR dependency? Any great
> > debugging
> > > > > tricks to share, or was it your existing depth of NiFi knowledge
> that
> > > led
> > > > > you to that solution?
> > > > >
> > > > > As a follow-on to that, I'm seeing that Controller Services created
> > via
> > > > > Template instantiation are created in "Disabled" state.  Is that
> > > > expected?
> > > > > Is there any way to drag-and-drop a template with controller
> services
> > > > onto
> > > > > a canvas and have the newly created controller services enabled?
> > > > >
> > > > > Thanks again,
> > > > > Brian
> > > > >
> > > > > On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bbende@gmail.com>
> > wrote:
> > > > >
> > > > > > Brian,
> > > > > >
> > > > > > I just ran into this exact scenario today while testing a custom
> > > > > controller
> > > > > > service.
> > > > > >
> > > > > > The cause of the problem was that my custom service NAR, did
not
> > > have a
> > > > > NAR
> > > > > > dependency on the service API NAR because I had accidentally
> marked
> > > the
> > > > > API
> > > > > > NAR as provided in the pom file. Once I removed the provided
> scope
> > > > > > everything worked.
> > > > > >
> > > > > > You can check the MANIFEST file in your custom service NAR...
> under
> > > > your
> > > > > > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF
it
> > > should
> > > > > > have a "Nar-Dependency-Id" on your API NAR. If it is missing
> that,
> > > that
> > > > > is
> > > > > > likely the problem.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <
> > > > brianghig@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > NiFi Gurus,
> > > > > > >
> > > > > > > Regarding Controller Services, I'm seeing some strange
behavior
> > > > > > regarding a
> > > > > > > custom Controller Service only showing its ID in a Processor
> and
> > > was
> > > > > > > wondering if anyone's encountered similar issues.  Seeing
this
> on
> > > the
> > > > > > > latest develop branch, 0.3.0-SNAPSHOT.
> > > > > > >
> > > > > > > Steps taken:
> > > > > > > 1. Create instance of Controller Service in the Controller
> > > Settings >
> > > > > > > Controller Services tab
> > > > > > > 1a. Enable that instance of the Controller Service
> > > > > > > 2. Add new Processor on canvas that will use Controller
Service
> > > > > > > 3. Configure Processor
> > > > > > > 3a. Existing Controller Service does not show in property
> > options.
> > > > Only
> > > > > > "No
> > > > > > > value" and "Create new service..."
> > > > > > > 3b. Clicking "Create new service..." opens modal with the
> custom
> > > > > > Controller
> > > > > > > Service as the only item in the dropdown
> > > > > > > 3c. Clicking "Create" button succeeds, but puts the UUID
of the
> > > > > > Controller
> > > > > > > Service in the Value of the Processor Property.
> > > > > > >
> > > > > > > Observed state:
> > > > > > > - Processor is still invalid
> > > > > > > - The ID in the Processor Property matches the ID of the
second
> > > > > > (disabled)
> > > > > > > instance of the Controller Service in the Controller Settings
> > > modal.
> > > > > > >
> > > > > > > There are no errors in nifi-app.log - only contains log
> messages
> > > from
> > > > > the
> > > > > > > StandardControllerServiceProvider that it "Created Controller
> > > Service
> > > > > of
> > > > > > > type [my-implementation-class] with identifier
> > > > > > > [UUID-that-shows-in-the-GUI]"
> > > > > > >
> > > > > > > Adding a DetectDuplicate processor with the
> > > > > > > DistributedMapCacheClientService works just fine, so it'd
hard
> to
> > > > point
> > > > > > > fingers at the framework.  Any thoughts on where to begin
> > debugging
> > > > > > and/or
> > > > > > > loggers to turn up would be greatly appreciated!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Brian
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Brian Ghigiarelli
> > > > > 570-878-9139
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Brian Ghigiarelli
> > 570-878-9139
> >
>



-- 
Brian Ghigiarelli
570-878-9139

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message