nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Chaves <e...@uolet.com>
Subject Re: How to get DBCP service inside ScriptedLookupService
Date Wed, 15 Nov 2017 21:55:24 GMT
Hi Folks, after thinking about my scripted components I decided to ditch
the ScriptedLookup in favor of writing a InvokeScriptedProcessor that seems
more aligned with the proper use explained so far.

I've implemented the base script as outlined and added some Properties into
my InvokeScriptedProcessor but one of them is keeping the processor in an
invalid state claiming that the property is invalid because it's not a
supported property. I've declared the property using the following code:

class GroovyProcessor implements Processor {

final static PropertyDescriptor LOOKUP_FIELD = new
PropertyDescriptor.Builder()
    .name("lookup-field")
    .displayName("Lookup field")
    .description("Field used in lookup queries")
    .dynamic(true) // have tried with both dynamic true/false with same
outcome.
    .defaultValue("id")
    .build()

...}

What is my mistake here?

Cheers,

Eric

2017-11-15 12:24 GMT-02:00 Eric Chaves <eric@uolet.com>:

> Matt, Mark, thanks for the great explanations! I'm learning a lot! :)
>
> So I went down the road described but I'm getting another error:
>  groovy.lang.MissingMethodException: No signature of method:
> org.apache.nifi.lookup.script.ScriptedLookupService$1.getProperty() is
> applicable for argument types: (org.apache.nifi.components.PropertyDescriptor)
> values: [PropertyDescriptor[Database Connection Pool Services]]
>
> Basically I declared a final static PropertyDescriptor DBCP_SERVICE and
> inside the initialize method I tried to get the DBCPService as outlined.
> Comparing my code to the QueryDatabaseTable processor I noticed when QDT
> grabs the DBCPService instance it's context is a ProcessContext while in my
> GroovyLookupClass's initialize method context is a
> ControllerServiceInitializationContext so it's seem I'm using the wrong
> object right? Where perform the call for context.getProperty(DBCP_
> SERVICE).asControllerService(DBCPService)?
>
> (https://github.com/apache/nifi/blob/c10ff574c4602fe05f5d1dae5eb0b1
> bd24026c02/nifi-nar-bundles/nifi-standard-bundle/nifi-
> standard-processors/src/main/java/org/apache/nifi/processors/standard/
> QueryDatabaseTable.java#L191)
>
> A minor notice for future reference by other users: it took me a while to
> get the PropertyDescriptor working because declaring it was not enough to
> make it shown at Properties dialog. I had to enable and than disable the
> ScriptedProcessor it at least once to have it shown (guessing the code was
> not executed).
>
> Thanks again for all the support.
>
>
> 2017-11-14 15:52 GMT-02:00 Matt Burgess <mattyb149@apache.org>:
>
>> Mark,
>>
>> Good point, I forgot the ScriptedLookupService is itself a
>> ConfigurableComponent and can add its own properties. The original
>> script from my blog comes from ExecuteScript, where you can't define
>> your own properties. I was just trying to make that work instead of
>> thinking about the actual problem, d'oh!
>>
>> Eric, rather than trying to get at a DBCPConnectionPool defined
>> elsewhere, you can add a property from your ScriptedLookupService that
>> is itself a reference to DBCPService. Then the user will see in the
>> properties a dropdown list of DBCPConnectionPool instances, just like
>> the other processors that use them (ExecuteSQL, e.g.). Mark outlined
>> that approach, and it is definitely way better. Sorry for the wild
>> goose chase, although I guess it was only me that wasted my time :P
>> Guess it's time to add a new post using this technique instead!
>>
>> Thanks,
>> Matt
>>
>>
>> On Tue, Nov 14, 2017 at 12:43 PM, Mark Payne <markap14@hotmail.com>
>> wrote:
>> > Matt, Eric,
>> >
>> > The typical pattern that you would follow for obtaining a Controller
>> Service would be to
>> > return a property that uses the identifiesControllerService() method.
>> For example:
>> >
>> > static final PropertyDescriptor MYSQL_CONNECTION_POOL = new
>> PropertyDescriptor.Builder()
>> > .name("Connection Pool")
>> > .identifiesControllerService(DBCPService.class)
>> > .required(true)
>> > .build();
>> >
>> > Then, to obtain that controller service, you would access it as:
>> >
>> > final DBCPService connectionPoolService = context.getProperty(MYSQL_CONN
>> ECTION_POOL).asControllerService(DBCPService.class)
>> >
>> > This allows the user to explicitly choose which controller service that
>> they want to use.
>> >
>> > Attempting to obtain a Controller Service by name will certainly cause
>> some problems,
>> > as you have already learned :) The API was not intended to work that
>> way, so you see
>> > that doing so can become difficult. There are a few reasons that we
>> don't want to retrieve the service
>> > by name:
>> >
>> > 1. It would require that the user know that they need to have a service
>> with
>> > that name. Then they would have to leave the configuration of your
>> service, and they would
>> > have to create a service with that name. Plus it would likely not be
>> obvious that they need to do this.
>> > 2. The framework would not know that your service is referencing the
>> connection pool service, so if
>> > the connection pool service is disabled, your service would still be
>> valid and the lifecycle management
>> > would not work as intended.
>> > 3. Controller Service names are not unique. So you may get the wrong
>> one if there are multiple with
>> > the same name. In fact, over 10 different iterations you could get 10
>> different services instead of always
>> > getting the same service.
>> >
>> > So I guess the question is: Is there a reason that the typical approach
>> of identifying the service in a
>> > Property Descriptor doesn't work for your use case?
>> >
>> > Thanks
>> > -Mark
>> >
>> >
>> >
>> >
>> >
>> >> On Nov 14, 2017, at 12:11 PM, Matt Burgess <mattyb149@apache.org>
>> wrote:
>> >>
>> >> Eric,
>> >>
>> >> So I just learned ALOT about the bowels of the context and
>> >> initialization framework while digging into this issue, and needless
>> >> to say we will need a better way of making this available to scripts.
>> >> Here's some info:
>> >>
>> >> 1) The ControllerServiceInitializationContext object passed into
>> >> initialize() is an anonymous object that passes along the
>> >> ScriptedLookupService's context objects, such as the
>> >> ControllerServiceLookup.
>> >> 2) The ControllerServiceLookup interface does not have a method
>> >> signature for getControllerServiceIdentifiers(Class, String) to pass
>> >> in the process group id.
>> >> 3) The ControllerServiceLookup object returned by the
>> >> ControllerServiceInitializationContext.getControllerServiceLookup()
>> >> method is a StandardControllerServiceInitializationContext
>> >> 4) Note that the context object passed into the initialize() method
>> >> and the one returned by context.getControllerServiceLookup() are
>> >> different (but both are ControllerServiceInitializationContext impls)
>> >> 5) The StandardControllerServiceInitializationContext object contains
>> >> a private ControllerServiceProvider called serviceProvider of type
>> >> StandardControllerServiceProvider, the anonymous context object does
>> >> not
>> >> 6) The StandardControllerServiceInitializationContext object delegates
>> >> the getControllerServiceIdentifiers(Class) method to the
>> >> serviceProvider
>> >> 7) serviceProvider (a StandardControllerServiceProvider) does not
>> >> allow the call to the getControllerServiceIdentifiers(Class)
>> >> signature, and instead throws the error you're seeing
>> >> 8) None of these objects can get at the process group ID. This is
>> >> because they are not associated with a ConfigurableComponent
>> >> 9) ScriptedLookupService, after calling the script's initialize()
>> >> method, will then call the script's onEnabled(ConfigurationContext)
>> >> method if it exists. This is currently undocumented [1]
>> >> 10) The script's onEnabled(ConfigurationContext) method will get a
>> >> StandardConfigurationContext object
>> >> 11) The StandardConfigurationContext object has a private
>> >> ConfiguredComponent named component, it is actually a
>> >> StandardControllerServiceNode object
>> >> 12) You can get the process group ID by calling the component's
>> >> getProcessGroupIdentifier() method
>> >> 13) The StandardConfigurationContext object also has a private
>> >> ControllerServiceLookup named serviceLookup, it is actually a
>> >> StandardControllerServiceProvider object
>> >> 14) Since we can get a process group ID from #11-12, we can now call
>> >> the supported method on the ControllerServiceProvider interface,
>> >> namely getControllerServiceIdentifiers(Class, String)
>> >> 15) Getting at private members (#11 &13) is allowed in Groovy, but
>> >> IIRC only works if you don't have a security manager/policies on the
>> >> JVM.
>> >>
>> >> TL;DR You can't currently get controller services by name in the
>> >> initialize() method, you have to implement onEnabled instead. If you
>> >> want to use logging, however, you'll need to save off the logger in
>> >> the initialize() method. Here's a working version of onEnabled:
>> >>
>> >> void onEnabled(ConfigurationContext context) {
>> >>    lookup = context.serviceLookup
>> >>    processGroupId = context.component?.processGroupIdentifier
>> >>    /* Get sql-connection */
>> >>    def dbcpServiceId =
>> >> lookup.getControllerServiceIdentifiers(ControllerService,
>> >> processGroupId).find {
>> >>      cs -> lookup.getControllerServiceName(cs) ==
>> 'MySQLConnectionPool'
>> >>    }
>> >>    def conn = lookup.getControllerService(db
>> cpServiceId)?.getConnection()
>> >>  }
>> >>
>> >> Hope this helps. I will think some more on how to make everything
>> >> fluid and legit -- Mark Payne, could use your help here :)
>> >>
>> >> Regards,
>> >> Matt
>> >>
>> >> On Tue, Nov 14, 2017 at 6:13 AM, Eric Chaves <eric@uolet.com> wrote:
>> >>> Hi Folks,
>> >>>
>> >>> I need to get an instance of DBCPService inside my
>> ScriptedLookupService and
>> >>> for that I'm following Matt's post
>> >>> http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-exec
>> utescript.html
>> >>>
>> >>> In my groovy class I've overrided the initialize method and
>> performing the
>> >>> lookup there but I'm getting the following error:
>> >>>
>> >>> java.lang.UnsupportedOperationException: Cannot obtain Controller
>> Service
>> >>> Identifiers for service type interface
>> >>> org.apache.nifi.controller.ControllerService without providing a
>> Process
>> >>> Group Identifier
>> >>>
>> >>>
>> >>> @Override
>> >>>  void initialize(ControllerServiceInitializationContext context)
>> throws
>> >>> InitializationException {
>> >>>    log = context.logger
>> >>>    /* Get sql-connection */
>> >>>    def lookup = context.controllerServiceLookup
>> >>>    def dbcpServiceId =
>> >>> lookup.getControllerServiceIdentifiers(ControllerService).find {
>> >>>      cs -> lookup.getControllerServiceName(cs) ==
>> 'MySQLConnectionPool'
>> >>>    }
>> >>>    conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>> >>>    log.info("sql conn {}", conn)
>> >>>  }
>> >>>
>> >>> Is there other way to find service identifiers?
>> >>>
>> >>> Regards,
>> >
>>
>
>

Mime
View raw message