nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James McMahon <jsmcmah...@gmail.com>
Subject Re: InvokeScriptedProcessor to set common flowfile attributes?
Date Wed, 20 Sep 2017 14:53:32 GMT
Sounds very promising. Thank you very much for pointing me in the right
direction Matt. I will start digging into this immediately, and if I'm able
to get it to work will follow-up later to post my solution.  --Jim

On Wed, Sep 20, 2017 at 10:04 AM, Matt Burgess <mattyb149@apache.org> wrote:

> Jim,
>
> You should be able to do everything you want with
> InvokeScriptedProcessor and a handy utility class in NiFi called
> SynchronousFileWatcher [1].  It is used by various NiFi components
> such as the ScanAttribute [2] processor. You should be able to import
> it via:
>
> from org.apache.nifi.util.file.monitor import SynchronousFileWatcher
>
> Check the code/javadoc for usage, you should also be able to port the
> relevant ScanAttribute code to Jython as well.  One caveat is that the
> Processor interface doesn't have a shutdown/stop method, and
> InvokeScriptedProcessor does not yet support annotated lifecycle
> methods (NIFI-2215 [3]), so be aware that there might be cleanup
> and/or memory leak issues if the ISP is started and stopped alot.
> However using SynchronousFileWatcher should allow you to keep the ISP
> running as long as you like.
>
> Regards,
> Matt
>
> [1] https://github.com/apache/nifi/blob/master/nifi-commons/
> nifi-utils/src/main/java/org/apache/nifi/util/file/monitor/
> SynchronousFileWatcher.java
> [2] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> main/java/org/apache/nifi/processors/standard/ScanAttribute.java#L153
> [3] https://issues.apache.org/jira/browse/NIFI-2215
>
> On Wed, Sep 20, 2017 at 8:43 AM, James McMahon <jsmcmahon3@gmail.com>
> wrote:
> > Our NiFi is 0.7.x. I understand that it is possible to set
> customer-specific
> > configuration parameters in a second config file similar to
> nifi.properties.
> > I have this requirement:
> >
> > R1- allow engineering leads to set valid message types in a simple python
> > list, like so:  validTypes = ['larry','curly','moe']
> >
> > R2- application specific parms such as the validTypes list should be
> easily
> > editable by lead engineers who have no knowledge of NiFi, and so should
> be
> > maintained in a simple system file (text or xml, for example)
> >
> > R3- after editing this file, a NiFi restart should not be required.
> Changes
> > should be auto-detected by a processor in the workflow.
> >
> > I'd like to ask three related questions.
> >
> > Q1- If I was to set such parms in the second configuration file, how
> would I
> > load those values into attributes that would be associated with each
> > flowfile handled by my workflow? Can you refer me to an example?
> >
> > Q2- I'm given to understand that the approach for Q1 would require a
> restart
> > of nifi in order to pick up the new values. Is it possible instead to
> use an
> > InvokeScriptedProcessor (ISP) in my workflow to do this instead? I have
> used
> > ISP to redirect log messages to a log file of my choosing. ISP permits
> you
> > to define actions that happen only once at processor start, and so do not
> > get repeated for each and every flowfile handled by the processor. I set
> up
> > logging in this manner now in a python script executed by ISP:
> > def initialize(self, context):
> >      try:
> >           self.logger = logging.getLogger(....)
> >           self.logger.setLevel(logging.DEBUG)
> >      except:
> >           ....
> >
> > How can I set attributes - common attributes - inside that initialize()
> > method that will be inherited by each flowfile passing through my ISP
> > instance?
> >      I am hoping this is possible, because then edits to a simple text
> file
> > will not require a restart of nifi following changes - I will only have
> to
> > restart our ISP. And while this is not entirely "hands free" as required
> by
> > #R3, it is preferred to disrupting the entire nifi instance with a
> restart.
> >
> > Q3- How can I incorporate processors in my workflow to auto-detect a
> change
> > in a system file, and reload those system file contents to attributes?
> >
> > Thanks in advance for your guidance.  -Jim
>

Mime
View raw message