nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Villard <pierre.villard...@gmail.com>
Subject Re: NiFi processor to execute a Java class
Date Wed, 08 Aug 2018 17:21:08 GMT
Quick remark: just wondering if (InputStream, OutputStream,
Map<String,String>) would be a better approach to consider flow files
attributes + dynamic properties configured at processor level.

Pierre

2018-08-08 17:06 GMT+02:00 Matt Burgess <mattyb149@apache.org>:

> Vitaly,
>
> I am indeed back from vacation and I started on the ExecuteClass
> processor but never wrote the Jira because I ran into a snag, namely
> how to get the FlowFile content into the class. In
> ExecuteStreamCommand and ExecuteProcess we create an external Process
> and route the content to the process's standard input, and collect the
> standard output from the "other side". I'd hoped ExecuteClass would be
> faster and not rely on an external process (i.e. a new JVM), but I
> also we don't want to redirect standard input/output for all of NiFi
> into the one class.  This prevents us from being able to "execute a
> Java class" in the sense of calling the main() method on some runnable
> class.
>
> So my current thought is that we need to have a convention for a
> method signature that ExecuteClass could call. I'm leery of creating a
> Java Interface that the class would implement, since the external
> class would have to bring in some dependency at compile time, and we'd
> have to use reflection to instantiate the class anyway. So I'm
> thinking maybe we have a property for the name of the method on the
> class, and that method is expected to have the parameters
> (InputStream, OutputStream, String[]). The method would read the flow
> file content from the InputStream and write any output to
> OutputStream. The String[] would be the array of arguments from the
> processor property, passed in similarly to how
> ExecuteStreamCommand/Process works, to enable dynamic configuration of
> the class.  We should probably move this discussion to the dev mailing
> list if you're interesting in talking implementation details :)
>
> The ExecuteScript cookbook isn't a real book, it's just at that link I
> sent in the last email. I thought about maybe extending it into a
> GitHub book someday, but I would probably pick 2-3 languages to do
> more complex "recipes", as even the simple recipes took quite a while
> to do in the 5 or so languages I had in the original. Once we require
> Java 9 I'll be looking to add Java as a scripting language to those
> components (via JShell), but will still probably focus on Groovy,
> Jython, and maybe Javascript (although IIRC Nashorn will be
> deprecated/removed in Java 11).
>
> Regards,
> Matt
>
> On Wed, Aug 8, 2018 at 10:39 AM Vitaly Krivoy
> <Vitaly_Krivoy@jhancock.com> wrote:
> >
> > Hi Matt,
> >
> > I just saw you message. In case you are back from your vacation and
> still want to talk software, may I ask you a couple of questions?
> > There is no description of ExecuteClass NiFi processor in version 1.7
> and googling for it returns nothing. Is this something you wrote? Where can
> I find it?
> > You’ve mentioned that you wrote a cook book on NiFi patterns. Did you
> publish it anywhere? Thanks.
> >
> > Vitaly Krivoy
> >
> >
> >
> > From: Matt Burgess <mattyb149@gmail.com>
> > Sent: Tuesday, July 17, 2018 2:15 PM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi processor to execute a Java class
> >
> >
> >
> > In addition to Andy’s great links and references, for the scripting
> stuff I wrote a little cookbook to help with the NiFi API in the scripting
> processors [1], and I play around with the scripting stuff quite a bit and
> keep a blog with fun stuff I experiment with [2].
> >
> >
> >
> > Also your use case is fairly popular, I think we should consider an
> ExecuteClass processor, which would work much like ExecuteStreamCommand
> (and/or ExecuteProcess), except you’d specify a class that has a main
> method, rather than an external command. This would allow the same behavior
> (flow file contents as input, resulting output as outgoing flow file
> contents), but would allow the class to run in NiFi’s JVM, rather than
> having to call out to a separate JVM, or having to deal with a ScriptEngine
> as we do with the scripting processors.
> >
> >
> >
> > I’m on vacation and not near a computer ATM, but will write this up as a
> new feature Jira when I rejoin civilization :)
> >
> >
> >
> > Regards,
> >
> > Matt
> >
> >
> >
> > [1] https://community.hortonworks.com/articles/75032/
> executescript-cookbook-part-1.html
> >
> >
> >
> > [2] http://funnifi.blogspot.com
> >
> >
> >
> >
> > On Jul 17, 2018, at 1:45 PM, Vitaly Krivoy <Vitaly_Krivoy@jhancock.com>
> wrote:
> >
> > Now this is awesome! Thanks!
> >
> >
> >
> > From: Andy LoPresto [mailto:alopresto@apache.org]
> > Sent: Tuesday, July 17, 2018 12:29 PM
> > To: Vitaly Krivoy <Vitaly_Krivoy@jhancock.com>
> > Cc: users@nifi.apache.org
> > Subject: Re: NiFi processor to execute a Java class
> >
> >
> >
> > Groovy is line-by-line compatible with pure Java and compiles to byte
> code. You can write a script that uses only Java and select the Groovy
> option, and it will work fine.
> >
> >
> >
> > ExecuteStreamCommand does require an incoming connection because the
> intent of that processor is to pipe input from an existing flowfile to some
> shell command, and then pipe the output back into flowfile content. If you
> would like to run a shell command without providing input, ExecuteProcess
> [1] is designed to do that.
> >
> >
> >
> > If the creation of incoming connections and processors is difficult, I
> recommend you read the NiFi Getting Started Guide [2]. If you have already
> read that and feel these concepts are not sufficiently explained, please
> let us know where we can make improvements to help new users understand the
> system.
> >
> >
> >
> > [1] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi/nifi-standard-nar/1.7.1/org.apache.nifi.processors.standard.
> ExecuteProcess/index.html
> >
> > [2] https://nifi.apache.org/docs/nifi-docs/html/getting-
> started.html#i-started-nifi-now-what
> >
> >
> >
> >
> >
> > Andy LoPresto
> >
> > alopresto@apache.org
> >
> > alopresto.apache@gmail.com
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> >
> >
> > On Jul 17, 2018, at 8:56 AM, Vitaly Krivoy <Vitaly_Krivoy@jhancock.com>
> wrote:
> >
> >
> >
> > Andy,
> >
> >
> > Thank you for your detailed example. Unfortunately I don’t know Groovy
> though from your example I see that it’s syntax is Scala-like and rather
> transparent.
> > My preference at this point is to use pure Java, if possible.
> > I managed to get my Java class executed by composing the following
> pipeline.
> > GenerateFlowFile -> ExecuteStreamCommand -> PutFile
> >                                                           |
> >
> >                                                          \/
> >                                            jar with custom Java class
> >
> >
> > The hardest part was figuring out that I must place GenerateFlowFile
> processor before ExecuteStreamCommand in order for it to work.
> > While documentation of ExecuteStreamCommand processor says that it must
> have an incoming connection, it says nothing about how to go about creating
> it. This is especially counterintuitive when a custom Java class jar
> doesn’t need any input from NiFi flow. It’s a kind of a situation where one
> really wishes for a cookbook on NiFi.
> >
> >
> >
> > From: Andy LoPresto [mailto:alopresto@apache.org]
> > Sent: Monday, July 16, 2018 1:21 PM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi processor to execute a Java class
> >
> >
> >
> > You can execute arbitrary Java logic via ExecuteScript and
> InvokeScriptedProcessor as well. Simply import the relevant Java class as
> usual, and invoke the methods as you would in any other scenario. It will
> be easiest to do this with Groovy as your scripting language, and ensure
> that the JAR containing the classes is available in the “modules” directory
> configured on the processor.
> >
> >
> >
> >
> >
> > import org.apache.commons.io.IOUtils
> >
> > import java.nio.charset.*
> >
> > import com.jhancock.CustomService
> >
> >
> >
> > def flowFile = session.get()
> >
> > if(!flowFile) return
> >
> >
> >
> > flowFile = session.write(flowFile, {inputStream, outputStream ->
> >
> >    // For demo purpose only; this is not memory-nice
> >
> >    def flowFileContents = IOUtils.toString(inputStream,
> StandardCharsets.UTF_8)
> >
> >
> >
> >    // Do something with your service
> >
> >    CustomService myCustomService = new CustomService()
> >
> >    def result = myCustomService.doSomething(flowFileContents)
> >
> >
> >
> >    outputStream.write(result.getBytes(StandardCharsets.UTF_8))
> >
> > } as StreamCallback)
> >
> >
> >
> > flowFile = session.putAttribute(flowFile, ‘custom_service_performed',
> ’true')
> >
> > session.transfer(flowFile, REL_SUCCESS)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Andy LoPresto
> >
> > alopresto@apache.org
> >
> > alopresto.apache@gmail.com
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> >
> >
> > On Jul 16, 2018, at 8:23 AM, Mike Thomsen <mikerthomsen@gmail.com>
> wrote:
> >
> >
> >
> > As far as I know, it'll capture the stdout/stderr output of your Java
> class and send that to a flowfile. Anything beyond that you'd need to do
> yourself.
> >
> >
> >
> > On Mon, Jul 16, 2018 at 11:18 AM Vitaly Krivoy <
> Vitaly_Krivoy@jhancock.com> wrote:
> >
> > Thank you Mike. The objective of my custom class is to query data from
> salesforce. Will ExecuteStreamCommand processor automatically capture the
> output of my custom Java class and generate flow files, or do I have to
> implement this capability myself?
> >
> >
> >
> > From: Mike Thomsen [mailto:mikerthomsen@gmail.com]
> > Sent: Monday, July 16, 2018 11:07 AM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi processor to execute a Java class
> >
> >
> >
> > The REPL is not going to help unless it provides a ScriptEngine
> implementation, which I don't believe it does. At this point, I think your
> only option between the three is ExecuteStreamCommand.
> >
> >
> >
> > On Mon, Jul 16, 2018 at 10:54 AM Vitaly Krivoy <
> Vitaly_Krivoy@jhancock.com> wrote:
> >
> > What would be the best NiFi processor to execute a Java class? I’ve seen
> some references to ExecuteScript, InvokeScriptedProcessor and
> ExecuteStreamCommand processors, but nothing conclusive. Speaking of
> ExecuteScript processor - would it work with Java 8, or should it be used
> with Java 9 or later because those releases have a REPL loop?  Thanks.
> >
> > Vitaly Krivoy
> >
> >
> >
> > STATEMENT OF CONFIDENTIALITY The information contained in this email
> message and any attachments may be confidential and legally privileged and
> is intended for the use of the addressee(s) only. If you are not an
> intended recipient, please: (1) notify me immediately by replying to this
> message; (2) do not use, disseminate, distribute or reproduce any part of
> the message or any attachment; and (3) destroy all copies of this message
> and any attachments.
> >
> >
> >
> > STATEMENT OF CONFIDENTIALITY The information contained in this email
> message and any attachments may be confidential and legally privileged and
> is intended for the use of the addressee(s) only. If you are not an
> intended recipient, please: (1) notify me immediately by replying to this
> message; (2) do not use, disseminate, distribute or reproduce any part of
> the message or any attachment; and (3) destroy all copies of this message
> and any attachments.
> >
> >
> >
> >
> >
> > STATEMENT OF CONFIDENTIALITY The information contained in this email
> message and any attachments may be confidential and legally privileged and
> is intended for the use of the addressee(s) only. If you are not an
> intended recipient, please: (1) notify me immediately by replying to this
> message; (2) do not use, disseminate, distribute or reproduce any part of
> the message or any attachment; and (3) destroy all copies of this message
> and any attachments.
> >
> >
> >
> >
> >
> > STATEMENT OF CONFIDENTIALITY The information contained in this email
> message and any attachments may be confidential and legally privileged and
> is intended for the use of the addressee(s) only. If you are not an
> intended recipient, please: (1) notify me immediately by replying to this
> message; (2) do not use, disseminate, distribute or reproduce any part of
> the message or any attachment; and (3) destroy all copies of this message
> and any attachments.
> >
> >
> >
> > STATEMENT OF CONFIDENTIALITY The information contained in this email
> message and any attachments may be confidential and legally privileged and
> is intended for the use of the addressee(s) only. If you are not an
> intended recipient, please: (1) notify me immediately by replying to this
> message; (2) do not use, disseminate, distribute or reproduce any part of
> the message or any attachment; and (3) destroy all copies of this message
> and any attachments.
>

Mime
View raw message