nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaly Krivoy <Vitaly_Kri...@jhancock.com>
Subject RE: NiFi processor to execute a Java class
Date Wed, 08 Aug 2018 14:39:09 GMT
Hi Matt,
I just saw you message. In case you are back from your vacation and still want to talk software,
may I ask you a couple of questions? 😊
There is no description of ExecuteClass NiFi processor in version 1.7 and googling for it
returns nothing. Is this something you wrote? Where can I find it?
You’ve mentioned that you wrote a cook book on NiFi patterns. Did you publish it anywhere?
Thanks.

Vitaly Krivoy

From: Matt Burgess <mattyb149@gmail.com>
Sent: Tuesday, July 17, 2018 2:15 PM
To: users@nifi.apache.org
Subject: Re: NiFi processor to execute a Java class

In addition to Andy’s great links and references, for the scripting stuff I wrote a little
cookbook to help with the NiFi API in the scripting processors [1], and I play around with
the scripting stuff quite a bit and keep a blog with fun stuff I experiment with [2].

Also your use case is fairly popular, I think we should consider an ExecuteClass processor,
which would work much like ExecuteStreamCommand (and/or ExecuteProcess), except you’d specify
a class that has a main method, rather than an external command. This would allow the same
behavior (flow file contents as input, resulting output as outgoing flow file contents), but
would allow the class to run in NiFi’s JVM, rather than having to call out to a separate
JVM, or having to deal with a ScriptEngine as we do with the scripting processors.

I’m on vacation and not near a computer ATM, but will write this up as a new feature Jira
when I rejoin civilization :)

Regards,
Matt

[1] https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html

[2] http://funnifi.blogspot.com<http://funnifi.blogspot.com/?m=1>


On Jul 17, 2018, at 1:45 PM, Vitaly Krivoy <Vitaly_Krivoy@jhancock.com<mailto:Vitaly_Krivoy@jhancock.com>>
wrote:
Now this is awesome! Thanks!

From: Andy LoPresto [mailto:alopresto@apache.org]
Sent: Tuesday, July 17, 2018 12:29 PM
To: Vitaly Krivoy <Vitaly_Krivoy@jhancock.com<mailto:Vitaly_Krivoy@jhancock.com>>
Cc: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi processor to execute a Java class

Groovy is line-by-line compatible with pure Java and compiles to byte code. You can write
a script that uses only Java and select the Groovy option, and it will work fine.

ExecuteStreamCommand does require an incoming connection because the intent of that processor
is to pipe input from an existing flowfile to some shell command, and then pipe the output
back into flowfile content. If you would like to run a shell command without providing input,
ExecuteProcess [1] is designed to do that.

If the creation of incoming connections and processors is difficult, I recommend you read
the NiFi Getting Started Guide [2]. If you have already read that and feel these concepts
are not sufficiently explained, please let us know where we can make improvements to help
new users understand the system.

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.1/org.apache.nifi.processors.standard.ExecuteProcess/index.html
[2] https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#i-started-nifi-now-what


Andy LoPresto
alopresto@apache.org<mailto:alopresto@apache.org>
alopresto.apache@gmail.com<mailto:alopresto.apache@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jul 17, 2018, at 8:56 AM, Vitaly Krivoy <Vitaly_Krivoy@jhancock.com<mailto:Vitaly_Krivoy@jhancock.com>>
wrote:

Andy,

Thank you for your detailed example. Unfortunately I don’t know Groovy though from your
example I see that it’s syntax is Scala-like and rather transparent.
My preference at this point is to use pure Java, if possible.
I managed to get my Java class executed by composing the following pipeline.
GenerateFlowFile -> ExecuteStreamCommand -> PutFile
                                                          |
                                                         \/
                                           jar with custom Java class

The hardest part was figuring out that I must place GenerateFlowFile processor before ExecuteStreamCommand
in order for it to work.
While documentation of ExecuteStreamCommand processor says that it must have an incoming connection,
it says nothing about how to go about creating it. This is especially counterintuitive when
a custom Java class jar doesn’t need any input from NiFi flow. It’s a kind of a situation
where one really wishes for a cookbook on NiFi.

From: Andy LoPresto [mailto:alopresto@apache.org]
Sent: Monday, July 16, 2018 1:21 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi processor to execute a Java class

You can execute arbitrary Java logic via ExecuteScript and InvokeScriptedProcessor as well.
Simply import the relevant Java class as usual, and invoke the methods as you would in any
other scenario. It will be easiest to do this with Groovy as your scripting language, and
ensure that the JAR containing the classes is available in the “modules” directory configured
on the processor.



import org.apache.commons.io.IOUtils

import java.nio.charset.*

import com.jhancock.CustomService



def flowFile = session.get()

if(!flowFile) return



flowFile = session.write(flowFile, {inputStream, outputStream ->

   // For demo purpose only; this is not memory-nice

   def flowFileContents = IOUtils.toString(inputStream, StandardCharsets.UTF_8)



   // Do something with your service

   CustomService myCustomService = new CustomService()

   def result = myCustomService.doSomething(flowFileContents)



   outputStream.write(result.getBytes(StandardCharsets.UTF_8))

} as StreamCallback)



flowFile = session.putAttribute(flowFile, ‘custom_service_performed', ’true')

session.transfer(flowFile, REL_SUCCESS)





Andy LoPresto
alopresto@apache.org<mailto:alopresto@apache.org>
alopresto.apache@gmail.com<mailto:alopresto.apache@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jul 16, 2018, at 8:23 AM, Mike Thomsen <mikerthomsen@gmail.com<mailto:mikerthomsen@gmail.com>>
wrote:

As far as I know, it'll capture the stdout/stderr output of your Java class and send that
to a flowfile. Anything beyond that you'd need to do yourself.

On Mon, Jul 16, 2018 at 11:18 AM Vitaly Krivoy <Vitaly_Krivoy@jhancock.com<mailto:Vitaly_Krivoy@jhancock.com>>
wrote:
Thank you Mike. The objective of my custom class is to query data from salesforce. Will ExecuteStreamCommand
processor automatically capture the output of my custom Java class and generate flow files,
or do I have to implement this capability myself?

From: Mike Thomsen [mailto:mikerthomsen@gmail.com<mailto:mikerthomsen@gmail.com>]
Sent: Monday, July 16, 2018 11:07 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: NiFi processor to execute a Java class

The REPL is not going to help unless it provides a ScriptEngine implementation, which I don't
believe it does. At this point, I think your only option between the three is ExecuteStreamCommand.

On Mon, Jul 16, 2018 at 10:54 AM Vitaly Krivoy <Vitaly_Krivoy@jhancock.com<mailto:Vitaly_Krivoy@jhancock.com>>
wrote:
What would be the best NiFi processor to execute a Java class? I’ve seen some references
to ExecuteScript, InvokeScriptedProcessor and ExecuteStreamCommand processors, but nothing
conclusive. Speaking of ExecuteScript processor - would it work with Java 8, or should it
be used with Java 9 or later because those releases have a REPL loop?  Thanks.

Vitaly Krivoy

STATEMENT OF CONFIDENTIALITY The information contained in this email message and any attachments
may be confidential and legally privileged and is intended for the use of the addressee(s)
only. If you are not an intended recipient, please: (1) notify me immediately by replying
to this message; (2) do not use, disseminate, distribute or reproduce any part of the message
or any attachment; and (3) destroy all copies of this message and any attachments.

STATEMENT OF CONFIDENTIALITY The information contained in this email message and any attachments
may be confidential and legally privileged and is intended for the use of the addressee(s)
only. If you are not an intended recipient, please: (1) notify me immediately by replying
to this message; (2) do not use, disseminate, distribute or reproduce any part of the message
or any attachment; and (3) destroy all copies of this message and any attachments.


STATEMENT OF CONFIDENTIALITY The information contained in this email message and any attachments
may be confidential and legally privileged and is intended for the use of the addressee(s)
only. If you are not an intended recipient, please: (1) notify me immediately by replying
to this message; (2) do not use, disseminate, distribute or reproduce any part of the message
or any attachment; and (3) destroy all copies of this message and any attachments.


STATEMENT OF CONFIDENTIALITY The information contained in this email message and any attachments
may be confidential and legally privileged and is intended for the use of the addressee(s)
only. If you are not an intended recipient, please: (1) notify me immediately by replying
to this message; (2) do not use, disseminate, distribute or reproduce any part of the message
or any attachment; and (3) destroy all copies of this message and any attachments.

STATEMENT OF CONFIDENTIALITY The information contained in this email message and any attachments
may be confidential and legally privileged and is intended for the use of the addressee(s)
only. If you are not an intended recipient, please: (1) notify me immediately by replying
to this message; (2) do not use, disseminate, distribute or reproduce any part of the message
or any attachment; and (3) destroy all copies of this message and any attachments.
Mime
View raw message