nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: NiFi processor to execute a Java class
Date Wed, 08 Aug 2018 15:06:05 GMT
Vitaly,

I am indeed back from vacation and I started on the ExecuteClass
processor but never wrote the Jira because I ran into a snag, namely
how to get the FlowFile content into the class. In
ExecuteStreamCommand and ExecuteProcess we create an external Process
and route the content to the process's standard input, and collect the
standard output from the "other side". I'd hoped ExecuteClass would be
faster and not rely on an external process (i.e. a new JVM), but I
also we don't want to redirect standard input/output for all of NiFi
into the one class.  This prevents us from being able to "execute a
Java class" in the sense of calling the main() method on some runnable
class.

So my current thought is that we need to have a convention for a
method signature that ExecuteClass could call. I'm leery of creating a
Java Interface that the class would implement, since the external
class would have to bring in some dependency at compile time, and we'd
have to use reflection to instantiate the class anyway. So I'm
thinking maybe we have a property for the name of the method on the
class, and that method is expected to have the parameters
(InputStream, OutputStream, String[]). The method would read the flow
file content from the InputStream and write any output to
OutputStream. The String[] would be the array of arguments from the
processor property, passed in similarly to how
ExecuteStreamCommand/Process works, to enable dynamic configuration of
the class.  We should probably move this discussion to the dev mailing
list if you're interesting in talking implementation details :)

The ExecuteScript cookbook isn't a real book, it's just at that link I
sent in the last email. I thought about maybe extending it into a
GitHub book someday, but I would probably pick 2-3 languages to do
more complex "recipes", as even the simple recipes took quite a while
to do in the 5 or so languages I had in the original. Once we require
Java 9 I'll be looking to add Java as a scripting language to those
components (via JShell), but will still probably focus on Groovy,
Jython, and maybe Javascript (although IIRC Nashorn will be
deprecated/removed in Java 11).

Regards,
Matt

On Wed, Aug 8, 2018 at 10:39 AM Vitaly Krivoy
<Vitaly_Krivoy@jhancock.com> wrote:
>
> Hi Matt,
>
> I just saw you message. In case you are back from your vacation and still want to talk
software, may I ask you a couple of questions?
> There is no description of ExecuteClass NiFi processor in version 1.7 and googling for
it returns nothing. Is this something you wrote? Where can I find it?
> You’ve mentioned that you wrote a cook book on NiFi patterns. Did you publish it anywhere?
Thanks.
>
> Vitaly Krivoy
>
>
>
> From: Matt Burgess <mattyb149@gmail.com>
> Sent: Tuesday, July 17, 2018 2:15 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi processor to execute a Java class
>
>
>
> In addition to Andy’s great links and references, for the scripting stuff I wrote a
little cookbook to help with the NiFi API in the scripting processors [1], and I play around
with the scripting stuff quite a bit and keep a blog with fun stuff I experiment with [2].
>
>
>
> Also your use case is fairly popular, I think we should consider an ExecuteClass processor,
which would work much like ExecuteStreamCommand (and/or ExecuteProcess), except you’d specify
a class that has a main method, rather than an external command. This would allow the same
behavior (flow file contents as input, resulting output as outgoing flow file contents), but
would allow the class to run in NiFi’s JVM, rather than having to call out to a separate
JVM, or having to deal with a ScriptEngine as we do with the scripting processors.
>
>
>
> I’m on vacation and not near a computer ATM, but will write this up as a new feature
Jira when I rejoin civilization :)
>
>
>
> Regards,
>
> Matt
>
>
>
> [1] https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html
>
>
>
> [2] http://funnifi.blogspot.com
>
>
>
>
> On Jul 17, 2018, at 1:45 PM, Vitaly Krivoy <Vitaly_Krivoy@jhancock.com> wrote:
>
> Now this is awesome! Thanks!
>
>
>
> From: Andy LoPresto [mailto:alopresto@apache.org]
> Sent: Tuesday, July 17, 2018 12:29 PM
> To: Vitaly Krivoy <Vitaly_Krivoy@jhancock.com>
> Cc: users@nifi.apache.org
> Subject: Re: NiFi processor to execute a Java class
>
>
>
> Groovy is line-by-line compatible with pure Java and compiles to byte code. You can write
a script that uses only Java and select the Groovy option, and it will work fine.
>
>
>
> ExecuteStreamCommand does require an incoming connection because the intent of that processor
is to pipe input from an existing flowfile to some shell command, and then pipe the output
back into flowfile content. If you would like to run a shell command without providing input,
ExecuteProcess [1] is designed to do that.
>
>
>
> If the creation of incoming connections and processors is difficult, I recommend you
read the NiFi Getting Started Guide [2]. If you have already read that and feel these concepts
are not sufficiently explained, please let us know where we can make improvements to help
new users understand the system.
>
>
>
> [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.1/org.apache.nifi.processors.standard.ExecuteProcess/index.html
>
> [2] https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#i-started-nifi-now-what
>
>
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> alopresto.apache@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Jul 17, 2018, at 8:56 AM, Vitaly Krivoy <Vitaly_Krivoy@jhancock.com> wrote:
>
>
>
> Andy,
>
>
> Thank you for your detailed example. Unfortunately I don’t know Groovy though from
your example I see that it’s syntax is Scala-like and rather transparent.
> My preference at this point is to use pure Java, if possible.
> I managed to get my Java class executed by composing the following pipeline.
> GenerateFlowFile -> ExecuteStreamCommand -> PutFile
>                                                           |
>
>                                                          \/
>                                            jar with custom Java class
>
>
> The hardest part was figuring out that I must place GenerateFlowFile processor before
ExecuteStreamCommand in order for it to work.
> While documentation of ExecuteStreamCommand processor says that it must have an incoming
connection, it says nothing about how to go about creating it. This is especially counterintuitive
when a custom Java class jar doesn’t need any input from NiFi flow. It’s a kind of a situation
where one really wishes for a cookbook on NiFi.
>
>
>
> From: Andy LoPresto [mailto:alopresto@apache.org]
> Sent: Monday, July 16, 2018 1:21 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi processor to execute a Java class
>
>
>
> You can execute arbitrary Java logic via ExecuteScript and InvokeScriptedProcessor as
well. Simply import the relevant Java class as usual, and invoke the methods as you would
in any other scenario. It will be easiest to do this with Groovy as your scripting language,
and ensure that the JAR containing the classes is available in the “modules” directory
configured on the processor.
>
>
>
>
>
> import org.apache.commons.io.IOUtils
>
> import java.nio.charset.*
>
> import com.jhancock.CustomService
>
>
>
> def flowFile = session.get()
>
> if(!flowFile) return
>
>
>
> flowFile = session.write(flowFile, {inputStream, outputStream ->
>
>    // For demo purpose only; this is not memory-nice
>
>    def flowFileContents = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
>
>
>
>    // Do something with your service
>
>    CustomService myCustomService = new CustomService()
>
>    def result = myCustomService.doSomething(flowFileContents)
>
>
>
>    outputStream.write(result.getBytes(StandardCharsets.UTF_8))
>
> } as StreamCallback)
>
>
>
> flowFile = session.putAttribute(flowFile, ‘custom_service_performed', ’true')
>
> session.transfer(flowFile, REL_SUCCESS)
>
>
>
>
>
>
>
>
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> alopresto.apache@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Jul 16, 2018, at 8:23 AM, Mike Thomsen <mikerthomsen@gmail.com> wrote:
>
>
>
> As far as I know, it'll capture the stdout/stderr output of your Java class and send
that to a flowfile. Anything beyond that you'd need to do yourself.
>
>
>
> On Mon, Jul 16, 2018 at 11:18 AM Vitaly Krivoy <Vitaly_Krivoy@jhancock.com> wrote:
>
> Thank you Mike. The objective of my custom class is to query data from salesforce. Will
ExecuteStreamCommand processor automatically capture the output of my custom Java class and
generate flow files, or do I have to implement this capability myself?
>
>
>
> From: Mike Thomsen [mailto:mikerthomsen@gmail.com]
> Sent: Monday, July 16, 2018 11:07 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi processor to execute a Java class
>
>
>
> The REPL is not going to help unless it provides a ScriptEngine implementation, which
I don't believe it does. At this point, I think your only option between the three is ExecuteStreamCommand.
>
>
>
> On Mon, Jul 16, 2018 at 10:54 AM Vitaly Krivoy <Vitaly_Krivoy@jhancock.com> wrote:
>
> What would be the best NiFi processor to execute a Java class? I’ve seen some references
to ExecuteScript, InvokeScriptedProcessor and ExecuteStreamCommand processors, but nothing
conclusive. Speaking of ExecuteScript processor - would it work with Java 8, or should it
be used with Java 9 or later because those releases have a REPL loop?  Thanks.
>
> Vitaly Krivoy
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email message and any
attachments may be confidential and legally privileged and is intended for the use of the
addressee(s) only. If you are not an intended recipient, please: (1) notify me immediately
by replying to this message; (2) do not use, disseminate, distribute or reproduce any part
of the message or any attachment; and (3) destroy all copies of this message and any attachments.
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email message and any
attachments may be confidential and legally privileged and is intended for the use of the
addressee(s) only. If you are not an intended recipient, please: (1) notify me immediately
by replying to this message; (2) do not use, disseminate, distribute or reproduce any part
of the message or any attachment; and (3) destroy all copies of this message and any attachments.
>
>
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email message and any
attachments may be confidential and legally privileged and is intended for the use of the
addressee(s) only. If you are not an intended recipient, please: (1) notify me immediately
by replying to this message; (2) do not use, disseminate, distribute or reproduce any part
of the message or any attachment; and (3) destroy all copies of this message and any attachments.
>
>
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email message and any
attachments may be confidential and legally privileged and is intended for the use of the
addressee(s) only. If you are not an intended recipient, please: (1) notify me immediately
by replying to this message; (2) do not use, disseminate, distribute or reproduce any part
of the message or any attachment; and (3) destroy all copies of this message and any attachments.
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email message and any
attachments may be confidential and legally privileged and is intended for the use of the
addressee(s) only. If you are not an intended recipient, please: (1) notify me immediately
by replying to this message; (2) do not use, disseminate, distribute or reproduce any part
of the message or any attachment; and (3) destroy all copies of this message and any attachments.

Mime
View raw message