nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "A.B. Srinivasan" <>
Subject Re: Capturing stdout *and* stderr of command executed by ExecuteStreamCommandProcessor
Date Wed, 14 Dec 2016 05:39:54 GMT
Hi Koji,

Thanks for your very detailed response and for your efforts in creating and
publishing the example. I had missed the 'execution.error' entirely -
thanks for pointing that out. I do see useful info (such as the OutOfMemory
error) in some of the failed runs of my Java program in the
'execution.error' attribute. That should certainly provide better
visibility on some of the failures I run into.

I will keep an eye out for an instance where the Java program fails and /
or seems hung in a way that it doesn't break out and / or follow through to
completion to set an execution.status. The last time I ran into that
situation, I attempted to take a thread dump (using 'kill -3 <pid>) but
then did not know where to look for the resulting output apart from the
'filename' path which I had set up to receive the output stream for that
Java program. And it did not have the output from the thread dump. I
presume ExecuteStreamCommand redirects the output to an interim buffer (it
sure would be nice have visibility into the interim output being generated
much like 'tail -f') for the output stream that probably gets flushed to
the path specified by the 'filename' attribute only on program completion.
When / if I run into that situation, I will try to see if I can better
details that can help reproduce the situation.

Meanwhile - thanks again for your help,

On Tue, Dec 13, 2016 at 5:28 PM, Koji Kawamura <>

> Hello A.B.
> I wrote a simple Java class which produces OOM error, and tested how
> ExecuteStreamCommand works.
> In the case that after the Java program writes some output, then
> encounters OOM, it sends an outgoing flow file to "output stream"
> containing data produced until the OOM.
> The OOM error message can be seen at the "execution.error" attribute
> of the outgoing flow file. Also, "execution.status" attribute is set
> as "1" (none zero) indicating the command failed.
> If you'd like to split subsequent flow based on whether the Java
> program succeeded, then adding RouteOnAttribute after
> ExecuteStreamCommand.
> Add 'success' route with NiFi expression
> "${execution.status:equals('0')}" can route flow files.
> I put these experiment result as a Gist:
> Hope this helps.
> Please let us know if you're seeing different result, there may be a
> corner case.
> Thanks,
> Koji
> On Tue, Dec 13, 2016 at 4:14 PM, A.B. Srinivasan
> <> wrote:
> > Hello all,
> >
> > I have a Java program being invoked within a ExecuteStreamCommand
> processor.
> > Prior to the invocation of the ExecuteStreamCommand, I set up the
> 'filename'
> > property containing the output file into which the ExecuteStreamCommand
> > processor directs its output stream.
> >
> > This works pretty well to capture logger messages output by the Java
> > program. However, on the rare occasion when the Java program dies with
> > something like an OutOfMemory error, the message is lost and I have no
> info
> > on why the program failed.
> > Additionally, if the Java program dies fairly early in its launch before
> the
> > loggers are set up, ExecuteStreamCommand does not even break out and
> return
> > an error.
> > In both of these situations, I have to run the program standalone
> outside of
> > the ExecuteStreamCommand to see the errors.
> >
> > What configuration options do I have for the ExecuteStreamCommand
> processor
> > / its output stream to both capture *all* output directed to stdout and
> > stderr and reliably be able to break out and return if the Java program
> has
> > failed to launch?
> >
> > Thanks,
> > A.B.

View raw message