nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Using InvokeHTTP to send GET but without content
Date Sun, 08 Nov 2015 00:02:17 GMT
Mark,

What you're trying to with regard to auditing is exactly what the
ReportingTask extension is designed to support.  It has access to the
provenance events which include for each event the flow file
attributes.  If you built a ReportingTask which pulls whatever
attributes you want and formats requests however you want you could in
a rather straightforward manner report this data to a RESTful
endpoint.  That approach requires you to build your own custom
ReportingTask though.  As a reminder though probably not what you want
here you could also have the auditing service pull the information
from NiFi using NiFi's REST API.

Regarding access to grep/awk/etc.. for audit log/provenance type data.
There are others who have shared your viewpoint and I don't think
anyone is opposed to providing support for this. This again would be a
good thing to build as a reporting task where it would basically
serialize events of interest to some log file in some deterministic
format.  In fact perhaps we should just build a standard one in like
syslog format or something similar.  If folks have ideas here on a
good format to use please advise.  Now this approach simply means
you'll end up with reliably formatted log files.  You'd still need to
get this over to your REST endpoint but of course there are ways to
slice that.

If you'd like a zero-coding option then you should be able to approach
it as you are.  InvokeHTTP has a known issue where it requires flow
files to be present to fire off.  That is being addressed (potentially
as soon as Monday) [1].  In your case though you will have FlowFiles
available so the issue of why it isn't firing needs to be looked into
further.  Your use case where you don't want to send the content of
the FlowFile through makes sense.  I was surprised to see it wasn't
already an option.  That too would be a good JIRA as obviously simply
using the attributes themselves to craft a request would often be
sufficient.  Am actually surprised we've not already run into that.

[1] https://issues.apache.org/jira/browse/NIFI-1009

Thanks
Joe

On Sat, Nov 7, 2015 at 4:57 PM, Mark Petronic <markpetronic@gmail.com> wrote:
> I thought this seemed like a simple plan... I wanted to send an audit
> message to a REST server every time I process every file in my flow. The sub
> flow in question is:
>
> +---------------+   +--------------+           +---------+
> | UnpackContent +-->+ MergeContent +--merged+->+ PutFile |
> +---------------+   +------+-------+           +---------+
>                            |
>                            v original
>                    +-------+-------+    +----------+
>                    |UpdateAttribute+--->+InvokeHTTP|
>                    +---------------+    +----------+
>
> I want to record every zip file unpacked so I feed (original) into
> UpdateAttribute where I create a message like "Processed: ${ZipPath}", where
> ${ZipPath} was added as an attribute earlier in the flow when I pull in the
> zip files to process. I also want to send a another message from PutFile
> once the file is delivered but my drawing does not show that part yet. Same
> concept, hit a REST API to record the event. I also want to do this sort of
> 'general' audit logging for other parts of the flow. For example, I have a
> bunch of conditions/rule/actions to conditionally control various
> processing. They are set up for files and content types of files that I am
> aware of today. So, the [match] flow from some upstream UpdateAttribute will
> flow the files that match the definitions. But, if some unsuspecting file
> gets ingested, then it might NOT match a rule. So, I want to route the
> [unmatched] into this audit logging where I can then see this and decide
> what to do to handle this new file type. I know there is providence but I
> want long term audits and don't need to retain the flow file content and,
> frankly, it is just easier to work with plain audit logs (grep/awk/etc),
> IMO, and for my needs at this time.
>
> Problem is, InvokeHTTP never invokes. Why not? I think it would be nice if
> InvokeHTTP would hit the URL without a flow file, as in my case, I never
> want to actually send any body content but rather access attributes to build
> up a URL with param=value pairs to send. I am probably off base here. Any
> ideas? I saw this exchange but this seems kludgy using the generate flow
> file just to "trick" Invoke HTTP into doing something.
>
> https://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3CCALJK9a4U29Ua9zVT9oPCYHMg=k4Vneao30EMszRe3yPkz6xxJQ@mail.gmail.com%3E
>
> I am trying to adapt to the "Nifi" paradigm but feel like I am swimming
> upstream at times.
>
> Thanks
>
>

Mime
View raw message