nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "davidrsmith@btinternet.com" <davidrsm...@btinternet.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Mon, 02 Nov 2015 22:39:04 GMT
Hi

Where I work we have created an attribute loggers of our own. It is a fairly simple affair
which used a regex to determine which attributes to log, and writes them as key value pairs
to a file, whose location is determined by a user properly. I'm happy to put this out there
if anyone is interested.

Sent from my HTC

----- Reply message -----
From: "Adam Taft" <adam@adamtaft.com>
Date: Mon, Nov 2, 2015 19:23
Subject: LogAttribute - Sending that output to a custom logger?
To: <dev@nifi.apache.org>

This thread has forked into two different conversations:  1. improvements
to LogAttribute processor; 2. improvements to processor documentation.

1)  re: improvements to LogAttribute - we already have NIFI-67 [1] that
suggests a number of improvements to LogAttribute.  One of these is the use
of a custom name for the logger so that logback rules can be written
against that name.

While the provenance engine is great for many scenarios, in my opinion, it
doesn't replace the need for true text-based logging.  The tooling for log
processing is very mature and there's no ability to "grep" a provenance
repository, migrate or offload provenance logs into deep storage, store log
events into a database, or do any other cool syslogd or logback type
things.  Being able to capture and log a flowfile at the exact right place
in the data flow and processing it using the command line is an extremely
valuable tool in the toolkit.

For a long time, I've wanted to work on at least some of the things
mentioned in NIFI-67 and will hopefully get to do so time willing.  Having
a custom "name" for the LogAttribute processor seems like a no-brainer.
Contributions for this should definitely be welcome!

2) improvements to processor document - I agree, even as a somewhat
seasoned NIFI user, I still have a hard time reading and understanding the
processor documentation.  I often do exactly what Mark P. suggests and
instead go directly to the source.  Any contribution towards better
processor documentation is greatly appreciated!

[1] https://issues.apache.org/jira/browse/NIFI-67


On Mon, Nov 2, 2015 at 1:54 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:

> We greatly appreciate contributions.  Your prescribed approach sounds great
> and if you are willing to give us a few cycles pointing out, and optionally
> correcting, the items that are in need of improvement, we will certainly
> incorporate.
>
> Thanks!
>
> On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <markpetronic@gmail.com>
> wrote:
>
> > I'm sort of in the camp of "don't come with a complaint if you don't come
> > with a solution" and hesitated to even raise the documentation comment
> > without just fixing it myself. How about this, I just do some updates on
> > some processor docs myself and use that as my first contribution to work
> > through the process of committing to this project?
> >
> > But, to give you one quick example, EvaluateJSONPath (which, btw has
> pretty
> > good docs otherwise) does not mention HOW to extract the JSON you are
> > interested in. I had to look at the code to figure out it used this:
> > https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but,
> > as
> > a user, should I need to look at the code for such information? I submit,
> > no. Me personally, I like to dig into the code. So, this is more a
> comment
> > on "overall goodness" for the general new user experience.
> >
> > I agree with your assessment of 'new user vibe' as I am starting to not
> > notice it as much. lol
> >
> > On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <joe.witt@gmail.com> wrote:
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message