nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Taft <a...@adamtaft.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Tue, 03 Nov 2015 02:04:44 GMT
David,

This sounds like a slightly different use case than the NiFi standard
LogAttribute processor.  It sounds like your processor is more of a generic
attribute converter and file writer.  The LogAttribute processor is
designed to interact with the underlying NiFi logging subsystem, not
necessarily just to write files.

That being said, your processor may be a useful contribution to Apache
NiFi.  Specifically, the value-add of your processor might be in the
key-value format you've defined to output the flowfile attributes.  It
might be interesting to see this expressed as an attribute-to-payload
converter, chained together with potentially other processors like PutFile
in the dataflow.

If you want to contribute your processor, I would recommend making it
available on GitHub (or similar) for review by the Apache NiFi community.
Just post a link of your contribution here or even issue a pull request for
your processor.  It would at least be evaluated and considered for
inclusion.

Hope this helps.

Adam


On Mon, Nov 2, 2015 at 5:39 PM, davidrsmith@btinternet.com <
davidrsmith@btinternet.com> wrote:

> Hi
>
> Where I work we have created an attribute loggers of our own. It is a
> fairly simple affair which used a regex to determine which attributes to
> log, and writes them as key value pairs to a file, whose location is
> determined by a user properly. I'm happy to put this out there if anyone is
> interested.
>
> Sent from my HTC
>
>
> ----- Reply message -----
> From: "Adam Taft" <adam@adamtaft.com>
> Date: Mon, Nov 2, 2015 19:23
> Subject: LogAttribute - Sending that output to a custom logger?
> To: <dev@nifi.apache.org>
>
> This thread has forked into two different conversations:  1. improvements
> to LogAttribute processor; 2. improvements to processor documentation.
>
> 1)  re: improvements to LogAttribute - we already have NIFI-67 [1] that
> suggests a number of improvements to LogAttribute.  One of these is the use
> of a custom name for the logger so that logback rules can be written
> against that name.
>
> While the provenance engine is great for many scenarios, in my opinion, it
> doesn't replace the need for true text-based logging.  The tooling for log
> processing is very mature and there's no ability to "grep" a provenance
> repository, migrate or offload provenance logs into deep storage, store log
> events into a database, or do any other cool syslogd or logback type
> things.  Being able to capture and log a flowfile at the exact right place
> in the data flow and processing it using the command line is an extremely
> valuable tool in the toolkit.
>
> For a long time, I've wanted to work on at least some of the things
> mentioned in NIFI-67 and will hopefully get to do so time willing.  Having
> a custom "name" for the LogAttribute processor seems like a no-brainer.
> Contributions for this should definitely be welcome!
>
> 2) improvements to processor document - I agree, even as a somewhat
> seasoned NIFI user, I still have a hard time reading and understanding the
> processor documentation.  I often do exactly what Mark P. suggests and
> instead go directly to the source.  Any contribution towards better
> processor documentation is greatly appreciated!
>
> [1] https://issues.apache.org/jira/browse/NIFI-67
>
>
> On Mon, Nov 2, 2015 at 1:54 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
>
> > We greatly appreciate contributions.  Your prescribed approach sounds
> great
> > and if you are willing to give us a few cycles pointing out, and
> optionally
> > correcting, the items that are in need of improvement, we will certainly
> > incorporate.
> >
> > Thanks!
> >
> > On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <markpetronic@gmail.com>
> > wrote:
> >
> > > I'm sort of in the camp of "don't come with a complaint if you don't
> come
> > > with a solution" and hesitated to even raise the documentation comment
> > > without just fixing it myself. How about this, I just do some updates
> on
> > > some processor docs myself and use that as my first contribution to
> work
> > > through the process of committing to this project?
> > >
> > > But, to give you one quick example, EvaluateJSONPath (which, btw has
> > pretty
> > > good docs otherwise) does not mention HOW to extract the JSON you are
> > > interested in. I had to look at the code to figure out it used this:
> > > https://github.com/jayway/JsonPath. Ok, that was not hard, I admit,
> but,
> > > as
> > > a user, should I need to look at the code for such information? I
> submit,
> > > no. Me personally, I like to dig into the code. So, this is more a
> > comment
> > > on "overall goodness" for the general new user experience.
> > >
> > > I agree with your assessment of 'new user vibe' as I am starting to not
> > > notice it as much. lol
> > >
> > > On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <joe.witt@gmail.com> wrote:
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message