nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Taft <a...@adamtaft.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Mon, 02 Nov 2015 19:23:52 GMT
This thread has forked into two different conversations:  1. improvements
to LogAttribute processor; 2. improvements to processor documentation.

1)  re: improvements to LogAttribute - we already have NIFI-67 [1] that
suggests a number of improvements to LogAttribute.  One of these is the use
of a custom name for the logger so that logback rules can be written
against that name.

While the provenance engine is great for many scenarios, in my opinion, it
doesn't replace the need for true text-based logging.  The tooling for log
processing is very mature and there's no ability to "grep" a provenance
repository, migrate or offload provenance logs into deep storage, store log
events into a database, or do any other cool syslogd or logback type
things.  Being able to capture and log a flowfile at the exact right place
in the data flow and processing it using the command line is an extremely
valuable tool in the toolkit.

For a long time, I've wanted to work on at least some of the things
mentioned in NIFI-67 and will hopefully get to do so time willing.  Having
a custom "name" for the LogAttribute processor seems like a no-brainer.
Contributions for this should definitely be welcome!

2) improvements to processor document - I agree, even as a somewhat
seasoned NIFI user, I still have a hard time reading and understanding the
processor documentation.  I often do exactly what Mark P. suggests and
instead go directly to the source.  Any contribution towards better
processor documentation is greatly appreciated!

[1] https://issues.apache.org/jira/browse/NIFI-67


On Mon, Nov 2, 2015 at 1:54 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:

> We greatly appreciate contributions.  Your prescribed approach sounds great
> and if you are willing to give us a few cycles pointing out, and optionally
> correcting, the items that are in need of improvement, we will certainly
> incorporate.
>
> Thanks!
>
> On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <markpetronic@gmail.com>
> wrote:
>
> > I'm sort of in the camp of "don't come with a complaint if you don't come
> > with a solution" and hesitated to even raise the documentation comment
> > without just fixing it myself. How about this, I just do some updates on
> > some processor docs myself and use that as my first contribution to work
> > through the process of committing to this project?
> >
> > But, to give you one quick example, EvaluateJSONPath (which, btw has
> pretty
> > good docs otherwise) does not mention HOW to extract the JSON you are
> > interested in. I had to look at the code to figure out it used this:
> > https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but,
> > as
> > a user, should I need to look at the code for such information? I submit,
> > no. Me personally, I like to dig into the code. So, this is more a
> comment
> > on "overall goodness" for the general new user experience.
> >
> > I agree with your assessment of 'new user vibe' as I am starting to not
> > notice it as much. lol
> >
> > On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Mark
> > >
> > > All fair points.  Can you please point out which processor docs
> > > specifically should be better.  Let's fix em..you will quickly lose
> that
> > > new user vibe and not notice what needs to improve as much.  We need to
> > > make the new user experience awesome.
> > >
> > > Thanks
> > > Joe
> > > On Nov 2, 2015 10:08 AM, "Mark Petronic" <markpetronic@gmail.com>
> wrote:
> > >
> > > > My primary use is for understanding Nifi. I like to direct various
> > > > processors output into both their logical next processor stage as
> well
> > as
> > > > into a log attribute processor. Then I tail the Nifi app log file and
> > > watch
> > > > what happens - in real time. I do not intend to use this for long
> term
> > > log
> > > > retention. I agree that providence is the right choice for that. So,
> > the
> > > > only reason I wanted to allow configuration of a custom logger was
> > simply
> > > > to isolate all the attribute-rich logging from the normal logging
> > > because I
> > > > was primarily interested in the attribute flows as a way to (a)
> better
> > > > understand what a processor emits because, frankly, the documentation
> > of
> > > > some of the processors is very sparse. So, I learn imperatively, so
> to
> > > > speak. I say that as a new user. I feel I should be able to get a
> > pretty
> > > > good understanding of a processor by reading the usage. But I am
> > finding
> > > > that the documentation, in some cases, is more like what I like to
> > refer
> > > to
> > > > as, "note to self" documentation. Great if you are the guy who wrote
> > the
> > > > processor with those "insights" - not so great if you are not the
> > > > developer. So, then I need to dig up the code. That should not be
> > needed
> > > as
> > > > the first step of understanding a processor as a new user. There is
> > some
> > > > well documented processors but not all are, IMHO. (b) Validate my
> flows
> > > > with some test data and verify attribute values look correct and
> > routing
> > > is
> > > > happen on them as expected, etc. Again, easier, IMO, to see in the
> logs
> > > > than digging into the providence data.
> > > >
> > > > Maybe this is just a good "private" feature for me so maybe I will
> just
> > > > create a private version to use on my own. I already have it working
> > but
> > > > would need more polish to achieve PR status. Maybe this is the sort
> of
> > > > thing that others would not find beneficial? That's fine. There are
> > > others
> > > > ways I can contribute in the future. I'm still having fun! :)
> > > >
> > > > On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <joe.witt@gmail.com>
> wrote:
> > > >
> > > > > Mark Petronic,
> > > > >
> > > > > I share Payne's perspective on this.  But I'd also like to work
> with
> > > > > you to better understand the workflow.  For those of us that have
> > used
> > > > > this tool for a long time there is a lot we take for granted from
a
> > > > > new user perspective.  We believe the provenance feature to
> provide a
> > > > > far superior option to understanding how an item went through the
> > > > > system and the timing and what we knew when and so on.  But, it
> would
> > > > > be great to understand it from your perspective as someone learning
> > > > > NiFi.  Not meaning to take away from your proposed contrib - that
> > > > > would be great too.  Just want to see if the prov user experience
> > > > > solves what you're looking for and if not can we make it do that.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <markap14@hotmail.com>
> > > > wrote:
> > > > > > Mark,
> > > > > >
> > > > > > To make sure that I understand what you're proposing, you want
to
> > > add a
> > > > > property to
> > > > > > LogAttribute that allows users to provide a custom logger name?
> > > > > >
> > > > > > If that is indeed what you are suggesting then I think it's
a
> great
> > > > idea.
> > > > > >
> > > > > > That being said, in practice I rarely ever use LogAttribute
and
> we
> > > even
> > > > > considered removing
> > > > > > it from the codebase before we open sourced, because the Data
> > > > Provenance
> > > > > provides a
> > > > > > much better view of what's going on to debug your flows.
> > > > > >
> > > > > > I know you're pretty new to NiFi, so if you've not yet had a
> chance
> > > to
> > > > > play with the Provenance,
> > > > > > you can see the section in the User Guide at
> > > > >
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > > <
> > > > >
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > > >
> > > > > >
> > > > > > If you're interested in updating the LogAttribute processor,
> > though,
> > > > > we'd be happy to have
> > > > > > that contribution added, as it does make the Processor more
> usable.
> > > > > >
> > > > > > Thanks
> > > > > > -Mark
> > > > > >
> > > > > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <
> > markpetronic@gmail.com
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> From the code, it appears it cannot be done as the attribute
> > logging
> > > > > >> goes the same getLogger() instance as the normal nifi-app
> traces.
> > > Has
> > > > > >> anyone considered making that configurable, maybe allowing
you
> do
> > > > > >> define a different logger name for LogAttribute then creating
> that
> > > > > >> logger definition in log back conf allowing flexibility?
I'm
> using
> > > > > >> attribute logging heavily as I try to better learn/debug
Nifi
> (it
> > > give
> > > > > >> you a nice 'under the hood' view of the flow) and build
up some
> > > flows
> > > > > >> and feel it would be beneficial to be able to capture the
> > > LogAttribte
> > > > > >> message by themselves for more clarity on what is happening.
I
> > would
> > > > > >> not mind maybe trying to implement this feature as my first
> crack
> > at
> > > > > >> contributing to the project. Seems like a fairly easy one
that
> > would
> > > > > >> allow me to "go through the motions" of a full pull request
> > process
> > > > > >> and iron out the process. Anyone have any thoughts on this?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message