nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Petronic <markpetro...@gmail.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Mon, 02 Nov 2015 15:07:54 GMT
My primary use is for understanding Nifi. I like to direct various
processors output into both their logical next processor stage as well as
into a log attribute processor. Then I tail the Nifi app log file and watch
what happens - in real time. I do not intend to use this for long term log
retention. I agree that providence is the right choice for that. So, the
only reason I wanted to allow configuration of a custom logger was simply
to isolate all the attribute-rich logging from the normal logging because I
was primarily interested in the attribute flows as a way to (a) better
understand what a processor emits because, frankly, the documentation of
some of the processors is very sparse. So, I learn imperatively, so to
speak. I say that as a new user. I feel I should be able to get a pretty
good understanding of a processor by reading the usage. But I am finding
that the documentation, in some cases, is more like what I like to refer to
as, "note to self" documentation. Great if you are the guy who wrote the
processor with those "insights" - not so great if you are not the
developer. So, then I need to dig up the code. That should not be needed as
the first step of understanding a processor as a new user. There is some
well documented processors but not all are, IMHO. (b) Validate my flows
with some test data and verify attribute values look correct and routing is
happen on them as expected, etc. Again, easier, IMO, to see in the logs
than digging into the providence data.

Maybe this is just a good "private" feature for me so maybe I will just
create a private version to use on my own. I already have it working but
would need more polish to achieve PR status. Maybe this is the sort of
thing that others would not find beneficial? That's fine. There are others
ways I can contribute in the future. I'm still having fun! :)

On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <joe.witt@gmail.com> wrote:

> Mark Petronic,
>
> I share Payne's perspective on this.  But I'd also like to work with
> you to better understand the workflow.  For those of us that have used
> this tool for a long time there is a lot we take for granted from a
> new user perspective.  We believe the provenance feature to provide a
> far superior option to understanding how an item went through the
> system and the timing and what we knew when and so on.  But, it would
> be great to understand it from your perspective as someone learning
> NiFi.  Not meaning to take away from your proposed contrib - that
> would be great too.  Just want to see if the prov user experience
> solves what you're looking for and if not can we make it do that.
>
> Thanks
> Joe
>
> On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <markap14@hotmail.com> wrote:
> > Mark,
> >
> > To make sure that I understand what you're proposing, you want to add a
> property to
> > LogAttribute that allows users to provide a custom logger name?
> >
> > If that is indeed what you are suggesting then I think it's a great idea.
> >
> > That being said, in practice I rarely ever use LogAttribute and we even
> considered removing
> > it from the codebase before we open sourced, because the Data Provenance
> provides a
> > much better view of what's going on to debug your flows.
> >
> > I know you're pretty new to NiFi, so if you've not yet had a chance to
> play with the Provenance,
> > you can see the section in the User Guide at
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> <
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> >
> >
> > If you're interested in updating the LogAttribute processor, though,
> we'd be happy to have
> > that contribution added, as it does make the Processor more usable.
> >
> > Thanks
> > -Mark
> >
> >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <markpetronic@gmail.com>
> wrote:
> >>
> >> From the code, it appears it cannot be done as the attribute logging
> >> goes the same getLogger() instance as the normal nifi-app traces. Has
> >> anyone considered making that configurable, maybe allowing you do
> >> define a different logger name for LogAttribute then creating that
> >> logger definition in log back conf allowing flexibility? I'm using
> >> attribute logging heavily as I try to better learn/debug Nifi (it give
> >> you a nice 'under the hood' view of the flow) and build up some flows
> >> and feel it would be beneficial to be able to capture the LogAttribte
> >> message by themselves for more clarity on what is happening. I would
> >> not mind maybe trying to implement this feature as my first crack at
> >> contributing to the project. Seems like a fairly easy one that would
> >> allow me to "go through the motions" of a full pull request process
> >> and iron out the process. Anyone have any thoughts on this?
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message