nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Mon, 02 Nov 2015 18:54:28 GMT
We greatly appreciate contributions.  Your prescribed approach sounds great
and if you are willing to give us a few cycles pointing out, and optionally
correcting, the items that are in need of improvement, we will certainly
incorporate.

Thanks!

On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <markpetronic@gmail.com>
wrote:

> I'm sort of in the camp of "don't come with a complaint if you don't come
> with a solution" and hesitated to even raise the documentation comment
> without just fixing it myself. How about this, I just do some updates on
> some processor docs myself and use that as my first contribution to work
> through the process of committing to this project?
>
> But, to give you one quick example, EvaluateJSONPath (which, btw has pretty
> good docs otherwise) does not mention HOW to extract the JSON you are
> interested in. I had to look at the code to figure out it used this:
> https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but,
> as
> a user, should I need to look at the code for such information? I submit,
> no. Me personally, I like to dig into the code. So, this is more a comment
> on "overall goodness" for the general new user experience.
>
> I agree with your assessment of 'new user vibe' as I am starting to not
> notice it as much. lol
>
> On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <joe.witt@gmail.com> wrote:
>
> > Mark
> >
> > All fair points.  Can you please point out which processor docs
> > specifically should be better.  Let's fix em..you will quickly lose that
> > new user vibe and not notice what needs to improve as much.  We need to
> > make the new user experience awesome.
> >
> > Thanks
> > Joe
> > On Nov 2, 2015 10:08 AM, "Mark Petronic" <markpetronic@gmail.com> wrote:
> >
> > > My primary use is for understanding Nifi. I like to direct various
> > > processors output into both their logical next processor stage as well
> as
> > > into a log attribute processor. Then I tail the Nifi app log file and
> > watch
> > > what happens - in real time. I do not intend to use this for long term
> > log
> > > retention. I agree that providence is the right choice for that. So,
> the
> > > only reason I wanted to allow configuration of a custom logger was
> simply
> > > to isolate all the attribute-rich logging from the normal logging
> > because I
> > > was primarily interested in the attribute flows as a way to (a) better
> > > understand what a processor emits because, frankly, the documentation
> of
> > > some of the processors is very sparse. So, I learn imperatively, so to
> > > speak. I say that as a new user. I feel I should be able to get a
> pretty
> > > good understanding of a processor by reading the usage. But I am
> finding
> > > that the documentation, in some cases, is more like what I like to
> refer
> > to
> > > as, "note to self" documentation. Great if you are the guy who wrote
> the
> > > processor with those "insights" - not so great if you are not the
> > > developer. So, then I need to dig up the code. That should not be
> needed
> > as
> > > the first step of understanding a processor as a new user. There is
> some
> > > well documented processors but not all are, IMHO. (b) Validate my flows
> > > with some test data and verify attribute values look correct and
> routing
> > is
> > > happen on them as expected, etc. Again, easier, IMO, to see in the logs
> > > than digging into the providence data.
> > >
> > > Maybe this is just a good "private" feature for me so maybe I will just
> > > create a private version to use on my own. I already have it working
> but
> > > would need more polish to achieve PR status. Maybe this is the sort of
> > > thing that others would not find beneficial? That's fine. There are
> > others
> > > ways I can contribute in the future. I'm still having fun! :)
> > >
> > > On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <joe.witt@gmail.com> wrote:
> > >
> > > > Mark Petronic,
> > > >
> > > > I share Payne's perspective on this.  But I'd also like to work with
> > > > you to better understand the workflow.  For those of us that have
> used
> > > > this tool for a long time there is a lot we take for granted from a
> > > > new user perspective.  We believe the provenance feature to provide a
> > > > far superior option to understanding how an item went through the
> > > > system and the timing and what we knew when and so on.  But, it would
> > > > be great to understand it from your perspective as someone learning
> > > > NiFi.  Not meaning to take away from your proposed contrib - that
> > > > would be great too.  Just want to see if the prov user experience
> > > > solves what you're looking for and if not can we make it do that.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <markap14@hotmail.com>
> > > wrote:
> > > > > Mark,
> > > > >
> > > > > To make sure that I understand what you're proposing, you want to
> > add a
> > > > property to
> > > > > LogAttribute that allows users to provide a custom logger name?
> > > > >
> > > > > If that is indeed what you are suggesting then I think it's a great
> > > idea.
> > > > >
> > > > > That being said, in practice I rarely ever use LogAttribute and we
> > even
> > > > considered removing
> > > > > it from the codebase before we open sourced, because the Data
> > > Provenance
> > > > provides a
> > > > > much better view of what's going on to debug your flows.
> > > > >
> > > > > I know you're pretty new to NiFi, so if you've not yet had a chance
> > to
> > > > play with the Provenance,
> > > > > you can see the section in the User Guide at
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > <
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > >
> > > > >
> > > > > If you're interested in updating the LogAttribute processor,
> though,
> > > > we'd be happy to have
> > > > > that contribution added, as it does make the Processor more usable.
> > > > >
> > > > > Thanks
> > > > > -Mark
> > > > >
> > > > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <
> markpetronic@gmail.com
> > >
> > > > wrote:
> > > > >>
> > > > >> From the code, it appears it cannot be done as the attribute
> logging
> > > > >> goes the same getLogger() instance as the normal nifi-app traces.
> > Has
> > > > >> anyone considered making that configurable, maybe allowing you
do
> > > > >> define a different logger name for LogAttribute then creating
that
> > > > >> logger definition in log back conf allowing flexibility? I'm
using
> > > > >> attribute logging heavily as I try to better learn/debug Nifi
(it
> > give
> > > > >> you a nice 'under the hood' view of the flow) and build up some
> > flows
> > > > >> and feel it would be beneficial to be able to capture the
> > LogAttribte
> > > > >> message by themselves for more clarity on what is happening.
I
> would
> > > > >> not mind maybe trying to implement this feature as my first crack
> at
> > > > >> contributing to the project. Seems like a fairly easy one that
> would
> > > > >> allow me to "go through the motions" of a full pull request
> process
> > > > >> and iron out the process. Anyone have any thoughts on this?
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message