nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Petronic <markpetro...@gmail.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Mon, 02 Nov 2015 18:28:35 GMT
I'm sort of in the camp of "don't come with a complaint if you don't come
with a solution" and hesitated to even raise the documentation comment
without just fixing it myself. How about this, I just do some updates on
some processor docs myself and use that as my first contribution to work
through the process of committing to this project?

But, to give you one quick example, EvaluateJSONPath (which, btw has pretty
good docs otherwise) does not mention HOW to extract the JSON you are
interested in. I had to look at the code to figure out it used this:
https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but, as
a user, should I need to look at the code for such information? I submit,
no. Me personally, I like to dig into the code. So, this is more a comment
on "overall goodness" for the general new user experience.

I agree with your assessment of 'new user vibe' as I am starting to not
notice it as much. lol

On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <joe.witt@gmail.com> wrote:

> Mark
>
> All fair points.  Can you please point out which processor docs
> specifically should be better.  Let's fix em..you will quickly lose that
> new user vibe and not notice what needs to improve as much.  We need to
> make the new user experience awesome.
>
> Thanks
> Joe
> On Nov 2, 2015 10:08 AM, "Mark Petronic" <markpetronic@gmail.com> wrote:
>
> > My primary use is for understanding Nifi. I like to direct various
> > processors output into both their logical next processor stage as well as
> > into a log attribute processor. Then I tail the Nifi app log file and
> watch
> > what happens - in real time. I do not intend to use this for long term
> log
> > retention. I agree that providence is the right choice for that. So, the
> > only reason I wanted to allow configuration of a custom logger was simply
> > to isolate all the attribute-rich logging from the normal logging
> because I
> > was primarily interested in the attribute flows as a way to (a) better
> > understand what a processor emits because, frankly, the documentation of
> > some of the processors is very sparse. So, I learn imperatively, so to
> > speak. I say that as a new user. I feel I should be able to get a pretty
> > good understanding of a processor by reading the usage. But I am finding
> > that the documentation, in some cases, is more like what I like to refer
> to
> > as, "note to self" documentation. Great if you are the guy who wrote the
> > processor with those "insights" - not so great if you are not the
> > developer. So, then I need to dig up the code. That should not be needed
> as
> > the first step of understanding a processor as a new user. There is some
> > well documented processors but not all are, IMHO. (b) Validate my flows
> > with some test data and verify attribute values look correct and routing
> is
> > happen on them as expected, etc. Again, easier, IMO, to see in the logs
> > than digging into the providence data.
> >
> > Maybe this is just a good "private" feature for me so maybe I will just
> > create a private version to use on my own. I already have it working but
> > would need more polish to achieve PR status. Maybe this is the sort of
> > thing that others would not find beneficial? That's fine. There are
> others
> > ways I can contribute in the future. I'm still having fun! :)
> >
> > On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Mark Petronic,
> > >
> > > I share Payne's perspective on this.  But I'd also like to work with
> > > you to better understand the workflow.  For those of us that have used
> > > this tool for a long time there is a lot we take for granted from a
> > > new user perspective.  We believe the provenance feature to provide a
> > > far superior option to understanding how an item went through the
> > > system and the timing and what we knew when and so on.  But, it would
> > > be great to understand it from your perspective as someone learning
> > > NiFi.  Not meaning to take away from your proposed contrib - that
> > > would be great too.  Just want to see if the prov user experience
> > > solves what you're looking for and if not can we make it do that.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <markap14@hotmail.com>
> > wrote:
> > > > Mark,
> > > >
> > > > To make sure that I understand what you're proposing, you want to
> add a
> > > property to
> > > > LogAttribute that allows users to provide a custom logger name?
> > > >
> > > > If that is indeed what you are suggesting then I think it's a great
> > idea.
> > > >
> > > > That being said, in practice I rarely ever use LogAttribute and we
> even
> > > considered removing
> > > > it from the codebase before we open sourced, because the Data
> > Provenance
> > > provides a
> > > > much better view of what's going on to debug your flows.
> > > >
> > > > I know you're pretty new to NiFi, so if you've not yet had a chance
> to
> > > play with the Provenance,
> > > > you can see the section in the User Guide at
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > <
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > >
> > > >
> > > > If you're interested in updating the LogAttribute processor, though,
> > > we'd be happy to have
> > > > that contribution added, as it does make the Processor more usable.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <markpetronic@gmail.com
> >
> > > wrote:
> > > >>
> > > >> From the code, it appears it cannot be done as the attribute logging
> > > >> goes the same getLogger() instance as the normal nifi-app traces.
> Has
> > > >> anyone considered making that configurable, maybe allowing you do
> > > >> define a different logger name for LogAttribute then creating that
> > > >> logger definition in log back conf allowing flexibility? I'm using
> > > >> attribute logging heavily as I try to better learn/debug Nifi (it
> give
> > > >> you a nice 'under the hood' view of the flow) and build up some
> flows
> > > >> and feel it would be beneficial to be able to capture the
> LogAttribte
> > > >> message by themselves for more clarity on what is happening. I would
> > > >> not mind maybe trying to implement this feature as my first crack
at
> > > >> contributing to the project. Seems like a fairly easy one that would
> > > >> allow me to "go through the motions" of a full pull request process
> > > >> and iron out the process. Anyone have any thoughts on this?
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message