nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DAVID SMITH <davidrsm...@btinternet.com>
Subject Re: LogAttribute - Sending that output to a custom logger?
Date Wed, 11 Nov 2015 21:41:13 GMT
Adam
I have uploaded to github the bundle as I described earlier, for evaluation and possible future
inclusion into the NiFi codebase, I would appreciate user feedback.
The github link is :    helicopterman22/nifi-whatsflowing-bundle
|   |
|   |  |   |   |   |   |   |
| helicopterman22/nifi-whatsflowing-bundlenifi-whatsflowing-bundle - A NiFi bundle that contains
a processor which uses a regex to define which FlowFile attributes are written to a user defined
file |
|  |
| View on github.com | Preview by Yahoo |
|  |
|   |




Dave 


     On Tuesday, 3 November 2015, 2:09, Joe Witt <joe.witt@gmail.com> wrote:
   

 https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide

On Mon, Nov 2, 2015 at 9:04 PM, Adam Taft <adam@adamtaft.com> wrote:
> David,
>
> This sounds like a slightly different use case than the NiFi standard
> LogAttribute processor.  It sounds like your processor is more of a generic
> attribute converter and file writer.  The LogAttribute processor is
> designed to interact with the underlying NiFi logging subsystem, not
> necessarily just to write files.
>
> That being said, your processor may be a useful contribution to Apache
> NiFi.  Specifically, the value-add of your processor might be in the
> key-value format you've defined to output the flowfile attributes.  It
> might be interesting to see this expressed as an attribute-to-payload
> converter, chained together with potentially other processors like PutFile
> in the dataflow.
>
> If you want to contribute your processor, I would recommend making it
> available on GitHub (or similar) for review by the Apache NiFi community.
> Just post a link of your contribution here or even issue a pull request for
> your processor.  It would at least be evaluated and considered for
> inclusion.
>
> Hope this helps.
>
> Adam
>
>
> On Mon, Nov 2, 2015 at 5:39 PM, davidrsmith@btinternet.com <
> davidrsmith@btinternet.com> wrote:
>
>> Hi
>>
>> Where I work we have created an attribute loggers of our own. It is a
>> fairly simple affair which used a regex to determine which attributes to
>> log, and writes them as key value pairs to a file, whose location is
>> determined by a user properly. I'm happy to put this out there if anyone is
>> interested.
>>
>> Sent from my HTC
>>
>>
>> ----- Reply message -----
>> From: "Adam Taft" <adam@adamtaft.com>
>> Date: Mon, Nov 2, 2015 19:23
>> Subject: LogAttribute - Sending that output to a custom logger?
>> To: <dev@nifi.apache.org>
>>
>> This thread has forked into two different conversations:  1. improvements
>> to LogAttribute processor; 2. improvements to processor documentation.
>>
>> 1)  re: improvements to LogAttribute - we already have NIFI-67 [1] that
>> suggests a number of improvements to LogAttribute.  One of these is the use
>> of a custom name for the logger so that logback rules can be written
>> against that name.
>>
>> While the provenance engine is great for many scenarios, in my opinion, it
>> doesn't replace the need for true text-based logging.  The tooling for log
>> processing is very mature and there's no ability to "grep" a provenance
>> repository, migrate or offload provenance logs into deep storage, store log
>> events into a database, or do any other cool syslogd or logback type
>> things.  Being able to capture and log a flowfile at the exact right place
>> in the data flow and processing it using the command line is an extremely
>> valuable tool in the toolkit.
>>
>> For a long time, I've wanted to work on at least some of the things
>> mentioned in NIFI-67 and will hopefully get to do so time willing.  Having
>> a custom "name" for the LogAttribute processor seems like a no-brainer.
>> Contributions for this should definitely be welcome!
>>
>> 2) improvements to processor document - I agree, even as a somewhat
>> seasoned NIFI user, I still have a hard time reading and understanding the
>> processor documentation.  I often do exactly what Mark P. suggests and
>> instead go directly to the source.  Any contribution towards better
>> processor documentation is greatly appreciated!
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-67
>>
>>
>> On Mon, Nov 2, 2015 at 1:54 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
>>
>> > We greatly appreciate contributions.  Your prescribed approach sounds
>> great
>> > and if you are willing to give us a few cycles pointing out, and
>> optionally
>> > correcting, the items that are in need of improvement, we will certainly
>> > incorporate.
>> >
>> > Thanks!
>> >
>> > On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <markpetronic@gmail.com>
>> > wrote:
>> >
>> > > I'm sort of in the camp of "don't come with a complaint if you don't
>> come
>> > > with a solution" and hesitated to even raise the documentation comment
>> > > without just fixing it myself. How about this, I just do some updates
>> on
>> > > some processor docs myself and use that as my first contribution to
>> work
>> > > through the process of committing to this project?
>> > >
>> > > But, to give you one quick example, EvaluateJSONPath (which, btw has
>> > pretty
>> > > good docs otherwise) does not mention HOW to extract the JSON you are
>> > > interested in. I had to look at the code to figure out it used this:
>> > > https://github.com/jayway/JsonPath. Ok, that was not hard, I admit,
>> but,
>> > > as
>> > > a user, should I need to look at the code for such information? I
>> submit,
>> > > no. Me personally, I like to dig into the code. So, this is more a
>> > comment
>> > > on "overall goodness" for the general new user experience.
>> > >
>> > > I agree with your assessment of 'new user vibe' as I am starting to not
>> > > notice it as much. lol
>> > >
>> > > On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <joe.witt@gmail.com> wrote:
>>
>>
>>


  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message