nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Chaves <e...@uolet.com>
Subject Re: Someone recommend a good Avro studdy guide for newbie?
Date Thu, 23 Nov 2017 15:44:00 GMT
Hi Matt, thanks! as you said the AbstractRecordProcessor was the missing
bits to get me started.  There is one more question thou that I would like
to learn: how to create/manipulate schemas on the fly. Scouting some tests
I saw how to create a new basic schema but I would like to understand how
to create a new schema from a given pojo object, and how to add strcut
attributes in a existing schema.

By the way I'm testing and cleaning my invokedscriptprocessor used to
enhance data with multiples sql lookups, and I intent to share it here with
all. If you want you can share it on you blog too.

Cheers,

Eric

2017-11-22 19:28 GMT-02:00 Matt Burgess <mattyb149@apache.org>:

> Eric,
>
> If you're looking for examples on implementing a scripted record
> reader/writer, you can see the unit test examples [1] or Drew Lim's
> blog [2].  However I suspect you are looking to leverage an existing
> RecordReader/Writer from a scripted processor such as ExecuteScript or
> InvokeScriptedProcessor, to prototype a "record-aware" processor.  A
> general approach is implemented in AbstractRecordProcessor[3], I don't
> believe the NiFi Scripting NAR has access to that class, but you can
> certainly copy-and-paste the relevant bits of code.  I don't think you
> can (easily) get access to an existing RecordReader/Writer controller
> service, but Mark Payne outlined the alternative in your mailing list
> question about getting a DBCPService property. The recommended
> approach is to use InvokeScriptedProcessor and specify your own
> RecordReader/Writer properties which will be presented to the user in
> the InvokeScriptedProcessor dialog (once the script has been applied).
> Then you can get the RecordReader/Writer objects (see [3] lines
> 100-101) and use them in a similar way as [3].
>
> This (and the answer(s) to your questions in the other mailing list
> post) would make excellent blog posts/articles, I hope to do some
> relatively soon, will keep the list posted :)
>
> Regards,
> Matt
>
> [1] https://github.com/apache/nifi/tree/master/nifi-nar-
> bundles/nifi-scripting-bundle/nifi-scripting-processors/src/
> test/resources/groovy
> [2] https://community.hortonworks.com/articles/115311/convert-
> csv-to-json-avro-xml-using-convertrecord-p.html
> [3] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> main/java/org/apache/nifi/processors/standard/
> AbstractRecordProcessor.java#L94
>
>
> On Wed, Nov 22, 2017 at 3:00 PM, Eric Chaves <eric@uolet.com> wrote:
> > Hi folks,
> >
> > Due to my lack of knowledge with Avro in my scripted processors I'm
> required
> > to convert them to JSON in order to be able to manipulate them.
> >
> > I would like to avoid this and write record based processors but I have
> no
> > clue about how should I do to actually read/write/modify record based
> > content (including modifying underlying schemas).
> >
> > Do you have any reading material to recommend?
> >
> > Thanks in advance,
> >
> > Eric
>

Mime
View raw message