nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From István <lecc...@gmail.com>
Subject Re: Creating a custom processor for JSON documents
Date Wed, 04 Feb 2015 22:53:37 GMT
Thank you Bryan,

Much appreciated!

I.

On Wed, Feb 4, 2015 at 2:51 PM, Bryan Bende <bbende@gmail.com> wrote:

> Not sure if this is helpful, but I recently attempted writing a processor
> to rename and/or exclude fields from JSON documents. My thought being that
> one system is producing documents that need to be slightly modified before
> sending to another system.
>
>
> https://github.com/bbende/nifi-example-bundles/blob/master/nifi-example-utils-bundle/nifi-example-utils-processors/src/main/java/org/apache/nifi/processors/example/utils/RenameJSONFields.java
>
> I'll put a big disclaimer on this and say that it hasn't really been tested
> other than the unit tests, but it might be helpful in getting you started
> writing your own EvaluateJsonPath processor.
>
> On Wed, Feb 4, 2015 at 5:25 PM, István <leccine@gmail.com> wrote:
>
> > Mark,
> >
> > This is exactly the feedback I was looking for. Thank you very much! I am
> > trying to play with it for a while and get back if I have any questions.
> >
> > Regards,
> > Istvan
> >
> >
> > On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <markap14@hotmail.com> wrote:
> >
> > > Istvan,
> > > I think this is a great use case for NiFi! The TransformXml processor
> > > could be applicable, but I would actually guide you more toward the
> > > EvaluateXPath. Specifically, I would break the problem into two
> > processors:
> > > EvaluateJsonPath and RouteOnAttribute.
> > > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > > expressions and put the results into user-defined attributes.Then, you
> > can
> > > route and filter on those attributes via the RouteOnAttribute
> processor.
> > > I have actually considered doing something like this a few times, but I
> > > know there are a few different "JsonPath" types of languages/libs out
> > there
> > > and wasn't sure which library made most sense to use.  I know there is
> > one
> > > at https://github.com/jayway/JsonPath that has an Apache 2.0 License.
> I
> > > would want to look a bit closer to make sure that all of its
> dependencies
> > > are also Apache-compatible, but that may be a good starting point.
> > > Then, the user would configure this processor by adding their own
> > > properties. For instance, a property named "orderId" could have value
> > > "/order/orderId". That would create an attribute on the FlowFile named
> > > "orderId" whose value would be the result of evaluating the
> > > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > > attribute.
> > > Does this make sense at all?
> > > I would be happy to go into more detail or explain further with better
> > > examples if it doesn't make sense.
> > > Thanks-Mark
> > >
> > > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > > Subject: Creating a custom processor for JSON documents
> > > > From: leccine@gmail.com
> > > > To: dev@nifi.incubator.apache.org
> > > >
> > > > Hi,
> > > >
> > > > I was wondering what would be the best way of doing this. The basic
> > > > scenario is that we have a Kafka cluster that has lots of messages
> > (JSON
> > > > documents). I would like to go though all of those messages and parse
> > > each
> > > > individual message to extract fields from it, like if you are
> thinking
> > > > about the HashMap representation of a JSON document I need a value
> of a
> > > > certain key. I need some filtering capabilities as well, only process
> > > > messages that has a certain key and the value of that key is
> > configurable
> > > > in the UI.
> > > >
> > > > I have seen that there are standard processors here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > > >
> > > > Which one should I use as a kind of example to do this? Am I on the
> > right
> > > > track that this is achievable with Nifi and I am not trying to do
> > > something
> > > > silly?
> > > >
> > > > It seems that TransformXml.java would be a good start, using it as an
> > > > example to implement something like this.
> > > >
> > > > Thank you in advance,
> > > > Istvan
> > > >
> > > > --
> > > > the sun shines for all
> > >
> > >
> >
> >
> >
> > --
> > the sun shines for all
> >
>



-- 
the sun shines for all

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message