nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Creating a custom processor for JSON documents
Date Wed, 04 Feb 2015 22:51:07 GMT
Not sure if this is helpful, but I recently attempted writing a processor
to rename and/or exclude fields from JSON documents. My thought being that
one system is producing documents that need to be slightly modified before
sending to another system.

https://github.com/bbende/nifi-example-bundles/blob/master/nifi-example-utils-bundle/nifi-example-utils-processors/src/main/java/org/apache/nifi/processors/example/utils/RenameJSONFields.java

I'll put a big disclaimer on this and say that it hasn't really been tested
other than the unit tests, but it might be helpful in getting you started
writing your own EvaluateJsonPath processor.

On Wed, Feb 4, 2015 at 5:25 PM, István <leccine@gmail.com> wrote:

> Mark,
>
> This is exactly the feedback I was looking for. Thank you very much! I am
> trying to play with it for a while and get back if I have any questions.
>
> Regards,
> Istvan
>
>
> On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <markap14@hotmail.com> wrote:
>
> > Istvan,
> > I think this is a great use case for NiFi! The TransformXml processor
> > could be applicable, but I would actually guide you more toward the
> > EvaluateXPath. Specifically, I would break the problem into two
> processors:
> > EvaluateJsonPath and RouteOnAttribute.
> > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > expressions and put the results into user-defined attributes.Then, you
> can
> > route and filter on those attributes via the RouteOnAttribute processor.
> > I have actually considered doing something like this a few times, but I
> > know there are a few different "JsonPath" types of languages/libs out
> there
> > and wasn't sure which library made most sense to use.  I know there is
> one
> > at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I
> > would want to look a bit closer to make sure that all of its dependencies
> > are also Apache-compatible, but that may be a good starting point.
> > Then, the user would configure this processor by adding their own
> > properties. For instance, a property named "orderId" could have value
> > "/order/orderId". That would create an attribute on the FlowFile named
> > "orderId" whose value would be the result of evaluating the
> > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > attribute.
> > Does this make sense at all?
> > I would be happy to go into more detail or explain further with better
> > examples if it doesn't make sense.
> > Thanks-Mark
> >
> > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > Subject: Creating a custom processor for JSON documents
> > > From: leccine@gmail.com
> > > To: dev@nifi.incubator.apache.org
> > >
> > > Hi,
> > >
> > > I was wondering what would be the best way of doing this. The basic
> > > scenario is that we have a Kafka cluster that has lots of messages
> (JSON
> > > documents). I would like to go though all of those messages and parse
> > each
> > > individual message to extract fields from it, like if you are thinking
> > > about the HashMap representation of a JSON document I need a value of a
> > > certain key. I need some filtering capabilities as well, only process
> > > messages that has a certain key and the value of that key is
> configurable
> > > in the UI.
> > >
> > > I have seen that there are standard processors here:
> > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > >
> > > Which one should I use as a kind of example to do this? Am I on the
> right
> > > track that this is achievable with Nifi and I am not trying to do
> > something
> > > silly?
> > >
> > > It seems that TransformXml.java would be a good start, using it as an
> > > example to implement something like this.
> > >
> > > Thank you in advance,
> > > Istvan
> > >
> > > --
> > > the sun shines for all
> >
> >
>
>
>
> --
> the sun shines for all
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message