nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Processor to use
Date Thu, 05 Nov 2015 22:28:13 GMT
The main processors we have for interacting with JSON are EvaluateJsonPath
and SplitJson.

Regarding a lookup file you probably want to take a look at ScanAttribute
and ScanContent:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ScanAttribute/index.html
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ScanContent/index.html

You could potentially use ScanContent directly against the json, or you
could first use EvaluateJsonPath to extract a field of interest from the
json into a FlowFile attribute, and then use ScanAttribute to check that
attribute against the lookup values.

-Bryan

On Thu, Nov 5, 2015 at 5:13 PM, Madhire, Naveen <
Naveen.Madhire@capitalone.com> wrote:

> The data format is JSON coming from Kafka. Is there any processor for
> actually matching the contents of the flow files against a lookup file ?
>
>
> From: Bryan Bende <bbende@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Thursday, November 5, 2015 at 4:07 PM
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: Processor to use
>
> Hello,
>
> Can you share the data format coming from Kafka? json, avro, something
> else?
>
> The reason I ask is because there are many different processors that can
> modify FlowFiles in different ways, but a lot depends on the type of data
> involved.
>
> For general text-based operations we have...
> - ReplaceText to modify the content of the FlowFile
> - ExtractText to extract parts of the content into FlowFile attributes
> - UpdateAttribute to add/update a FlowFile attribute, can use expression
> language here to do dynamic stuff
>
> There are processors that interact with json, xml, and avro as well.
>
> If you have a need that is outside all of the existing processors, it is
> also possible to develop your own custom processor. Let us know and we can
> point you in the right direction.
>
> -Bryan
>
>
> On Thu, Nov 5, 2015 at 4:55 PM, Madhire, Naveen <
> Naveen.Madhire@capitalone.com> wrote:
>
>> Hi All,
>>
>> Just need few suggestions on which Processor to use. I have the below
>> need and I need to figure out which processor to use in the 2nd point as I
>> don’t see any processor to write customize logic.
>>
>>
>>    1. Read the data from Kafka Topic ( I will use the getKafka processor)
>>    2. *Apply business logic to the data fetched from Kafka like few
>>    calculations (Here I would need which processor to use to modify the flow
>>    files on the flow)*
>>    3. Write the data into HDFS. (putHDFS processor)
>>
>> Is there any processor which I can use to make the changes to the flow
>> files?
>>
>> Thanks.
>>
>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
View raw message