nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: In nifi usgae of Evaluate jsonpath processor will it affect performance impact because of attribute creation
Date Tue, 28 May 2019 17:41:02 GMT
Hi Vikram, 

I replied to your question on Stack Overflow. I’m pasting the same answer here for the list.


> Your concern about having too many attributes in memory should not be an issue here;
having 30 attributes per flowfile is higher than usual, but if these are all strings between
0 - ~100-200 characters, there should be minimal impact. If you start trying to extract KB
worth of data from the flowfile content to the attributes on each flowfile, you will see increased
heap usage, but the framework should still be able to handle this until you reach very high
throughput (1000's of flowfiles per second on commodity hardware like a modern laptop).
> 
> You may want to investigate ReplaceTextWithMapping [1], as that processor can load from
a definition file and handle many replace operations using a single processor.
> 
> It is usually a flow design "smell" to have multiple copies of the same flow process
with different configuration values (with the occasional exception of database interaction).
Rather, see if there is a way you can genericize the process and populate the relevant values
for each flowfile using variable population [2] (from the incoming flowfile attributes, the
variable registry [3], environment variables, etc.).

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ReplaceTextWithMapping/
[2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Using_Custom_Properties
[3] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window


Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 26, 2019, at 7:51 AM, vikramas <vikram.as123@gmail.com> wrote:
> 
> I'm trying to integrate nifi REST API's with my application. So by mapping
> input and output from my application, I am trying to call nifi REST api for
> flow creation. So, in my use case most of the times I will extract the JSON
> values and will apply expression languages.
> 
> So, for simplifying all the use-cases I am using evaluate JSONpath processor
> for fetching all attributes using jsonpath and apply expression language
> function on that in extract processor. Below is the flow diagram regarding
> that.
> 
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/t1186/jsonpath_attributes.png>

> 
> Is it the right approach because for JSON to JSON manipulation having 30
> keys this is the simplest way, and as I am trying to integrate nifi REST
> API's with my application I cannot generate JOLT transformation logic
> dynamically based on the user mapping.
> 
> So, in this case, does the usage of evaluating JSONpath processor creates
> any performance issues for about 50 use case with different transformation
> logic because as I saw in documentation attribute usage creates
> performance(regarding memory) issues.
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message