nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: MergeRecord performance
Date Fri, 24 Apr 2020 20:41:31 GMT
Robert,

Can you please detail the record readers and writers involved and how
schemas are accessed?  There can be very important performance related
changes in the parsers/serializers of the given formats.  And we've added a
lot to make schema caching really capable but you have to opt into it.  It
is of course possible MergeRecord itself is the culprit for performance
reduction but lets get a more full picture here.

Are you able to share a template and sample data which we can use to
replicate?

Thanks

On Fri, Apr 24, 2020 at 4:38 PM Robert R. Bruno <rbruno8@gmail.com> wrote:

> I wanted to see if anyone else has experienced performance issues with the
> newest version of nifi and MergeRecord?  We have been running on nifi 1.9.2
> for awhile now, and recently upgraded to nifi 1.11.4.  Once upgraded, our
> identical flows were no longer able to keep up with our data mainly at
> MergeRecord processors.
>
> We ended up downgrading back to nifi 1.9.2.  Once we downgraded, all was
> keeping up again.  There were no errors to speak of when we were running
> the flow with 1.11.4.  We did see higher load on the OS, but this may have
> been caused by the fact there was such a tremendous backlog built up in the
> flow.
>
> Another side note, we saw one UpdateRecord processor producing errors when
> I tested the flow with nifi 1.11.4 with a small test flow.  I was able to
> fix this issue by changing some parameters in my RecordWriter.  So perhaps
> some underlying ways records are being handled since 1.9.2 caused the
> performance issue we saw?
>
> Any insight anyone has would be greatly appreciated, as we very much would
> like to upgrade to nifi 1.11.4.  One thought was switching the MergeRecord
> processors to MergeContent since I've been told MergeContent seems to
> perform better, but not sure if this is actually true.  We are using the
> pattern of chaining a few MergeRecord processors together to help with
> performance.
>
> Thanks in advance!
>

Mime
View raw message