nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Meyer <charlie.me...@civitaslearning.com>
Subject Re: NiFi consumers concurrency
Date Fri, 18 Jan 2019 15:09:37 GMT
Hi Boris,

I have seen behavior similar to this before on other flows I have run. In
your Controller settings (in the hamburger menu at the top right of the
UI), have you adjusted the Maximum Timer Driven Thread Count?

On Fri, Jan 18, 2019 at 9:04 AM Boris Tyukin <boris@boristyukin.com> wrote:

> Hi all, Happy Friday!
>
> I wonder if you have any ideas how to improve concurrency with NiFi Kafka
> consumer processors.
>
> We have 3 NiFi consumer processors in the flow, each listening to 250
> topics. Each topic has 1 partition (it is critical for us to preserve
> order). List of topic names is given as a comma-delimited list.
>
> When we set concurrency to 6 for each consumer, hoping that NiFi will
> spawn 6 consumers per processor and they will be working concurrently,
> feeding data from 6 topics at once. And since we have 3 consumer
> processors, it should give us 18 concurrent feeds total.
>
> [image: image.png]
>
> Apparently, it does not work like that. NiFi does create 6 consumers per
> processor (if I set concurrency to 6) but for some reason only one consumer
> is reading from Kafka and one topic at the time while other 5 are sitting
> and doing nothing. Because of that, total throughput is not really good.
>
> We are looking at source code but I am hoping for some quick tips /
> direction.
>
> We could create 750 NiFi consumer processors instead but we do not really
> like that idea.
>
> Boris
>

Mime
View raw message