nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject NiFi consumers concurrency
Date Fri, 18 Jan 2019 15:02:53 GMT
Hi all, Happy Friday!

I wonder if you have any ideas how to improve concurrency with NiFi Kafka
consumer processors.

We have 3 NiFi consumer processors in the flow, each listening to 250
topics. Each topic has 1 partition (it is critical for us to preserve
order). List of topic names is given as a comma-delimited list.

When we set concurrency to 6 for each consumer, hoping that NiFi will spawn
6 consumers per processor and they will be working concurrently, feeding
data from 6 topics at once. And since we have 3 consumer processors, it
should give us 18 concurrent feeds total.

[image: image.png]

Apparently, it does not work like that. NiFi does create 6 consumers per
processor (if I set concurrency to 6) but for some reason only one consumer
is reading from Kafka and one topic at the time while other 5 are sitting
and doing nothing. Because of that, total throughput is not really good.

We are looking at source code but I am hoping for some quick tips /
direction.

We could create 750 NiFi consumer processors instead but we do not really
like that idea.

Boris

Mime
View raw message