kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From BYEONG-GI KIM <bg...@bluedigm.com>
Subject Scalability of Kafka Consumer
Date Wed, 01 Jun 2016 00:56:59 GMT

I've implemented a Kafka Consumer Application which consume large number of
monitoring data from Kafka Broker and analyze those data accordingly.

I referred to a guide,
since I thought the app needs to implement multi-threading for Kafka
Consumer per Topic. Actually, A topic is assigned to each open-source
monitoring software, e.g., Nagios, Collectd, etc., in order to distinguish
those because each of these uses its own message format, such as JSON,
String, and so on.

There was, however, an Exception even though my source code for the Kafka
Consumer are mostly copied and pasted from the guide;
KafkaConsumer is not safe for multi-threaded access*

First Question. Could the implementation in the guide really prevent the

And second Question is, could the KafkaConsumer support such huge amount of
data with one thread? The KafkaConsumer seems not thread-safe, and it can
subscribe multi-topics at once. Do I need to change the implementation from
the multi-threaded to one-thread and subscribing multi-topics?... I'm just
wonder whether a KafkaConsumer is able to stand the bunch of data without
performance degradation.

Thanks in advance!

Best regards


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message