kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Klijs <gerard.kl...@dizzit.com>
Subject Re: Scalability of Kafka Consumer
Date Wed, 01 Jun 2016 07:11:28 GMT
If I understand it correctly each consumer should have it's 'own' thread,
and should not be accessible from other threads. But you could
(dynamically) create enough threads to cover all the partitions, so each
consumer only reads from one partition. You could also let all those
consumers access some threadsafe object if you need to combine the result.
In your linked example the consumers just each do there part, with solves
the multi-threaded issue, but when you want to combine data from different
consumer threads it becomes more tricky.

On Wed, Jun 1, 2016 at 2:57 AM BYEONG-GI KIM <bgkim@bluedigm.com> wrote:

> Hello.
> I've implemented a Kafka Consumer Application which consume large number of
> monitoring data from Kafka Broker and analyze those data accordingly.
> I referred to a guide,
> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
> ,
> since I thought the app needs to implement multi-threading for Kafka
> Consumer per Topic. Actually, A topic is assigned to each open-source
> monitoring software, e.g., Nagios, Collectd, etc., in order to distinguish
> those because each of these uses its own message format, such as JSON,
> String, and so on.
> There was, however, an Exception even though my source code for the Kafka
> Consumer are mostly copied and pasted from the guide;
> *java.util.ConcurrentModificationException:
> KafkaConsumer is not safe for multi-threaded access*
> First Question. Could the implementation in the guide really prevent the
> Exception?
> And second Question is, could the KafkaConsumer support such huge amount of
> data with one thread? The KafkaConsumer seems not thread-safe, and it can
> subscribe multi-topics at once. Do I need to change the implementation from
> the multi-threaded to one-thread and subscribing multi-topics?... I'm just
> wonder whether a KafkaConsumer is able to stand the bunch of data without
> performance degradation.
> Thanks in advance!
> Best regards

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message