kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Garcia <dav...@spiceworks.com>
Subject Re: Identifying Number of Kafka Consumers
Date Tue, 12 Sep 2017 14:53:35 GMT
Consumers can be split up based on partitions.  So, you can tell a consumer group to listen
to several topics and it will divvy up the work.  Your use case sounds very canonical.  I
would take a look at Kafka connect (if you’re using the confluent stack).

-Daivd

http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html

On 9/11/17, 4:48 PM, "Nishanth S" <nishanth.2884@gmail.com> wrote:

    All,
    I am very new to kafka . We have a  case where we need to ingest multiple
    avro record types . These avro record types vary vastly in volume  and size
    and I am thinking of sending  each of these message types to a different
    topic and creating partitions based on volume and through put needed.  What
    the  kafka consumer has to do is read the record of from  partitions and
     write to  different hdfs locations based on record type  . I am guessing
    we should at least start with one consumer per  topic  . Is this
    understanding correct or is there a better way to look at it?
    

Mime
View raw message