kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pulkit Manchanda <pulkit....@gmail.com>
Subject Re: Aw: Re: Doubts in Kafka
Date Fri, 11 Jan 2019 14:49:24 GMT
Aruna,
Instead of using 1000 partitions you can chose to create dynamic topics and
have a consumer for each topic.

On Fri, Jan 11, 2019 at 6:43 AM Peter Levart <peter.levart@gmail.com> wrote:

>
>
> On 1/10/19 2:26 PM, Sven Ludwig wrote:
> > Okay, but
> >
> > what if one also needs to preserve the order of messages coming from a
> particular device?
> >
> > With Kafka, this is perhaps possible if all messages from a particular
> device go into the same partition.
> >
> > Would it be a good and efficient solution for this approach to set the
> key of each Kafka ProducerRecord to the unique ID of the Device
>
> Exactly!
>
> > AND deactivate the key-based log-cleaner on the broker so that it does
> not delete older records that have the same key?
>
> How old records are cleaned is independent of what you do with processed
> records. You usually retain them for enough time so you don't loose them
> before processing them + some safety time...
>
> Regards, Peter
>
> >
> > Sven
> >
> >
> > Gesendet: Donnerstag, 10. Januar 2019 um 08:35 Uhr
> > Von: "Peter Levart" <peter.levart@gmail.com>
> > An: users@kafka.apache.org, "aruna ramachandran" <arunaeienec@gmail.com>
> > Betreff: Re: Doubts in Kafka
> > Hi Aruna,
> >
> > On 1/10/19 8:19 AM, aruna ramachandran wrote:
> >> I am using keyed partitions with 1000 partitions, so I need to create
> 1000
> >> consumers because consumers groups and re balancing concepts is not
> worked
> >> in the case of manually assigned consumers.Is there any replacement for
> the
> >> above problem.
> >>
> > What API are you using in the KafkaConsumer? Are you using
> > subscribe(Collection<String> topics) or are you using
> > assign(Collection<TopicPartition> partitions) ?
> >
> > The 1st one (subscribe) is the one you should be using for your usecase.
> > With that call, when you subscribe to a multi-partition topic and you
> > have multiple KafkaConsumer(s) configured with the same consumer group
> > id, then partitions of the topic are dynamically assigned (and possibly
> > reassigned when consumers come or go) to a set of live consumers. Will
> > this work for you (and why not)?
> >
> > Regards, Peter
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message