kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryanne Dolan <ryannedo...@gmail.com>
Subject Re: multiple consumers on one partition using assign? (not subscribe)?
Date Wed, 17 Oct 2018 15:08:01 GMT
David,

If the consumers never commit() or subscribe(), I don't think the group.id
is relevant.

Ryanne

On Wed, Oct 17, 2018 at 10:04 AM David Baraff <deb@pixar.com> wrote:

> I’m using Kafka for what I think is a somewhat non-standard purpose.  We
> have multiple producers which send messages to a topic.  We have say as
> many as 500 or 1000 consumers which each want to read every message posted
> to the topic, but *only* from the point in time that the consumer came
> alive and started listening.  (We’re using Kafka to replace a UDB based
> message system.)
>
> Given the above, I set things up as:
>
>         + topic has only 1 partition
>         + retention time is very short (10 seconds)
>         + every consumer uses the same group.id <http://group.id/>
>         + consumers DO NOT commit offsets ever
>         + new consumers set the offset to the “latest available” when they
> start
>         + consumers don’t subscribe: they assign themselves to partition 0
> of the topic
>
> This works fine: each consumer appears to get a copy of all the (new)
> messages that come in, after they start consuming.
>
> Question: Is using the same group.id <http://group.id/> for every
> consumer likely to wreck anything?  I could generate a random group.id <
> http://group.id/> so that it appears that every consumer is distinct, but
> given that each consumer does an assign to partition 0, and never commits
> offsets, is what I’m doing sound?
>
> I’m slightly worried I’m misusing Kafka somehow: will this likely continue
> working in the future?
> As I said, I’m slightly worried about reusing the same group.id <
> http://group.id/> each time.  But it seems bad to generate like 1,000
> different consumer records on the broker if there is no need…
>
> Also, I want to make sure I don’t bog the brokers up with needless
> logfiles for messages that nobody will see.  (Once a message gets to be 10
> seconds old, nobody should see it, because if you weren’t listening when it
> was produced, you don’t care about.)
>
> Thanks for any insights/opinions.
>
>         David Baraff
>         deb@pixar.com <mailto:deb@pixar.com>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message