kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Baraff <...@pixar.com>
Subject multiple consumers on one partition using assign? (not subscribe)?
Date Wed, 17 Oct 2018 15:03:50 GMT
I’m using Kafka for what I think is a somewhat non-standard purpose.  We have multiple producers
which send messages to a topic.  We have say as many as 500 or 1000 consumers which each want
to read every message posted to the topic, but *only* from the point in time that the consumer
came alive and started listening.  (We’re using Kafka to replace a UDB based message system.)

Given the above, I set things up as:

	+ topic has only 1 partition
	+ retention time is very short (10 seconds)
	+ every consumer uses the same group.id <http://group.id/>
	+ consumers DO NOT commit offsets ever
	+ new consumers set the offset to the “latest available” when they start
	+ consumers don’t subscribe: they assign themselves to partition 0 of the topic

This works fine: each consumer appears to get a copy of all the (new) messages that come in,
after they start consuming.

Question: Is using the same group.id <http://group.id/> for every consumer likely to
wreck anything?  I could generate a random group.id <http://group.id/> so that it appears
that every consumer is distinct, but given that each consumer does an assign to partition
0, and never commits offsets, is what I’m doing sound?

I’m slightly worried I’m misusing Kafka somehow: will this likely continue working in
the future?
As I said, I’m slightly worried about reusing the same group.id <http://group.id/>
each time.  But it seems bad to generate like 1,000 different consumer records on the broker
if there is no need…

Also, I want to make sure I don’t bog the brokers up with needless logfiles for messages
that nobody will see.  (Once a message gets to be 10 seconds old, nobody should see it, because
if you weren’t listening when it was produced, you don’t care about.)

Thanks for any insights/opinions.

	David Baraff
	deb@pixar.com <mailto:deb@pixar.com>



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message