kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hampson <ch...@hampson.dev>
Subject Re: Question regarding pattern subscriptions
Date Mon, 22 Jul 2019 09:19:30 GMT
Thanks Alok,

I'd come across that and begun playing with different values for that. Tweaking it has done
exactly what I was after. I'm now getting the desired behaviour.

Thanks again for the speedy response,


> On 22 Jul 2019, at 09:36, Alok Dwivedi <alok.dwivedi@instaclustr.com> wrote:
> Hi Chris
> Since you are only missing the very first message sent to that topic, I wonder its related
to " auto.offset.reset" consumer setting. This setting controls what happens when no offset
is present for a topic (most commonly the case when there has been no consumer yet for that
> From documentation:
> "Use auto.offset.reset to define the behaviour of the consumer when there is no committed
position (which would be the case when the group is first initialized) or when an offset is
out of range. You can choose either to reset the position to the "earliest" offset or the
"latest" offset (the default). You can also select "none" if you would rather set the initial
offset yourself and you are willing to handle out of range errors manually."
> With default behaviour of "latest" there is a possibility that consumer is not yet ready
to listen to messages by the time 1st message was sent but then its ready and starts consuming
from 2nd (latest since its stared listening). You can prove or disprove this theory of mine
by changing default to "earliest" and see if you always get messages from beginning once default
is overridden (or not). 
> Thanks 
> Alok Dwivedi
> Senior Consultant
> https://www.instaclustr.com/platform/
> ´╗┐On 19/07/2019, 14:01, "Chris Hampson" <chris@hampson.dev> wrote:
>    Hi all,
>    Tl;dr: I want to set up a subscription based on patterns that will match non-existent
topics and receive all messages that match (e.g. test.*), missing first message to any new
matching topic, set auto.create.topics.enable=true and allow.auto.create.topics=true and very
low metadata.max.age.ms
>    I've been playing around with Kafka trying to build a PoC for a cluster event notification
system. I can't /quite/ get it doing what I'm after (at the moment). No doubt this is me missing
some little nugget in the docs, or mis-understanding something. I've also searched around,
but perhaps am using the wrong terms.
>    I'd like to set up a consumer with a topic pattern along the lines of "test.*" then
have it pick up all messages sent to topics that match that pattern. So far I'm missing the
first new message sent to a new pattern.
>    e.g. it will miss the first message sent to test.a, but catch subsequent ones.
>    I've set the following:
>    auto.create.topics.enable=true
>    allow.auto.create.topics=true
>    And in the consumer properties I've set metadata.max.age.ms very low (around 50).
(I'm not sure what kind of performance problems doing this could do in production with multiple
>    Yet I'll still (almost always) miss the first message to a new topic in the pattern.
>    If anyone has any ideas, or even better things to google for to help solve this I'd
be most appreciative even if the result is "you just won't be able to do what you want" ;-)
>    Many thanks in advance,
>    Chris

View raw message