kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nimi Wariboko Jr <nimi.warib...@livereachmedia.com>
Subject Re: Programmatic method of setting consumer groups offsets
Date Fri, 04 Jan 2019 01:00:09 GMT
I’ll look into assign - I think the issue I am having is often I will want to do these offset
resets after the my consumer group has crashed. I guess what may be happening is that kafka
still thinks there are active consumers even though they have crashed.


________________________________
From: Eric Azama <eazama001@gmail.com>
Sent: Thursday, January 3, 2019 4:31 PM
To: users@kafka.apache.org
Subject: Re: Programmatic method of setting consumer groups offsets

Adding on to Ryanne's point, if subscribe() isn't giving your consumer all
of the partitions for a topic, that implies there are still active
consumers running for that group. The consumer groups CLI command does not
allow you to modify offsets for consumer groups that have active consumers.
I believe this is because running consumers will not reset to the offsets
that the command commits until they are restarted for some reason.
Committing is presumably part of the consumer's processing loop, so they'll
just overwrite whatever you committed with their current position.

On Thu, Jan 3, 2019 at 4:25 PM Ryanne Dolan <ryannedolan@gmail.com> wrote:

> Nimi, try using the low-level consumer API via assign() instead of
> subscribe(), and then commit() each partition. I suspect you are trying to
> commit partitions that the consumer is not subscribed to.
>
> Ryanne
>
> On Thu, Jan 3, 2019, 6:13 PM Nimi Wariboko <
> nimi.wariboko@livereachmedia.com
> wrote:
>
> > Hi,
> >
> > Is there a guide or API that is implemented that can modify stored
> offsets
> > in Kafka? For example, I'm looking to set a consumer groups stored
> offsets
> > in Kafka to a specific value. I know there is the `kafka-consumer-groups`
> > CLI command, but I'm looking for a way to do so from an application
> > (without calling out to the shell).
> >
> > I know I could potentially create a consumer, while joining a consumer
> and
> > committing offsets this way, however I've been trying this and it seems
> > error prone - very often I end up with issues where this method is only
> > able to commit a subset of offsets.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message