kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Lloyd <gllo...@gmail.com>
Subject Re: consumer groups
Date Wed, 17 Dec 2014 23:35:40 GMT
Thanks, now I understand largest.

On Wed, Dec 17, 2014 at 5:42 PM, Gwen Shapira <gshapira@cloudera.com> wrote:
>
> When you add a new consumer group, you can choose where it will start
> reading.
>
> With high-level consumer you can set auto.offset.reset to "largest"
> (new messages only) or "smallest" (all messages) , with simple
> consumer you can pick specific offsets.
>
>
> On Wed, Dec 17, 2014 at 2:33 PM, Greg Lloyd <glloyd1@gmail.com> wrote:
> > Thanks for the reply,
> >
> > So if I wanted to add a new group of consumers 6 months into the lifespan
> > of my implementation and I didn't want that new group to process all the
> > last six months is there a method to manage this?
> >
> >
> >
> > On Tue, Dec 16, 2014 at 9:48 PM, Gwen Shapira <gshapira@cloudera.com>
> wrote:
> >>
> >> " If all
> >> the consumers stop listening how long will Kafka continue to store
> messages
> >> for that group?"
> >>
> >> Kafka retains data for set amount of time, regardless of whether
> >> anyone is listening or not. This amount of time is configurable.
> >> Because Kafka performance is generally constant with the amount of
> >> data it stores, storing terabytes of data for months is not uncommon.
> >>
> >> Gwen
> >>
> >> On Tue, Dec 16, 2014 at 1:09 PM, Greg Lloyd <glloyd1@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I am planning to use kafka for a work queue type use case with
> multiple
> >> > consumers. I also plan to use it with multiple consumer groups. What
> is
> >> not
> >> > clear to me from the documentation is how to define+manage consumer
> >> groups
> >> > or if that is possible at all. It appears to me that a consumer group
> is
> >> > created just by connecting with a new group-id. What is not clear to
> me
> >> is
> >> > how to manage this: if I no longer want a consumer group how do I
> remove
> >> > it? What happens when there are no consumers of a group listening? If
> all
> >> > the consumers stop listening how long will Kafka continue to store
> >> messages
> >> > for that group?
> >> >
> >> > Thanks for any clarification,
> >> >
> >> > Greg Lloyd
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message