kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Fetch Latest messages from Kafka
Date Tue, 11 Sep 2012 14:35:37 GMT
That's an interesting use case and it is not what the ZK consumer is really
designed for. First, having a topic per user may not be ideal, especially
if you have millions of users. Second, the ZK consumer is designed for
continuously consuming data, not skipping data all the time.

With that said, you can probably try the following trick and see if it
works. Set both the log segment size and the log retention size to a value
that matches the size of about 100 messages. This should make sure that at
any given point time, only the most recent 100 messages are kept in the
broker. Set autooffset.reset to earliest. This will make sure that if the
consumer's offset is out of range, it will automatically switch to the
current smallest offset.

Thanks,

Jun

On Mon, Sep 10, 2012 at 9:07 PM, Ajit Koti <ajitkoti@gmail.com> wrote:

> Hi,
>
> No Consumers are not falling behind .
>   What happens is the users of the consumers don't fetch messages for
> certain period of time and when then the users want to fetch the want to
> fetch the latest messages from the topic.
>
> For example in case of displaying news feed each user has got its own topic
> , if the users don't login for couple of day
> he wouldn't consume the messages from the topic.But when he logs in he want
> to get the latest update for which we have to fetch the latest messages
> from the topic.
>
> So is it possible to use the zkconsumer and set the offset to last 100
> messages and then fetch them.
>
> Thanks
> Ajit Koti
>
>
> On Tue, Sep 11, 2012 at 9:04 AM, Jun Rao <junrao@gmail.com> wrote:
>
> > Is consuming falling behind? Can you have more consumers in the same
> group
> > to increase the consumption throughput?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 10, 2012 at 12:22 PM, Ajit Koti <ajitkoti@gmail.com> wrote:
> >
> > > Hi All ,
> > >
> > > Was happy consuming messages from the consumer .
> > > Recently got a new requirement  where I have to process the latest
> > message
> > > always .
> > > Is there anyways I can fetch the latest 100 messages.I know apart from
> > > simple consumer I cannot specify the offset .
> > > But still wondering is there any way to start consuming from a
> particular
> > > offset.
> > >
> > >
> > > Note : I have only one consumer per topic.
> > >
> > >
> > >
> > > Thanks
> > > Ajit
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message