kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathias Söderberg <mathias.soederb...@gmail.com>
Subject Re: Fetch messages since a specific time?
Date Mon, 17 Dec 2012 18:40:25 GMT
The SimpleConsumer API [1] has a method called getOffsetsBefore which takes
a topic, partition, timestamp (UNIX I assume since it's a long) and integer
limit on how many offsets to get.

Might not solve your problem *exactly*, but could be useful, unless you're
using the ConsumerConnector?

[1]: http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/


On 17 December 2012 19:23, Jason Huang <jason.huang@icare.com> wrote:

> Hello,
>
> Is it possible to fetch messages from the Kafka message queue since a
> specific time? For example, a user may subscribe to a topic and the
> producer will continuously publish messages related to this topic. The
> first time this user logs in, we will fetch all the messages from the
> beginning. However, the next time this user logs in, we want to only
> fetch the "new" messages. In other words, messages since the user's
> last log out time.
>
> Is there any API in Kafka that allows us to do that? I am not sure if
> Kafka actually stores a timestamp with each message as the message's
> meta data. If not, is there any way to fetch the offset related to the
> user's last log out time?
>
> One way that I can think of to do this is to store the offset of the
> last message this user consumers before he logged out of the system
> (persist this offset at a DB). The next time this user logs in, we
> will read the DB to get that offset and start from there to fetch
> messages. However, if there is a better way to do this in Kafka, then
> it will save me the work to write/read from the DB.
>
> thanks!
>
> Jason
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message