kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Huang <jason.hu...@icare.com>
Subject Fetch messages since a specific time?
Date Mon, 17 Dec 2012 18:23:19 GMT

Is it possible to fetch messages from the Kafka message queue since a
specific time? For example, a user may subscribe to a topic and the
producer will continuously publish messages related to this topic. The
first time this user logs in, we will fetch all the messages from the
beginning. However, the next time this user logs in, we want to only
fetch the "new" messages. In other words, messages since the user's
last log out time.

Is there any API in Kafka that allows us to do that? I am not sure if
Kafka actually stores a timestamp with each message as the message's
meta data. If not, is there any way to fetch the offset related to the
user's last log out time?

One way that I can think of to do this is to store the offset of the
last message this user consumers before he logged out of the system
(persist this offset at a DB). The next time this user logs in, we
will read the DB to get that offset and start from there to fetch
messages. However, if there is a better way to do this in Kafka, then
it will save me the work to write/read from the DB.



View raw message