kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Caleb Spare <ca...@liftoff.io>
Subject How to find the latest timestamp for a partition
Date Thu, 25 Jan 2018 00:41:22 GMT
Hi. I have a general question about the broker and the protocol (not
specific to a particular client).

How do I efficiently discover the latest timestamp? By efficiently, I
mean: (1) with a fixed number of API calls and (2) without fetching
any messages.

I can get the earliest timestamp by doing a ListOffsets call with a
time far in the past. This will return the earliest offset plus the
timestamp for that offset. (Annoyingly, if I make the call with a
timestamp of -2, I get back the earliest offset but -1 for the
timestamp.)

There doesn't appear to be an equivalent way of getting the latest
timestamp since the API returns the first offset/timestamp *after* the
provided time. If I make a ListOffsets call with a timestamp in the
future, I just get an offset/timestamp of -1/-1.

I can obtain the latest offset by doing ListOffsets with a timestamp
of -1, but as in the -2 case that returns the desired offset and a
timestamp of -1.

Further, there is no API to go from offset to timestamp without
fetching a message.

Am I missing something?

If this is all correct, it seems like what would be most helpful is if
the ListOffsets API returned the accompanying timestamps when the
client uses the magic -1/-2 timestamps in the request.

Caleb

Mime
View raw message