kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiangjie Qin <j...@linkedin.com.INVALID>
Subject Re: querying messages based on timestamps
Date Wed, 01 Jul 2015 03:55:22 GMT
Yes, we have plan to add better support for this.

Thanks,

Jiangjie (Becket) Qin

On 6/30/15, 4:23 PM, "Zaiming Shi" <zmstone@gmail.com> wrote:

>Hi Jiangjie !
>Does the word 'Currently' imply any plan in introducing timestamp in log
>entries?
>
>Regards
>/Zaiming
>On Jun 30, 2015 11:08 PM, "Jiangjie Qin" <jqin@linkedin.com.invalid>
>wrote:
>
>> Currently Kafka only have a very coarse solution to find offset by time,
>> which is based on the segment last modified time.
>> This interface is only available in simple consumer. You may issue an
>> OffsetRequest to specify a timestamp. The offset returned will be the
>> first offset of segments whose last modification time is earlier than
>>the
>> timestamp you provided.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On 6/30/15, 1:16 PM, "Adam Dubiel" <dubiel.adam@gmail.com> wrote:
>>
>> >We faced similar problem and ended up with implementing variant of
>>golden
>> >section search, that reads message using simple consumer and checks the
>> >timestamp (timestamps are appended by our producer though, they do not
>> >come
>> >from any Kafka metadata) till it finds message closest to given date.
>> >
>> >Adam
>> >
>> >2015-06-30 21:52 GMT+02:00 Shushant Arora <shushantarora09@gmail.com>:
>> >
>> >> Is it possible using low level consumer to get kafka messages based
>>on
>> >> timestamp, say I want to get all messages from last 5 minutes. I
>>don't
>> >>know
>> >> what were offsets of partitions 5 minutes back.
>> >>
>> >> In low level consumer : when I gave epoch for  whichTime , it failed.
>> >>  requestInfo.put(topicAndPartition, new
>> >> PartitionOffsetRequestInfo(whichTime,
>> >> 1));
>> >>
>> >> Is only latest and earliest supported in timestamp,Is there any way
>>to
>> >> filter messages based on timestamp?
>> >>
>> >> Thanks
>> >> Shushant
>> >>
>>
>>


Mime
View raw message