kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Gautier <tgaut...@tagged.com>
Subject Re: get offsets?
Date Fri, 23 Sep 2011 16:24:17 GMT
We thought of another idea - if you treat the messages as a reverse linked
list, then you write the offset of the previous message at the end of each
message.  Thus a message format would look like:

header
msg
offset off msg

This would allow for walking backwards at the cost of linear scans - if you
assume that you don't do this often enough this wouldn't have any cost at
write or read time except a tiny (negligible) penalty in space.

On Thu, Sep 22, 2011 at 8:22 AM, Taylor Gautier <tgautier@tagged.com> wrote:

> I think we are going to go with Kafka itself hopefully if the code isn't
> too hard to update - it already has everything we need, we just change it's
> receive message operation from:
>
> receive msg => write into log file
>
> receive msg => write into log file and get offset, write offset into index
> log file
>
> where index log file is just another topic that contains 64 bit offsets of
> the original log file.
>
> of course with batching and sendfile calls this may be trickier than we
> anticipate….
>
> On Thu, Sep 22, 2011 at 8:17 AM, Jeffrey Damick <jeffreydamick@gmail.com>wrote:
>
>> This was something i've asked for in the past as well.  To neha's comment,
>> sounds like you'd need some kind of table to maintain the list of offsets
>> and which segment they live in?
>>
>>
>>
>> On Thu, Sep 22, 2011 at 9:07 AM, Chris Burroughs
>> <chris.burroughs@gmail.com>wrote:
>>
>> > On 09/21/2011 10:06 PM, Taylor Gautier wrote:
>> > > I see that kafka-87 addresses this with a request for having a time
>> based
>> > > index, this would be relatively useful, but I also would like to have
>> a
>> > way
>> > > to go back say 1,000 messages.  Other than walking backwards one
>> segment
>> > at
>> > > a time, can then scanning forward from there, do you have any
>> suggestions
>> > > how this might be done or is it also a feature request?
>> >
>> > Could you elaborate a little on your use case where you need to rewind
>> > by a fixed number of messages?
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message