kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: zero size ".index" files
Date Tue, 05 Nov 2013 04:33:31 GMT
Take a look at log.index.size.max.bytes described in
http://kafka.apache.org/documentation.html#brokerconfigs

Thanks,

Jun


On Sun, Nov 3, 2013 at 4:55 PM, lee yeonok <iyeonok@gmail.com> wrote:

> Hi, all.
> I am not quite understood how *.index files are used or created.
>
> I started a broker1, and set the property "log.segment.bytes" to "100".
> Here's a file tree in log directory what I've got after that broker1
> received some messages.
>
> kafka-logs-1/
> |-- preview-0
> |   |-- 00000000000000000000.index
> |   |-- 00000000000000000000.log
> |   |-- 00000000000000000031.index
> |   |-- 00000000000000000031.log
> |   |-- 00000000000000000033.index
> |   |-- 00000000000000000033.log
> |   |-- 00000000000000000035.index
> |   |-- 00000000000000000035.log
> |   |-- 00000000000000000037.index
> |   |-- 00000000000000000037.log
> |   |-- 00000000000000000039.index
> |   `-- 00000000000000000039.log
>
>
> At the point the segment size reached at 100 bytes, a new file is created
> as I expected, however, I am not sure why the old *.index files are being
> truncated.
> In my log directory,  00000000000000000000.index,
>  00000000000000000031.index, 00000000000000000033.index,
> 00000000000000000035.index,
> 00000000000000000037.index are zero size file. Is that all right?
>
> I thought that in this case the consumers are not able to get the messages
> from beginning since the old index files are being truncated, however I
> suspect that there's no problem to consume the messages from beginning.
>
> I removed all the index files, and found that when the broker get new
> message the latest index file is recovered. At this moment the old index
> files doesn't exists though, consuming all messages from beginning is not
> problem.
>
> So my conclusion is this Kafka only has the latest index, and in the case
> the old message requested Kafka re-parse every log files(not refer index
> file).
> Is it right? If then, having a zero size index files is unnecessary....
> Why does Kafka recreate all the missing index files when the broker shut
> down?
> I shut down a broker1, and the removed index files are recreated.
>
>
> --
> yeonok lee
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message