kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lee yeonok <iyeo...@gmail.com>
Subject zero size ".index" files
Date Mon, 04 Nov 2013 00:55:50 GMT
Hi, all.
I am not quite understood how *.index files are used or created.

I started a broker1, and set the property "log.segment.bytes" to "100".
Here's a file tree in log directory what I've got after that broker1
received some messages.

kafka-logs-1/
|-- preview-0
|   |-- 00000000000000000000.index
|   |-- 00000000000000000000.log
|   |-- 00000000000000000031.index
|   |-- 00000000000000000031.log
|   |-- 00000000000000000033.index
|   |-- 00000000000000000033.log
|   |-- 00000000000000000035.index
|   |-- 00000000000000000035.log
|   |-- 00000000000000000037.index
|   |-- 00000000000000000037.log
|   |-- 00000000000000000039.index
|   `-- 00000000000000000039.log


At the point the segment size reached at 100 bytes, a new file is created
as I expected, however, I am not sure why the old *.index files are being
truncated.
In my log directory,  00000000000000000000.index,
 00000000000000000031.index, 00000000000000000033.index,
00000000000000000035.index,
00000000000000000037.index are zero size file. Is that all right?

I thought that in this case the consumers are not able to get the messages
from beginning since the old index files are being truncated, however I
suspect that there's no problem to consume the messages from beginning.

I removed all the index files, and found that when the broker get new
message the latest index file is recovered. At this moment the old index
files doesn't exists though, consuming all messages from beginning is not
problem.

So my conclusion is this Kafka only has the latest index, and in the case
the old message requested Kafka re-parse every log files(not refer index
file).
Is it right? If then, having a zero size index files is unnecessary....
Why does Kafka recreate all the missing index files when the broker shut
down?
I shut down a broker1, and the removed index files are recreated.


-- 
yeonok lee

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message