kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugen Dueck <eu...@tworks.co.jp>
Subject Sequential writes make Kafka fast, or so they say
Date Thu, 12 Mar 2020 02:30:02 GMT
A question about something that was always in the back of my mind.

According to Jay Kreps

> The first [reason that Kafka is so fast despite writing to disk] is that Kafka does only
sequential file I/O.

I wonder how true this statement is, because Kafka uses 3 segments per partition. so even
with a single topic and partition per broker and disk, it would not be sequential. Now say
we have 1000 partitions per broker/disk, i.e. 3000 files. How can concurrent/interleaved writes
to thousands of files on a single disk be considered sequential file I/O?

Isn't the reason Kafka is so fast despite writing to disk the fact that it does not fsync
to disk, leaving that to the OS? The OS would, I assume, be smart enough to order the writes
when it flushes its caches to disk in a way that minimizes random seeks. But then, wouldn't
the manner in which Kafka writes to files be more or less irrelevant? Or put differently:
If Kafka was synchronously flushing to disk, wouldn't it have to limit itself to writing all
partitions for a broker/disk to a single file, if it wanted to do sequential file I/O?

For reading (historical, non-realtime) data that is not in the OS cache, keeping it in append-only
files, the statement makes of course sense.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message