kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Bukowinski <pmb...@gmail.com>
Subject Re: multi-disk brokers data replication
Date Thu, 10 May 2018 18:41:14 GMT
Adrian,

Replicas are *always* assigned to different brokers. You cannot, for
example, deploy a single broker with a replication factor of 2 or 3 (with
min.insync.replicas of 2 or 3, respectively), even with multiple data
directories.

At the cluster level, kafka is not aware of an individual broker's storage
topology (single or multiple storage locations). Topic partitions on a
single, multi-data directory broker are distributed among storage locations
in a round-robin manner.

In a disk failure scenario, you will only lose one replica of all the topic
partitions that existed on that disk, assuming you're running 1.0+. If
you're not running 1.0+, then a single disk failure on a broker configured
with JBOD will bring down the broker.

Hope this helps,

Peter Bukowinski

On Thu, May 10, 2018 at 1:49 AM, Andrian Jardan <andrianjardan@gmail.com>
wrote:

> Hello everyone,
>
> I was wondering how data is spread across disks when more than 1 data
> folder is specified on a broker ?
>
> I am specifically interested to understand if failure of 3 disks may lead
> to data loss (with replication factor at 3)?
>
> Or is the data replicated so it resides on 3 brokers, and not 3 different
> data folders ?
>
> Thanks !
>
> —
> Andrian Jardan
> Infrastructure and DevOps expert
> cell: +49 174 2815994
> Skype: macrosdnb
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message