kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Rann <br...@zendesk.com.INVALID>
Subject Re: Queries on Kafka Capacity
Date Fri, 15 Dec 2017 09:44:47 GMT
You would add new brokers to the cluster, and then do a partition
reassignment to move some partitions to the new broker.

In the simplest example:

Say you have 1 topic with 3 partitions.

partition 0: brokers: 1,2
partition 1: brokers: 2,3
partition 2: brokers: 3,1

If you added 3 more brokers, you could do a reassignment so it is:

partition 0: brokers: 1,2
partition 1: brokers: 3,4
partition 2: brokers: 5,6

Which could be done with the following partition reassignment json:

{"partitions":[
             {"topic": "test", "partition": 0, "replicas": [1,2] },
             {"topic": "test", "partition": 1, "replicas": [3,4] },
             {"topic": "test", "partition": 2, "replicas": [5,6] }
  ], "version":1
}


/usr/local/bin/kafka-reassign-partitions --execute --reassignment-json-file
reassignment.json --throttle 80000000
and remember to verify:
/usr/local/bin/kafka-reassign-partitions --verify --reassignment-json-file
reassignment.json

In reality you'd have multiple topics and probably many partitions, but the
principle is the same.

Documentation here:
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-4.ReassignPartitionsTool


On Fri, Dec 15, 2017 at 12:12 PM, ajay chaudhary <
juniorajay@yahoo.com.invalid> wrote:

>
> Adding Users list to this email for help on below queries. please help us.
> Regards,Ajay chaudhary
>
>     On Thursday 14 December 2017, 1:07:56 PM IST, ajay chaudhary <
> juniorajay@yahoo.com> wrote:
>
>  Hi Team,
> This is Ajay working with Yodlee India. We are trying to setup Kafka
> cluster for streaming and we need some clarifications.
> Could you please help us to understand this behavior.
> How do we add a capacity to existing Kafka cluster. Let us assume, we have
> Kafka cluster with 3 brokers and all brokers has single mount point
> allocated(log.dirs). we create a topic with some partitions and after some
> days mount point becomes 100% full may be due to unexpected data growth.
> how do we add space to Kafka cluster so that existing partitions (say
> partition 0 where it originally resides is full) make use of newly
> provisioned space?
> how would new messages be handled coming to this partition? Please help us
> understand this?
> Regards,Ajay chaudhary
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message