kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chi Hoang <...@groupon.com>
Subject Re: Producer does not recognize new brokers
Date Mon, 13 Apr 2015 23:11:36 GMT
I highly recommend https://github.com/airbnb/kafkat, which will simplify
your partition management tasks.  Use it with
https://github.com/airbnb/kafkat/pull/3 for partition specific reassignment.

Chi

On Mon, Apr 13, 2015 at 4:08 AM, Jan Filipiak <Jan.Filipiak@trivago.com>
wrote:

> Hey,
>
> try to not have newlines \n in your jsonfile. I think the parser dies on
> those and then claims the file is empty
>
> Best
> Jan
>
>
>
>
>
> On 13.04.2015 12:06, Ashutosh Kumar wrote:
>
>> Probably you should first try to generate proposed plan using --generate
>> option and then edit that if needed.
>> thanks
>>
>>
>> On Mon, Apr 13, 2015 at 3:12 PM, shadyxu <shadyxu@gmail.com> wrote:
>>
>>  Thanks guys. You are right and then here comes another problems:
>>>
>>> I added new brokers 4, 5 and 6. Now I want to move partitions 3, 4 and
>>> 5(currently on broker 1, 2 and 3) of topic test to these brokers. I wrote
>>> r.json file like this:
>>>
>>> {"partitions":
>>> [{"topic": "test","partition": 3,"replicas": [4]},
>>> {"topic":"test","partition":4,"replicas":[5]},
>>> {"topic":"test","partition":5,"replicas":[6]},],
>>> "version":1}
>>>
>>> and then ran:
>>>
>>>      bin/kafka-reassign-partitions.sh --zookeeper [some-kafka-address]
>>> --reassignment-json-file r.json --execute
>>>
>>> Kafka gave me this error message:
>>>
>>>      kafka.common.AdminCommandFailedException: Partition reassignment
>>> data
>>> file r.json is empty
>>>
>>> I googled, seems Kafka parse the json file but found that no partitions
>>> were needed to be removed. Was my json file not properly configured?
>>>
>>> 2015-04-13 14:00 GMT+08:00 Ashutosh Kumar <kmr.ashutosh16@gmail.com>:
>>>
>>>  I think you need to re balance the cluster.
>>>> something like
>>>>
>>>> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
>>>> --topics-to-move-json-file topics-to-move.json --broker-list "5,6"
>>>> --generate
>>>>
>>>>
>>>> On Mon, Apr 13, 2015 at 11:22 AM, shadyxu <shadyxu@gmail.com> wrote:
>>>>
>>>>  I added several new brokers to the cluster, there should'v been a
>>>>>
>>>> rebalance
>>>>
>>>>> but it seemed that the producer was not aware of the new brokers. Data
>>>>>
>>>> kept
>>>>
>>>>> being sent to the old brokers and there were no partitions on the new
>>>>> brokers.
>>>>>
>>>>> I configured the old brokers to the producer and did not restart the
>>>>> producer or add the new brokers to the configuration.
>>>>>
>>>>> What may be the problems?
>>>>>
>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message