kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject Re: producer not pushing the data into broker
Date Mon, 24 Feb 2014 06:04:21 GMT
Hi,

thanks for the input. I will try to increase the number of retries and 
check.

Thanks
Arjun Narasimha kota
On Monday 24 February 2014 10:44 AM, Jun Rao wrote:
> If those errors only show up transiently when brokers are started, then
> it's normal. It takes a bit of time for metadata to be probably to the
> newly started brokers. You may have to increase # retries to avoid the
> failure in the producer.
>
> Thanks,
>
> Jun
>
>
> On Sun, Feb 23, 2014 at 6:33 PM, Arjun <arjun@socialtwist.com> wrote:
>
>> Hi,
>> No i have 3 kafka brokers running, but in the same system. I tried with
>> the replication factor of 1 but it gives the same result.
>>
>> Thanks
>> Arjun Narasimha Kota
>>
>>
>> On Monday 24 February 2014 04:21 AM, Jun Rao wrote:
>>
>>> With only one broker, do you really want to have replication factor 2?
>>> Maybe you could retry with replication factor 1.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>>
>>>
>>> On Sat, Feb 22, 2014 at 12:54 AM, Arjun <arjun@socialtwist.com> wrote:
>>>
>>>   Hi please find below the o/p of the list and the console producer. There
>>>> are no errors in the state change log or in the controller log.
>>>>
>>>> One thing i found is this one happens only at starting time. I mean the
>>>> first time, when i start kafka and try to insert the messages i see this
>>>> error.
>>>>
>>>>
>>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
>>>> kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
>>>> localhost:2181,localhost:2182,localhost:2183 --topic test
>>>> Group           Topic                          Pid Offset logSize
>>>> Lag             Owner
>>>>
>>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>>>> bin/kafka-console-producer.sh --broker-list
>>>> localhost:9092,localhost:9093,localhost:9094
>>>> --topic test
>>>> THis ia s test message
>>>> [2014-02-22 14:16:26,924] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,120] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,259] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,293] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,443] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,499] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,630] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,693] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,881] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,884] ERROR Failed to send requests for topics test
>>>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
>>>> (kafka.producer.async.ProducerSendThread)
>>>>
>>>> kafka.common.FailedToSendMessageException: Failed to send messages after
>>>> 3 tries.
>>>>       at kafka.producer.async.DefaultEventHandler.handle(
>>>> DefaultEventHandler.scala:90)
>>>>       at kafka.producer.async.ProducerSendThread.tryToHandle(
>>>> ProducerSendThread.scala:104)
>>>>       at kafka.producer.async.ProducerSendThread$$anonfun$
>>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>>>       at kafka.producer.async.ProducerSendThread$$anonfun$
>>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>>>       at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>>>       at kafka.producer.async.ProducerSendThread.processEvents(
>>>> ProducerSendThread.scala:66)
>>>>       at kafka.producer.async.ProducerSendThread.run(
>>>> ProducerSendThread.scala:44)
>>>>
>>>>
>>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>>>> bin/kafka-list-topic.sh
>>>> --zookeeper localhost:2181
>>>> topic: test    partition: 0    leader: 2    replicas: 2,0    isr: 2,0
>>>> topic: test    partition: 1    leader: 0    replicas: 0,1    isr: 0,1
>>>> topic: test    partition: 2    leader: 1    replicas: 1,2    isr: 1,2
>>>> topic: test    partition: 3    leader: 2    replicas: 2,1    isr: 2,1
>>>> topic: test    partition: 4    leader: 0    replicas: 0,2    isr: 0,2
>>>> topic: test    partition: 5    leader: 1    replicas: 1,0    isr: 1,0
>>>> topic: test    partition: 6    leader: 2    replicas: 2,0    isr: 2,0
>>>> topic: test    partition: 7    leader: 0    replicas: 0,1    isr: 0,1
>>>> topic: test    partition: 8    leader: 1    replicas: 1,2    isr: 1,2
>>>> topic: test    partition: 9    leader: 2    replicas: 2,1    isr: 2,1
>>>> topic: test    partition: 10    leader: 0    replicas: 0,2    isr: 0,2
>>>> topic: test    partition: 11    leader: 1    replicas: 1,0    isr: 1,0
>>>>
>>>>
>>>> What i did:
>>>> started 3  kafka processes in my system.The config is default config with
>>>> just the ports and log dir's changed.There are 3 zookeeper servers
>>>> running.
>>>> If i start the console producer next time it will work fine.
>>>> To reproduce what did was, stopped all the kafka and zookeeper process,
>>>> deleted the log dirs of the kafka and the zookeeper and restated the
>>>> processes.
>>>>
>>>>
>>>> Thanks
>>>> Arjun Narasimha Kota
>>>>
>>>>
>>>>
>>>> On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
>>>>
>>>>   Could you do a list topic and show the output? Also, any error in the
>>>>> controller and state-change log?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <arjun@socialtwist.com>
wrote:
>>>>>
>>>>>    hi,
>>>>>
>>>>>> I am testing kafka 0.8 on my local machine. i have only one zookeeper
>>>>>> and
>>>>>> one kafka broker running.
>>>>>>
>>>>>> when i run the console producer i get this error:
>>>>>>
>>>>>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>>>>>> [{TopicMetadata for topic test ->
>>>>>> No partition metadata for topic test due to kafka.common.
>>>>>> LeaderNotAvailableException}]
>>>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>>>> (kafka.producer.BrokerPartitionInfo)
>>>>>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>>>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>>>> (kafka.producer.async.DefaultEventHandler)
>>>>>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>>>>>> [{TopicMetadata for topic test ->
>>>>>> No partition metadata for topic test due to kafka.common.
>>>>>> LeaderNotAvailableException}]
>>>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>>>> (kafka.producer.BrokerPartitionInfo)
>>>>>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics
test
>>>>>> with correlation ids in [0,8] (kafka.producer.async.
>>>>>> DefaultEventHandler)
>>>>>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>>>>>> ProducerSendThread)
>>>>>> kafka.common.FailedToSendMessageException: Failed to send messages
>>>>>> after
>>>>>> 3 tries.
>>>>>>        at kafka.producer.async.DefaultEventHandler.handle(
>>>>>> DefaultEventHandler.scala:90)
>>>>>>        at kafka.producer.async.ProducerSendThread.tryToHandle(
>>>>>> ProducerSendThread.scala:104)
>>>>>>        at kafka.producer.async.ProducerSendThread$$anonfun$
>>>>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>>>>>        at kafka.producer.async.ProducerSendThread$$anonfun$
>>>>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>>>>>        at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>>>>>        at kafka.producer.async.ProducerSendThread.processEvents(
>>>>>> ProducerSendThread.scala:66)
>>>>>>        at kafka.producer.async.ProducerSendThread.run(
>>>>>> ProducerSendThread.scala:44)
>>>>>>
>>>>>> Can some one please let me know why this is happening.
>>>>>> I checked my vhost files and localhost, my hostname are correct.
>>>>>>
>>>>>>
>>>>>>


Mime
View raw message