kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tanneru, Raj" <rstann...@ebay.com>
Subject RE: Error while producing messages
Date Mon, 11 Aug 2014 17:08:02 GMT
I shutdown 3 out 5. With 2 brokers I start seeing failures after successfully sending some
messages. Not all messages are failing. I wanted to understand the case/s when we log below
message? If you notice there is difference in send/receive buffer size of actual and requested.
I don’t see this message in log for successful messages. Is it normal?

[2014-08-08 16:47:35,179] DEBUG Accepted connection from /10.254.243.142 on /10.66.107.231:9092.
sendBufferSize [actual|requested]: [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576]
(kafka.network.Acceptor)

-----Original Message-----
From: Guozhang Wang [mailto:wangguoz@gmail.com] 
Sent: Monday, August 11, 2014 10:01 AM
To: users@kafka.apache.org
Subject: Re: Error while producing messages

I am not sure I understand completely. How many brokers did you shutdown out of the total
number of 5 brokers? With single-partition topics, if the replication factor is 3, this partition
will be hosted on 3 brokers.

Guozhang


On Mon, Aug 11, 2014 at 9:46 AM, Tanneru, Raj <rstanneru@ebay.com> wrote:

> Sorry I should have provided this information. All my topics have 
> single partition meaning there are 2 nodes that already have topic 
> partition. It's just that the node having 3rd partition is down. So if 
> a message fails there is no reason why other message should succeed, 
> unless there is network saturation, broker resource exhaustion etc.
>
> Thanks,
> Raj Tanneru
>
> -----Original Message-----
> From: Guozhang Wang [mailto:wangguoz@gmail.com]
> Sent: Sunday, August 10, 2014 10:14 PM
> To: users@kafka.apache.org
> Subject: Re: Error while producing messages
>
> With replication of 3, partitions that happen to be only hosted on 
> those 3 shutdown brokers would be not available, unless controlled shutdown is used.
>
> Guozhang
>
>
> On Sat, Aug 9, 2014 at 10:59 AM, Tanneru, Raj <rstanneru@ebay.com> wrote:
>
> > Guozhang,
> >
> > Our replication factor is 3.
> >
> >
> > Thanks,
> > Raj Tanneru
> >
> >
> > -------- Original message --------
> > From: Guozhang Wang
> > Date:08/09/2014 10:21 AM (GMT-08:00)
> > To: users@kafka.apache.org
> > Subject: Re: Error while producing messages
> >
> > Raj,
> >
> > What is your replication factor setting in kafka brokers?
> >
> > Guozhang
> >
> >
> > On Fri, Aug 8, 2014 at 5:02 PM, Tanneru, Raj <rstanneru@ebay.com> wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > I am trying to do capacity sizing estimate for our kafka cluster. 
> > > I started with 5 broker cluster and 3 node zk. Used a simple java 
> > > based producer to send messages to 5 topics that are created in 
> > > the cluster. I used 2 client machines with 100 worker threads each 
> > > sending messages continuously. I didn't see any exceptions or 
> > > issues when all 5 brokers
> > are
> > > up. When I take down 3 brokers out of 5, and 4 topics out of 5. I 
> > > am
> > seeing
> > > below in broker logs.
> > >
> > >
> > >
> > > [2014-08-08 16:47:34,977] DEBUG Closing connection from /
> > > 10.254.243.142:33944 (kafka.network.Processor)
> > >
> > > [2014-08-08 16:47:35,179] DEBUG Accepted connection from
> > > /10.254.243.142 on /10.66.107.231:9092. sendBufferSize
> [actual|requested]:
> > > [131071|1048576] recvBufferSize [actual|requested]: 
> > > [131071|1048576]
> > > (kafka.network.Acceptor)
> > >
> > > [2014-08-08 16:47:35,179] DEBUG Processor 860 listening to new 
> > > connection from /10.254.243.142:33947 (kafka.network.Processor)
> > >
> > > [2014-08-08 16:47:35,179] DEBUG [KafkaApi-1] Error while fetching
> > metadata
> > > for [item_topic_0,0]. Possible cause: null 
> > > (kafka.server.KafkaApis)
> > >
> > > [2014-08-08 16:47:35,207] INFO Closing socket connection to / 
> > > 10.254.243.142. (kafka.network.Processor)
> > >
> > > [2014-08-08 16:47:35,207] DEBUG Closing connection from /
> > > 10.254.243.142:33947 (kafka.network.Processor)
> > >
> > > [2014-08-08 16:47:35,294] DEBUG Accepted connection from
> > > /10.254.243.142 on /10.66.107.231:9092. sendBufferSize
> [actual|requested]:
> > > [131071|1048576] recvBufferSize [actual|requested]: 
> > > [131071|1048576]
> > > (kafka.network.Acceptor)
> > >
> > >
> > >
> > > On the client machine(producer) I am seeing below error. Again 
> > > only one topic and 2 broker nodes are running.
> > >
> > >
> > >
> > > Exception in thread "pool-1-thread-107"
> > > kafka.common.FailedToSendMessageException: Failed to send messages 
> > > after
> > 3
> > > tries.
> > >
> > > Sending message 181171284512 for topic item_topic_0
> > >
> > >         at
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > sc
> > ala:90)
> > >
> > >         at kafka.producer.Producer.send(Producer.scala:76)
> > >
> > >         at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > >
> > >         at
> > >
> > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafka
> > Me
> > ssage(WorkerThread.java:33)
> > >
> > >         at
> > >
> > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerTh
> > re
> > ad.java:27)
> > >
> > >         at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe
> > cu
> > tor.java:886)
> > >
> > >         at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > java:908)
> > >
> > >         at java.lang.Thread.run(Thread.java:662)
> > >
> > > Exception in thread "pool-1-thread-108"
> > > kafka.common.FailedToSendMessageException: Failed to send messages 
> > > after
> > 3
> > > tries.
> > >
> > > Sending message 221252534837 for topic item_topic_0
> > >
> > >         at
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > sc
> > ala:90)
> > >
> > >         at kafka.producer.Producer.send(Producer.scala:76)
> > >
> > >         at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > >
> > >         at
> > >
> > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafka
> > Me
> > ssage(WorkerThread.java:33)
> > >
> > >         at
> > >
> > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerTh
> > re
> > ad.java:27)
> > >
> > >         at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe
> > cu
> > tor.java:886)
> > >
> > >         at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > java:908)
> > >
> > >         at java.lang.Thread.run(Thread.java:662)
> > >
> > >
> > >
> > > Not all messages are erroring out. Only a few messages are failing.
> > > Any idea what could be going on?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Raj Tanneru
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>



--
-- Guozhang
Mime
View raw message