kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: More SocketTimeoutExceptions
Date Fri, 29 Mar 2013 23:08:38 GMT
This means that somehow the broker is not sending the response to the
producer and is sending it back to the wrong client. Any error in the
broker log?

Thanks,

Jun


On Fri, Mar 29, 2013 at 3:35 PM, Bob Jervis <bjervis@gmail.com> wrote:

> I now have the following settings (in various configs):
>
> In our producer configs:
>
> producer.request.timeout.ms=600000
>
> This producer just hangs there for 10 minutes before timing out
>
> Here is the stack dump for that timeout:
>
> java.net.SocketTimeoutException
>         at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
>         at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
>         at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:221)
>         at kafka.utils.Utils$.read(Utils.scala:372)
>         at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>         at
> kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>         at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>         at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
>         at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
>         at
>
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
>         at
>
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:98)
>         at
>
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:98)
>         at
>
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:98)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>         at
>
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:97)
>         at
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:97)
>         at
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:97)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>         at kafka.producer.SyncProducer.send(SyncProducer.scala:96)
>         at
>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:221)
>         at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:91)
>         at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:85)
>         at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
>         at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:631)
>         at
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
>         at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
>         at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>         at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
>         at
>
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:85)
>         at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:61)
>         at kafka.producer.Producer.send(Producer.scala:76)
>         at kafka.javaapi.producer.Producer.send(Producer.scala:41)
>         at
>
> com.visibletechnologies.platform.common.kafka.KafkaWriter.flush(KafkaWriter.java:131)
>         at
>
> com.visibletechnologies.platform.ingestion.ContentWriter.flushToKafka(ContentWriter.java:394)
>         at
>
> com.visibletechnologies.platform.ingestion.Midas.processPosts(Midas.java:430)
>         at
> com.visibletechnologies.platform.ingestion.Midas.doWork(Midas.java:194)
>         at
>
> com.visibletechnologies.framework.servicebase.ServiceBase.start(ServiceBase.java:187)
>         at
> com.visibletechnologies.platform.ingestion.Main.main(Main.java:413)
> 2013-03-29 22:31:58,633 INFO kafka.client.ClientUtils$: Fetching metadata
> for topic Set(VTFull-enriched)
>
>
> Kafka server.properties (both brokers):
>
> replica.socket.timeout.ms=200000
> controller.socket.timeout.ms=200000
> zk.connectiontimeout.ms=1000000
>
> After seeing these issues last week, the problems magically vanished, but
> yesterday they came back and we are now seeing consistent
> SocketTimeoutExceptions both in the producers and in the brokers
>
> We never see these issues where we have only one broker running.
>
> Thanks,
> Bob Jervis
> Visible Technologies
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message