kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vigneswaran Gunasekaran (vicky86)" <vick...@cisco.com>
Subject Re: Kafka Streams - Getting exception org.apache.kafka.common.network.InvalidReceiveException exception in cloud
Date Tue, 02 Jul 2019 13:28:43 GMT
Hi Jason,

Thanks for your reply.
I have no idea what the "client_id' field means. Because I am not having this field anywhere
else. 
For the corrupted data, we are receiving the data properly and we are getting this exception
intermediately. And after two to three days application stops working because of org.apache.kafka.common.errors.TimeoutException.

We couldn't able to debug this issue in local, because this happens in live and its real time.

Thanks,
Vigneswaran

On 02/07/19, 5:54 PM, "Jason Turim" <jason@signalvine.com> wrote:

    >
    > [2019-06-29 21:19:43,050] ERROR Exception while processing request from
    > 172.21.46.208:9092-172.21.4.208:38368-2446 (kafka.network.Processor)
    > org.apache.kafka.common.errors.InvalidRequestException: Error parsing
    > request header. Our best guess of the apiKey is: -32767
    > Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error
    > reading field 'client_id': Error reading string of length 27759, only 73
    > bytes available
    >         at org.apache.kafka.common.protocol.types.Schema.read(Schema.
    > java:77)
    >
    
    It looks like corrupt data is being introduced, maybe a null character in
    the `client_id` field?
    
    
    
    On Tue, Jul 2, 2019 at 8:13 AM Vigneswaran Gunasekaran (vicky86) <
    vicky86@cisco.com> wrote:
    
    > Can anybody help me on this issue?
    >
    > Thanks,
    > Vigneswaran
    >
    > From: "Vigneswaran Gunasekaran (vicky86)" <vicky86@cisco.com>
    > Date: Monday, 1 July 2019 at 12:45 PM
    > To: "users@kafka.apache.org" <users@kafka.apache.org>
    > Subject: Re: Kafka Streams - Getting exception
    > org.apache.kafka.common.network.InvalidReceiveException exception in cloud
    >
    > Hi Team,
    >
    > I am using kafka streams in my application and I am running in cloud with
    > 5 kafka instances. I am getting below exception in kafka machines and after
    > some point my application stop working because of
    > org.apache.kafka.common.errors.TimeoutException. Please help me on this as
    > we couldn’t  move further.
    >
    > server.properties(kafka performance configuration):
    > num.network.threads=8
    > socket.receive.buffer.bytes=-1
    > socket.send.buffer.bytes=-1
    > socket.request.max.bytes=2147483647
    >
    >
    > Kafka Exception:
    > [2019-06-29 21:17:57,649] INFO [Log
    > partition=location-analytics-live-v2-visit-processing-5,
    > dir=/mnt/data/kafka/kafka-logs] Incrementing log start offset to 263555323
    > (kafka.log.Log)
    > [2019-06-29 21:18:18,992] WARN [SocketServer brokerId=0] Unexpected error
    > from /172.21.4.208; closing connection
    > (org.apache.kafka.common.network.Selector)
    > org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
    > (size = -1145372416)
    >         at
    > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
    >         at
    > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
    >         at
    > org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
    >         at
    > org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
    >         at
    > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
    >         at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
    >         at kafka.network.Processor.poll(SocketServer.scala:689)
    >         at kafka.network.Processor.run(SocketServer.scala:594)
    >         at java.lang.Thread.run(Thread.java:748)
    > [2019-06-29 21:18:18,993] WARN [SocketServer brokerId=0] Unexpected error
    > from /172.21.4.208; closing connection
    > (org.apache.kafka.common.network.Selector)
    > org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
    > (size = -1145372671)
    >         at
    > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
    >         at
    > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
    >         at
    > org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
    >         at
    > org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
    >         at
    > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
    >         at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
    >         at kafka.network.Processor.poll(SocketServer.scala:689)
    >         at kafka.network.Processor.run(SocketServer.scala:594)
    >         at java.lang.Thread.run(Thread.java:748)
    > [2019-06-29 21:18:57,649] INFO [Log
    > partition=location-analytics-live-v2-visit-processing-5,
    > dir=/mnt/data/kafka/kafka-logs] Deleting segment 256880856 (kafka.log.Log)
    > [2019-06-29 21:18:57,676] INFO Deleted log
    > /mnt/data/kafka/kafka-logs/location-analytics-live-v2-visit-processing-5/00000000000256880856.log.deleted.
    > (kafka.log.LogSegment)
    > [2019-06-29 21:18:57,676] INFO Deleted offset index
    > /mnt/data/kafka/kafka-logs/location-analytics-live-v2-visit-processing-5/00000000000256880856.index.deleted.
    > (kafka.log.LogSegment)
    > [2019-06-29 21:18:57,676] INFO Deleted time index
    > /mnt/data/kafka/kafka-logs/location-analytics-live-v2-visit-processing-5/00000000000256880856.timeindex.deleted.
    > (kafka.log.LogSegment)
    > [2019-06-29 21:19:19,034] ERROR Closing socket for 172.21.46.208:9092
    > -172.21.4.208:36798-2446 because of error (kafka.network.Processor)
    > org.apache.kafka.common.errors.InvalidRequestException: Error parsing
    > request header. Our best guess of the apiKey is: 1032
    > [2019-06-29 21:19:19,034] ERROR Closing socket for 172.21.46.208:9092
    > -172.21.4.208:36798-2446 because of error (kafka.network.Processor)
    > org.apache.kafka.common.errors.InvalidRequestException: Error parsing
    > request header. Our best guess of the apiKey is: 1032
    > Caused by: java.nio.BufferUnderflowException
    >         at java.nio.Buffer.nextGetIndex(Buffer.java:506)
    >         at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:310)
    >         at
    > org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:118)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
    >         at scala.collection.Iterator.foreach(Iterator.scala:937)
    >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
    >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    >         at
    > kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
    >         at kafka.network.Processor.run(SocketServer.scala:595)
    >         at java.lang.Thread.run(Thread.java:748)
    > [2019-06-29 21:19:19,039] ERROR Exception while processing request from
    > 172.21.46.208:9092-172.21.4.208:36798-2446 (kafka.network.Processor)
    > org.apache.kafka.common.errors.InvalidRequestException: Error parsing
    > request header. Our best guess of the apiKey is: 1032
    > Caused by: java.nio.BufferUnderflowException
    >         at java.nio.Buffer.nextGetIndex(Buffer.java:506)
    >         at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:310)
    >         at
    > org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:118)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
    >         at scala.collection.Iterator.foreach(Iterator.scala:937)
    >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
    >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    >         at
    > kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
    >         at kafka.network.Processor.run(SocketServer.scala:595)
    >         at java.lang.Thread.run(Thread.java:748)
    > org.apache.kafka.common.errors.InvalidRequestException: Error parsing
    > request header. Our best guess of the apiKey is: -32767
    > Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error
    > reading field 'client_id': Error reading string of length 27759, only 73
    > bytes available
    >         at
    > org.apache.kafka.common.protocol.types.Schema.read(Schema.java:77)
    >         at
    > org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:121)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
    >         at scala.collection.Iterator.foreach(Iterator.scala:937)
    >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
    >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    >         at
    > kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
    >         at kafka.network.Processor.run(SocketServer.scala:595)
    >         at java.lang.Thread.run(Thread.java:748)
    > [2019-06-29 21:19:43,050] ERROR Exception while processing request from
    > 172.21.46.208:9092-172.21.4.208:38368-2446 (kafka.network.Processor)
    > org.apache.kafka.common.errors.InvalidRequestException: Error parsing
    > request header. Our best guess of the apiKey is: -32767
    > Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error
    > reading field 'client_id': Error reading string of length 27759, only 73
    > bytes available
    >         at
    > org.apache.kafka.common.protocol.types.Schema.read(Schema.java:77)
    >         at
    > org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:121)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
    >         at scala.collection.Iterator.foreach(Iterator.scala:937)
    >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
    >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    >         at
    > kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
    >         at kafka.network.Processor.run(SocketServer.scala:595)
    >         at java.lang.Thread.run(Thread.java:748)
    > [2019-06-29 21:19:43,229] ERROR Closing socket for 172.21.46.208:9092
    > -172.21.4.208:38370-2447 because of error (kafka.network.Processor)
    > java.lang.IndexOutOfBoundsException
    >         at java.nio.Buffer.checkIndex(Buffer.java:546)
    >         at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314)
    >         at
    > org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:126)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
    >         at scala.collection.Iterator.foreach(Iterator.scala:937)
    >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
    >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    >         at
    > kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
    >         at kafka.network.Processor.run(SocketServer.scala:595)
    >         at java.lang.Thread.run(Thread.java:748)
    > [2019-06-29 21:19:43,443] ERROR Exception while processing request from
    > 172.21.46.208:9092-172.21.4.208:38370-2447 (kafka.network.Processor)
    > java.lang.IndexOutOfBoundsException
    >         at java.nio.Buffer.checkIndex(Buffer.java:546)
    >         at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314)
    >         at
    > org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:126)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
    >         at
    > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
    >         at scala.collection.Iterator.foreach(Iterator.scala:937)
    >         at scala.collection.Iterator.foreach$(Iterator.scala:937)
    >         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    >         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    >         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    >         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    >         at
    > kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
    >         at kafka.network.Processor.run(SocketServer.scala:595)
    >        at java.lang.Thread.run(Thread.java:748)
    >
    >
    > Thanks,
    > Vigneswaran
    >
    
    
    -- 
    Jason Turim (he, him & his)
    ​Vice President of
    Software Engineering
    SignalVine Inc <http://www.signalvine.com>
    (m) 415-407-6501
    

Mime
View raw message