kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Producer Error: Exception in thread ProducerSendThread
Date Fri, 04 Apr 2014 18:34:30 GMT
Hi, 

We are replacing the current data pipeline that sends messages to the tomcat servlet that
in turn writes to files and then pushes to HDFS to process. 
Instead of replacing it entirely at first, we are sourcing the same messages and send it to
Kafka broker through its producers invoked by the same servlet. 

We were trying to benchmark the insertion of 2B records under the following configurations,
but got an error and not sure why!!!! 
And why is there a warning on zk.connectiontimeout.ms below as we do not override it and shouldn’t
it be default to 6000. 

— 10 topics and 5 partitions per topic
--10 producers, each having eReporter
--each of the 10 producers will execute the same message files (about 100) 2m times through
eReporter
--10 brokers
--2 zookeepers
--each message having 2 transaction(size of message file = 725 bytes)
--Test start time : Fri Apr  4 05:44:01 PDT 2014 -   it worked fine and records were also
getting populated in the topics as expected but 
--At  6:23:27 got the following error :

The error in the eReporter catalina.out:

upanda >> topic ::imessage
6  23  27
Residue::3
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Caused by: java.lang.ClassNotFoundException: kafka.producer.async.ProducerSendThread$$anonfun$run$1
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1678)
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523)
	... 1 more
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError: kafka/producer/async/ProducerSendThread$$anonfun$run$1
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)

The message in the eReporter log file :

2014-04-04 06:22:55,417 [pool-1-thread-10] INFO  VerifiableProperties:68 - Verifying properties
2014-04-04 06:22:55,417 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property batch.num.messages
is overridden to 500
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property client.id
is overridden to upanda
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property message.send.max.retries
is overridden to 3
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property metadata.broker.list
is overridden to nk11p16ic-erptapp005:3001
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property partitioner.class
is overridden to kafka.producer.DefaultPartitioner
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property producer.type
is overridden to async
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property queue.buffering.max.messages
is overridden to 10000
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property queue.buffering.max.ms
is overridden to 5000
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property queue.enqueue.timeout.ms
is overridden to -1
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property request.required.acks
is overridden to 1
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property retry.backoff.ms
is overridden to 100
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property serializer.class
is overridden to kafka.serializer.StringEncoder
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 - Property topic.metadata.refresh.interval.ms
is overridden to 600000
2014-04-04 06:22:55,419 [pool-1-thread-10] WARN  VerifiableProperties:83 - Property zk.connectiontimeout.ms
is not valid
2014-04-04 06:22:56,048 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)
2014-04-04 06:22:56,775 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)
2014-04-04 06:22:57,504 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)
2014-04-04 06:22:58,234 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)
2014-04-04 06:22:58,980 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)
2014-04-04 06:22:59,693 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)
2014-04-04 06:23:00,422 [ProducerSendThread-upanda] INFO  ClientUtils$:68 - Fetching metadata
from broker id:0,host:nk11p16ic-erptapp005,port:3001 with correlation id 0 for 1 topic(s)
Set(imessage)

Thanks,
Maung
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message