kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Re: Producer Error: Exception in thread ProducerSendThread
Date Mon, 07 Apr 2014 18:33:47 GMT
Hi Jun and Guozhang, 

Thanks for the feedback. 
We only have jars that come with the download; no rebuild. 
It could be the class loader related as Kafka producer, the servlet and Tomcat are running
in the same JVM. 
There could be version mismatch of jars (for Kafka) loaded into the JVM. 
We are looking into it more. 

Thanks,
Maung

On Apr 4, 2014, at 9:46 PM, Jun Rao <junrao@gmail.com> wrote:

> Do you have the right version of scala jar?
> 
> Thanks,
> 
> Jun
> 
> 
> On Fri, Apr 4, 2014 at 11:34 AM, Maung Than <maung_than@apple.com> wrote:
> 
>> Hi,
>> 
>> We are replacing the current data pipeline that sends messages to the
>> tomcat servlet that in turn writes to files and then pushes to HDFS to
>> process.
>> Instead of replacing it entirely at first, we are sourcing the same
>> messages and send it to Kafka broker through its producers invoked by the
>> same servlet.
>> 
>> We were trying to benchmark the insertion of 2B records under the
>> following configurations, but got an error and not sure why!!!!
>> And why is there a warning on zk.connectiontimeout.ms below as we do not
>> override it and shouldn't it be default to 6000.
>> 
>> -- 10 topics and 5 partitions per topic
>> --10 producers, each having eReporter
>> --each of the 10 producers will execute the same message files (about 100)
>> 2m times through eReporter
>> --10 brokers
>> --2 zookeepers
>> --each message having 2 transaction(size of message file = 725 bytes)
>> --Test start time : Fri Apr  4 05:44:01 PDT 2014 -   it worked fine and
>> records were also getting populated in the topics as expected but
>> --At  6:23:27 got the following error :
>> 
>> The error in the eReporter catalina.out:
>> 
>> upanda >> topic ::imessage
>> 6  23  27
>> Residue::3
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> Caused by: java.lang.ClassNotFoundException:
>> kafka.producer.async.ProducerSendThread$$anonfun$run$1
>>        at
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1678)
>>        at
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523)
>>        ... 1 more
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> Exception in thread "ProducerSendThread-upanda"
>> java.lang.NoClassDefFoundError:
>> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>>        at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>> 
>> The message in the eReporter log file :
>> 
>> 2014-04-04 06:22:55,417 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Verifying properties
>> 2014-04-04 06:22:55,417 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property batch.num.messages is overridden to 500
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property client.id is overridden to upanda
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property message.send.max.retries is overridden to 3
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property metadata.broker.list is overridden to nk11p16ic-erptapp005:3001
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property partitioner.class is overridden to
>> kafka.producer.DefaultPartitioner
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property producer.type is overridden to async
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property queue.buffering.max.messages is overridden to 10000
>> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property queue.buffering.max.ms is overridden to 5000
>> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property queue.enqueue.timeout.ms is overridden to -1
>> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property request.required.acks is overridden to 1
>> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property retry.backoff.ms is overridden to 100
>> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property serializer.class is overridden to kafka.serializer.StringEncoder
>> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
>> Property topic.metadata.refresh.interval.ms is overridden to 600000
>> 2014-04-04 06:22:55,419 [pool-1-thread-10] WARN  VerifiableProperties:83 -
>> Property zk.connectiontimeout.ms is not valid
>> 2014-04-04 06:22:56,048 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 2014-04-04 06:22:56,775 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 2014-04-04 06:22:57,504 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 2014-04-04 06:22:58,234 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 2014-04-04 06:22:58,980 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 2014-04-04 06:22:59,693 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 2014-04-04 06:23:00,422 [ProducerSendThread-upanda] INFO  ClientUtils$:68
>> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
>> with correlation id 0 for 1 topic(s) Set(imessage)
>> 
>> Thanks,
>> Maung


Mime
View raw message