kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Producer Error: Exception in thread ProducerSendThread
Date Fri, 04 Apr 2014 23:23:57 GMT
Maung,

How did you build the kafka jar? It is reporting NoClassDefFoundError on
some class, which is abnormal.

Guozhang


On Fri, Apr 4, 2014 at 11:34 AM, Maung Than <maung_than@apple.com> wrote:

> Hi,
>
> We are replacing the current data pipeline that sends messages to the
> tomcat servlet that in turn writes to files and then pushes to HDFS to
> process.
> Instead of replacing it entirely at first, we are sourcing the same
> messages and send it to Kafka broker through its producers invoked by the
> same servlet.
>
> We were trying to benchmark the insertion of 2B records under the
> following configurations, but got an error and not sure why!!!!
> And why is there a warning on zk.connectiontimeout.ms below as we do not
> override it and shouldn't it be default to 6000.
>
> -- 10 topics and 5 partitions per topic
> --10 producers, each having eReporter
> --each of the 10 producers will execute the same message files (about 100)
> 2m times through eReporter
> --10 brokers
> --2 zookeepers
> --each message having 2 transaction(size of message file = 725 bytes)
> --Test start time : Fri Apr  4 05:44:01 PDT 2014 -   it worked fine and
> records were also getting populated in the topics as expected but
> --At  6:23:27 got the following error :
>
> The error in the eReporter catalina.out:
>
> upanda >> topic ::imessage
> 6  23  27
> Residue::3
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Caused by: java.lang.ClassNotFoundException:
> kafka.producer.async.ProducerSendThread$$anonfun$run$1
>         at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1678)
>         at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523)
>         ... 1 more
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Exception in thread "ProducerSendThread-upanda"
> java.lang.NoClassDefFoundError:
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
>         at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
>
> The message in the eReporter log file :
>
> 2014-04-04 06:22:55,417 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Verifying properties
> 2014-04-04 06:22:55,417 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property batch.num.messages is overridden to 500
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property client.id is overridden to upanda
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property message.send.max.retries is overridden to 3
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property metadata.broker.list is overridden to nk11p16ic-erptapp005:3001
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property partitioner.class is overridden to
> kafka.producer.DefaultPartitioner
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property producer.type is overridden to async
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property queue.buffering.max.messages is overridden to 10000
> 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property queue.buffering.max.ms is overridden to 5000
> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property queue.enqueue.timeout.ms is overridden to -1
> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property request.required.acks is overridden to 1
> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property retry.backoff.ms is overridden to 100
> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property serializer.class is overridden to kafka.serializer.StringEncoder
> 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO  VerifiableProperties:68 -
> Property topic.metadata.refresh.interval.ms is overridden to 600000
> 2014-04-04 06:22:55,419 [pool-1-thread-10] WARN  VerifiableProperties:83 -
> Property zk.connectiontimeout.ms is not valid
> 2014-04-04 06:22:56,048 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
> 2014-04-04 06:22:56,775 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
> 2014-04-04 06:22:57,504 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
> 2014-04-04 06:22:58,234 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
> 2014-04-04 06:22:58,980 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
> 2014-04-04 06:22:59,693 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
> 2014-04-04 06:23:00,422 [ProducerSendThread-upanda] INFO  ClientUtils$:68
> - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001
> with correlation id 0 for 1 topic(s) Set(imessage)
>
> Thanks,
> Maung




-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message