spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatakrishna T <ve...@thekratos.com>
Subject Re: NoSuchMethodError in KafkaReciever
Date Fri, 07 Mar 2014 04:20:35 GMT
Will give it a shot, later. BTW, this forced me to move to Scala! Decided to design our aggregation
frame-work in scala for now.
On 07-Mar-2014, at 6:02 AM, Tathagata Das <tathagata.das1565@gmail.com> wrote:

> I dont have a Eclipse setup so I am not sure what is going on here. I would try to use
maven in the command line with a pom to see if this compiles. Also, try to cleanup your system
maven cache. Who knows if it had pulled in a wrong version of kafka 0.8 and using it all the
time. Blowing away the cache and clean compiling will make sure the right kafka will be loaded.

> 
> Hope this helps.
> 
> TD
> 
> 
> On Sat, Mar 1, 2014 at 8:26 PM, venki-kratos <venki@thekratos.com> wrote:
> I am trying to user code similar to following :
> public  JavaPairDStream<String, String> openStream() {
>             HashMap<String, String> kafkaParams = Maps.newHashMap();
>             kafkaParams.put(ZK_CONNECT,kafkaConfig.getString(ZK_CONNECT));
> 
> kafkaParams.put(CONSUMER_GRP_ID,kafkaConfig.getString(CONSUMER_GRP_ID));
> 
>             Map<String,Integer> topicMap = Maps.newHashMap();
>             topicMap.put(kafkaConfig.getString(ZK_TOPIC),
> kafkaConfig.getInteger(CONSUMER_THREAD_COUNT, 1));
>             JavaPairDStream<String, String> inputStream =
>                         KafkaUtils.createStream(streamingContext, String.class, String.class,
>                                                         StringDecoder.class,
>                                                         StringDecoder.class,
>                                                         kafkaParams, topicMap, StorageLevel.MEMORY_AND_DISK_SER_2());
>             return inputStream;
>         }
> 
> I have spark-streaming_2.10-0.9.0-incubating.jar and
> spark-streaming-kafka_2.10-0.9.0-incubating.jar
> in the classpath using POM and m2e in Eclipse. JVM version is set to 1.6
> 
> I get the following error,
> 
> 14/03/02 09:29:15 INFO kafka.KafkaReceiver: Connected to localhost:2181
> 14/03/02 09:29:15 ERROR kafka.KafkaReceiver: Error receiving data
> java.lang.NoSuchMethodException:
> java.lang.Object.<init>(kafka.utils.VerifiableProperties)
>         at java.lang.Class.getConstructor0(Class.java:2763)
>         at java.lang.Class.getConstructor(Class.java:1693)
>         at
> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108)
>         at
> org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126)
>         at
> org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:173)
>         at
> org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:169)
>         at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
>         at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
>         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
>         at org.apache.spark.scheduler.Task.run(Task.scala:53)
>         at .....
> 
> This is similar to code in JavaKafkaStreamSuite.testKafkaStream. I find that
> the kafka jar - kafka_2.10-0.8.0 does have such a constructor.
> 
> What is going wrong? Can someone help solve this mystery and help with my
> misery? Basically stuck for last 2 days - as I am a Java Guy and would like
> to develop downstream code in Java
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 


Mime
View raw message