spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-19776) Is the JavaKafkaWordCount example correct for Spark version 2.1?
Date Wed, 01 Mar 2017 11:39:46 GMT


Sean Owen resolved SPARK-19776.
    Resolution: Not A Problem

It sounds like you are trying to run the example without the examples JAR. If so, that's of
course why you are missing a lot of the dependencies. The example is correct on these points.

Questions should go to the mailing list, please.

> Is the JavaKafkaWordCount example correct for Spark version 2.1?
> ----------------------------------------------------------------
>                 Key: SPARK-19776
>                 URL:
>             Project: Spark
>          Issue Type: Question
>          Components: Examples, ML
>    Affects Versions: 2.1.0
>            Reporter: Russell Abedin
> My question is
> I'm pretty new to both Spark and Java.  I wanted to find an example of Spark Streaming
using Java, streaming from Kafka. The JavaKafkaWordCount at
looked to be perfect.
> However, when I tried running it, I found a couple of issues that I needed to overcome.
> 1. This line was unnecessary:
> {code}
> StreamingExamples.setStreamingLogLevels();
> {code}
> Having this line in there (and the associated import) caused me to go looking for a dependency
spark-examples_2.10 which of no real use to me.
> 2. After running it, this line: 
> {code}
> JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jssc,
args[0], args[1], topicMap);
> {code}
> Appeared to throw an error around logging:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(
>         at
>         at
>         at$100(
>         at$
>         at$
>         at Method)
>         at
>         at java.lang.ClassLoader.loadClass(
>         at java.lang.ClassLoader.loadClass(
>         at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:91
>         at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:66
>         at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:11
>         at org.apache.spark.streaming.kafka.KafkaUtils.createStream(KafkaUtils.scala)
>         at
> {code}
> To get around this, I found that the code sample in
helped me to come up with the right lines to see streaming from Kafka in action. Specifically
this called createDirectStream instead of createStream.
> So is the example in
or is there something I could have done differently to get that example working?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message