spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <cloud0...@gmail.com>
Subject Re: how can solve this error
Date Mon, 17 Sep 2018 11:13:43 GMT
have you read
https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
?

On Mon, Sep 17, 2018 at 4:46 AM hagersaleh <hager.saleh.fci@gmail.com>
wrote:

> I write code to connect kafka with spark using python and I run code on
> jupyer
> my code
> import os
> #os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars
>
> /home/hadoop/Desktop/spark-program/kafka/spark-streaming-kafka-0-8-assembly_2.10-2.0.0-preview.jar
> pyspark-shell'
> os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
> org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 pyspark-shell"
>
> os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
> org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 pyspark-shell"
>
> import pyspark
> from pyspark.streaming.kafka import KafkaUtils
> from pyspark.streaming import StreamingContext
>
> #sc = SparkContext()
> ssc = StreamingContext(sc,1)
>
> broker = "iotmsgs"
> directKafkaStream = KafkaUtils.createDirectStream(ssc, ["test1"],
>                         {"metadata.broker.list": broker})
> directKafkaStream.pprint()
> ssc.start()
>
> error display
> Spark Streaming's Kafka libraries not found in class path. Try one of the
> following.
>
>   1. Include the Kafka library and its dependencies with in the
>      spark-submit command as
>
>      $ bin/spark-submit --packages
> org.apache.spark:spark-streaming-kafka-0-8:2.3.0 ...
>
>   2. Download the JAR of the artifact from Maven Central
> http://search.maven.org/,
>      Group Id = org.apache.spark, Artifact Id =
> spark-streaming-kafka-0-8-assembly, Version = 2.3.0.
>      Then, include the jar in the spark-submit command as
>
>      $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message