spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From salemi <alireza.sal...@udo.edu>
Subject Structured Streaming + Kafka 0.10. connectors + valueDecoder and messageHandler with python
Date Sat, 09 Dec 2017 18:07:20 GMT
Hi All,

we are currently using direct streams to get the data from a kafka topic as
followed

KafkaUtils.createDirectStream(ssc=self.streaming_context,
                                            topics=topics,
                                            kafkaParams=kafka_params,
                                            valueDecoder=message_decoder,
                                            messageHandler=message_handler)

We would like to switch to to Structured Streaming approach such as 

self.spark_session \
            .readStream \
            .format("kafka") \
            .option("kafka.bootstrap.servers", kafkaServers) \
            .option("subscribe", self.topic_id) \
            .option("auto.offset.reset", self.msgoffset)\
            .load()

I was wondering how I can apply the existing message_decoder and
message_handler  functions to the message stream?


Thank you,

Ali




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message