spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Penny Espinoza <pesp...@societyconsulting.com>
Subject spark-streaming-kafka with broadcast variable
Date Fri, 05 Sep 2014 17:36:10 GMT
I need to use a broadcast variable inside the Decoder I use for class parameter T in org.apache.spark.streaming.kafka.KafkaUtils.createStream.
 I am using the override with this signature:

createStream<https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kafka/KafkaUtils.html#createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20java.util.Map,%20java.util.Map,%20org.apache.spark.storage.StorageLevel)>(JavaStreamingContext<https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/api/java/JavaStreamingContext.html>
jssc, Class<K> keyTypeClass, Class<V> valueTypeClass, Class<U> keyDecoderClass,
Class<T> valueDecoderClass, java.util.Map<String,String> kafkaParams, java.util.Map<String,Integer>
topics,StorageLevel<https://spark.apache.org/docs/latest/api/java/org/apache/spark/storage/StorageLevel.html>
storageLevel)

Anyone know how I might do that?  The actual Decoder instance is instantiated by Spark, so
I don’t know how to access a broadcast variable inside the fromBytes method.

thanks
Penny


Mime
View raw message