spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: Different Kafka createDirectStream implementations
Date Tue, 08 Sep 2015 17:32:48 GMT
If you're providing starting offsets explicitly, then auto offset reset
isn't relevant.

On Tue, Sep 8, 2015 at 11:44 AM, Dan Dutrow <dan.dutrow@gmail.com> wrote:

> The two methods of createDirectStream appear to have different
> implementations, the second checks the offset.reset flags and does some
> error handling while the first does not. Besides the use of a
> messageHandler, are they intended to be used in different situations?
>
> def createDirectStream[ K: ClassTag, V: ClassTag, KD <: Decoder[K]:
> ClassTag,
> VD <: Decoder[V]: ClassTag,* R: ClassTag] *
> ( ssc: StreamingContext, kafkaParams: Map[String, String], fromOffsets:
> Map[TopicAndPartition, Long], * messageHandler: MessageAndMetadata[K, V]
> => R *
> ):
>
> def createDirectStream[ K: ClassTag, V: ClassTag, KD <: Decoder[K]:
> ClassTag, VD <: Decoder[V]: ClassTag]
> ( ssc: StreamingContext, kafkaParams: Map[String, String], topics:
> Set[String] )
> --
> Dan📱

Mime
View raw message