spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Use cases for kafka direct stream messageHandler
Date Fri, 04 Mar 2016 21:39:14 GMT
Wanted to survey what people are using the direct stream
messageHandler for, besides just extracting key / value / offset.

Would your use case still work if that argument was removed, and the
stream just contained ConsumerRecord objects
(http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/ConsumerRecord.html)
which you could then use normal map transformations to access?

The only other valid use of messageHandler that I can think of is
catching serialization problems on a per-message basis.  But with the
new Kafka consumer library, that doesn't seem feasible anyway, and
could be handled with a custom (de)serializer.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message