spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <>
Subject Use cases for kafka direct stream messageHandler
Date Fri, 04 Mar 2016 21:39:14 GMT
Wanted to survey what people are using the direct stream
messageHandler for, besides just extracting key / value / offset.

Would your use case still work if that argument was removed, and the
stream just contained ConsumerRecord objects
which you could then use normal map transformations to access?

The only other valid use of messageHandler that I can think of is
catching serialization problems on a per-message basis.  But with the
new Kafka consumer library, that doesn't seem feasible anyway, and
could be handled with a custom (de)serializer.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message